Unlocking the Power of Rehearsal in Continual Learning: A Theoretical Perspective
Authors: Junze Deng, Qinhang Wu, Peizhong Ju, Sen Lin, Yingbin Liang, Ness Shroff
ICML 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This work provides the first comprehensive theoretical analysis of rehearsalbased CL. ... Our experiments on real datasets with DNNs verify that our hybrid approach can perform better than concurrent rehearsal. |
| Researcher Affiliation | Academia | 1Department of ECE, Ohio State University, Columbus, USA 2Department of CSE, Ohio State University, Columbus, USA 3Department of CS, University of Kentucky, Lexington, USA 4Department of CS, University of Houston, Houston, USA. Correspondence to: Junze Deng <EMAIL>. |
| Pseudocode | Yes | Algorithm 1 Hybrid Rehearsal Training Framework |
| Open Source Code | No | The paper does not provide a direct link to source code, an explicit statement of code release, or mention code in supplementary materials for the described methodology. It discusses the algorithm and experimental results but not code availability. |
| Open Datasets | Yes | We consider three real-world datasets under a task-incremental CL setup: CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and Tiny Imagenet200 (Le & Yang, 2015). |
| Dataset Splits | Yes | For example, in Split-CIFAR-10, the sequence of tasks {T1, . . . , T5} is constructed such that each task contains two distinct classes. ... For training on Split-CIFAR-10, Split CIFAR-100, and Split-Tiny Imagenet200, we employ a non-pretrained Res Net-18 as our DNN backbone. Following (Van de Ven et al., 2022), we adopt a multi-headed output layer such that each task is assigned its own output layer, consistent with the typical Task Incremental CL setup. During supervised training, we explicitly provide the task identifier (ranging from 1 to 5 for Split-CIFAR-10) alongside the image-label pairs as additional input to the model. |
| Hardware Specification | Yes | Operating system: Red Hat Enterprise Linux Server 7.9 (Maipo) Type of CPU: 2.4 GHz 14-Core Intel Xeon E5-2680 v4 (Broadwell) Type of GPU: NVIDIA P100 Pascal GPU with 16GB memory |
| Software Dependencies | No | The paper mentions software components like 'ResNet18' (a DNN architecture), 'SGD optimizer', and 'Step LR learning rate scheduler', but does not specify their version numbers or the versions of underlying libraries or programming languages (e.g., PyTorch, Python, CUDA). |
| Experiment Setup | Yes | Our buffer size is 200, 300, 30, 30 per class for Split-MNIST, Split-CIFAR-10, Split-CIFAR-100, and Split-Tiny Imagenet200, correspondingly. For all experiments on concurrent rehearsal and sequential rehearsal, we use the SGD optimizer with a Step LR learning rate scheduler, which decays the learning rate by a fixed factor at predefined intervals. The detailed parameters for each dataset are listed in Table 4. |