Semantic Aware Representation Learning for Lifelong Learning
Authors: Fahad Sarfraz, Elahe Arani, Bahram Zonooz
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our empirical analysis on challenging class-incremental learning scenarios across various datasets demonstrates that SARL significantly enhances lifelong learning performance by leveraging semantic structure for representation learning. By effectively aligning representations based on semantic relationships, SARL facilitates efficient knowledge transfer and consolidation, enabling the model to adapt seamlessly to new tasks while retaining previously acquired knowledge. These capabilities are further supported by our analysis, which reveals that SARL achieves a better balance between model stability and plasticity, mitigates forgetting, and reduces task recency bias. ... Table 1: Comparison analysis of single and dual-model CL methods across various CL settings. |
| Researcher Affiliation | Collaboration | Fahad Sarfraz1,2, Elahe Arani1,3, & Bahram Zonooz1,* 1 Dep. of Mathematics and Computer Science, Eindhoven University of Technology (TU/e), Netherlands 2 Tom Tom, Netherlands 3 Wayve Technologies Ltd, London, United Kingdom |
| Pseudocode | No | The paper describes the methodology using textual explanations and mathematical equations (e.g., Equation 1 to 7) but does not include any clearly labeled pseudocode or algorithm blocks. Figure 1 provides a high-level visual overview, but it is not a structured algorithm. |
| Open Source Code | Yes | 1Code is available at https://github.com/Neur AI-Lab/SARL.git. |
| Open Datasets | Yes | For the Class-IL setting, we follow established baselines and evaluate on three datasets: sequential CIFAR-10 (Seq-CIFAR10) (Krizhevsky et al., 2009), where 10 classes are split into 5 disjoint tasks with 2 classes per task; sequential CIFAR-100 (Seq-CIFAR100) (Krizhevsky et al., 2009), splitting 100 classes into 5 tasks with 20 classes each; and sequential Tiny Image Net (Le & Yang, 2015) (Seq-Tiny Image Net), where 200 classes are divided into 10 tasks of 20 classes each. |
| Dataset Splits | Yes | For the Class-IL setting, we follow established baselines and evaluate on three datasets: sequential CIFAR-10 (Seq-CIFAR10) (Krizhevsky et al., 2009), where 10 classes are split into 5 disjoint tasks with 2 classes per task; sequential CIFAR-100 (Seq-CIFAR100) (Krizhevsky et al., 2009), splitting 100 classes into 5 tasks with 20 classes each; and sequential Tiny Image Net (Le & Yang, 2015) (Seq-Tiny Image Net), where 200 classes are divided into 10 tasks of 20 classes each. ... For the Generalized Class-IL (GCIL) setting (Mi et al., 2020), probabilistic modeling is used to randomly vary three task characteristics: the number of classes, the specific classes included, and the sample sizes. As in Sarfraz et al. (2023), we apply this GCIL setting to CIFAR-100, using 20 tasks with 1,000 samples per task and a maximum of 50 classes per task. |
| Hardware Specification | No | The paper mentions using a ResNet-18 architecture and an SGD optimizer, but it does not specify any hardware details like GPU models, CPU types, or memory specifications used for running the experiments. |
| Software Dependencies | No | The paper mentions using a ResNet-18 architecture and an SGD optimizer but does not provide specific version numbers for any software libraries, frameworks, or programming languages used in the implementation or experimentation. |
| Experiment Setup | Yes | In line with Buzzega et al. (2020), we use a Res Net-18 architecture trained with an SGD optimizer and a batch size of 32 for both task data and memory buffer. We use an initial learning rate (lr) of 0.03 with multistep decay scheduler with factor 0.1. Standard data augmentations random cropping and horizontal flipping are applied. For all tasks, we include a warm-up stage consisting of 3 epochs... For Seq-CIFAR10 the model is trained for 20 epochs with lr decay steps at 15th epoch, for Seq CIFAR100 and GCIL-CIFAR100, we extend the training to 50 epochs with lr decay steps at 35 and 45. For the Tiny Image Net dataset, we further increase the number of epochs to 100 with lr decay steps at 70 and 90. ... For all our experiments, we use λSM=0.01, τs=0.8, and β=1. The value of kw% was selected from the set {0.7, 0.8, 0.9}, while λOP and α, were chosen from {0.2, 0.5, 1}. |