Continual Learning: Applications and the Road Forward
Authors: Eli Verwimp, Rahaf Aljundi, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler L. Hayes, Eyke Hüllermeier, Christopher Kanan, Dhireesha Kudithipudi, Christoph H. Lampert, Martin Mundt, Razvan Pascanu, Adrian Popescu, Andreas S. Tolias, Joost van de Weijer, Bing Liu, Vincenzo Lomonaco, Tinne Tuytelaars, Gido M van de Ven
TMLR 2024 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To achieve this, we examined continual learning papers accepted at four top machine learning conferences (ECCV 22, Neur IPS 22, CVPR 23 and ICML 23) to have a representative sample of the current field. We considered all papers with either incremental , continual , forgetting , lifelong or catastrophic in their titles, disregarding false positives. For our final set of 77 papers, we investigated how they balance the memory and compute cost trade-offs. We discern five categories... In Table 1 we provide a table of all the papers we used in the analysis of Section 2, showing their minimal and maximal sample store ratio (SSR) i.e. the percentage of samples stored, as well as possibly other memory consumption. The last column mentions how they approached the computational cost. Figure 1: Most papers strongly restrict data storage and do not discuss computational cost. Each dot represents one paper, illustrating what percentage of data their methods store (horizontal axis) and how computational cost is handled (vertical axis). |
| Researcher Affiliation | Collaboration | Eli Verwimp KU Leuven, Belgium Rahaf Aljundi Toyota Motor Europe, Belgium Shai Ben-David University of Waterloo, and Vector Institute, Ontario, Canada Matthias Bethge University of Tübingen, Germany Andrea Cossu University of Pisa, Italy Alexander Gepperth University of Applied Sciences Fulda, Germany Tyler L. Hayes NAVER LABS Europe, France Eyke Hüllermeier University of Munich (LMU), Germany Christopher Kanan University of Rochester, Rochester, NY, USA Dhireesha Kudithipudi University of Texas at San Antonio, TX, USA Christoph H. Lampert Institute of Science and Technology Austria (ISTA) Martin Mundt TU Darmstadt & hessian.AI, Germany Razvan Pascanu Google Deep Mind, UK Adrian Popescu Université Paris-Saclay, CEA, LIST, France Andreas S. Tolias Baylor College of Medicine, Houston, TX, USA Joost van de Weijer Computer Vision Center, UAB, Barcelona, Spain Bing Liu University of Illinois at Chicago, USA Vincenzo Lomonaco University of Pisa, Italy Tinne Tuytelaars KU Leuven, Belgium Gido M. van de Ven KU Leuven, Belgium |
| Pseudocode | No | The paper is a survey and perspective piece on continual learning and does not present new algorithms or methods that would typically include pseudocode. It discusses concepts, analyzes existing literature, and proposes future research directions. |
| Open Source Code | No | This paper does not present any new algorithm or dataset, hence the potential direct societal and ethical implications are rather limited. The paper is a survey and perspective piece, focusing on discussion and analysis of existing literature, and therefore does not provide code for any new methodology. |
| Open Datasets | No | This paper does not present any new algorithm or dataset, hence the potential direct societal and ethical implications are rather limited. While the paper references various well-known datasets in its discussion of other works (e.g., ImageNet1K, customnews benchmark, Atari-57 benchmark), it does not introduce a new dataset or provide access information for data used in its own analysis. |
| Dataset Splits | No | The paper is a survey and perspective piece that analyzes existing literature and discusses future directions in continual learning. It does not conduct experiments using its own datasets, and therefore does not specify training/test/validation dataset splits for reproducibility. |
| Hardware Specification | No | The paper is a survey and perspective piece that analyzes existing literature and discusses future directions in continual learning. It does not conduct its own experiments that would require specific hardware. While it mentions '10 000 GPU days of training' in the context of large models (Radford et al., 2021), this refers to the work of others and not the present paper's methodology. |
| Software Dependencies | No | The paper is a survey and perspective piece that analyzes existing literature and discusses future directions in continual learning. It does not present new algorithms or conduct its own experiments, and therefore does not list specific software dependencies with version numbers. |
| Experiment Setup | No | The paper is a survey and perspective piece that analyzes existing literature and discusses future directions in continual learning. It does not conduct its own experiments or propose a new method that would require detailing hyperparameters or training configurations for an experimental setup. |