Incremental Learning in Diagonal Linear Networks
Authors: Raphaël Berthier
JMLR 2023 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | In this paper, we describe the trajectory of the gradient flow of DLNs in the limit of small initialization. We show that incremental learning is effectively performed in the limit: coordinates are successively activated, while the iterate is the minimizer of the loss constrained to have support on the active coordinates only. This shows that the sparse implicit regularization of DLNs decreases with time. This work is restricted to the underparametrized regime with anti-correlated features for technical reasons. |
| Researcher Affiliation | Academia | Rapha el Berthier EMAIL EPFL Lausanne, Switzerland |
| Pseudocode | No | The paper does not contain any clearly labeled pseudocode or algorithm blocks. It focuses on mathematical proofs and theoretical analysis. |
| Open Source Code | No | The paper does not contain any explicit statements about code availability, specific repository links, or mention of code in supplementary materials. |
| Open Datasets | No | In this simulation, n = 3 and the data X Rn d, y Rn is generated randomly with i.i.d. standard Gaussian entries, conditionally on the event that Assumptions (A1) and (A2) hold. |
| Dataset Splits | No | The paper describes synthetic data generated randomly for simulations to illustrate theoretical concepts, rather than using pre-existing datasets with defined splits for training, validation, or testing. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, processor types, memory amounts) used for running its simulations or computations. |
| Software Dependencies | No | The paper does not mention any specific software dependencies or libraries with version numbers (e.g., Python, PyTorch, TensorFlow) that would be needed to replicate the work. |
| Experiment Setup | Yes | The simulations are run with ε = 10 8 (left plots) and ε = 10 20 (right plots). In this simulation, n = 5, d = 4 and the data X Rn d, y Rn is generated randomly with i.i.d. standard Gaussian entries, conditionally on the event that Assumptions (A1) and (A2) hold. The initialization is θ(ε)(0) = (ε, . . . , ε) and thus k = (1, . . . , 1). |