Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1]
What Causes the Test Error? Going Beyond Bias-Variance via ANOVA
Authors: Licong Lin, Edgar Dobriban
JMLR 2021 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We verify our results in numerical simulations and on empirical data examples. Keywords: Test Error, ANOVA, Double Descent, Ridge Regression, Random Matrix Theory... In this section, we perform several numerical experiments, to check the correctness of our theoretical results... We also show some experiments on empirical data, specifically on the superconductivity data set (Hamidieh, 2018), where we test our predictions for two-layer orthogonal nets. |
| Researcher Affiliation | Academia | Licong Lin EMAIL School of Mathematical Sciences Peking University 5 Yiheyuan Road, Beijing, China Edgar Dobriban EMAIL Departments of Statistics & Computer and Information Science University of Pennsylvania Philadelphia, PA, 19104-6340, USA |
| Pseudocode | No | The paper describes mathematical derivations and theoretical results. It does not include any explicitly labeled "Pseudocode" or "Algorithm" blocks, nor does it present structured steps in a code-like format in the main text. |
| Open Source Code | Yes | Code associated with the paper is available at https://github.com/licong-lin/Variance Decomposition. |
| Open Datasets | Yes | We use the Superconductivity Data Set (Hamidieh, 2018) retrieved from the UC Irvine Machine Learning Repository in our data analysis. |
| Dataset Splits | Yes | We then separate the data set into a training set containing the first 90% of the samples, and a test set containing the rest. |
| Hardware Specification | No | The paper does not explicitly mention any specific hardware (like GPU or CPU models, memory, or cloud instances) used for running the experiments or numerical simulations. |
| Software Dependencies | No | The paper does not provide specific version numbers for any software libraries, programming languages, or tools used in the numerical simulations or empirical data analysis. |
| Experiment Setup | Yes | Parameters: signal strength α = 1, noise level σ = 0.3, regularization parameter λ = 0.01, parametrization level π = 0.8. See Sections 2.2, 4.2 for details. ... Parameters: α = 1, σ = 0.3, π = 0.8, n = 150, d = nδ , p = dπ . ... For simplicity, we subtract the test point label noise σ2 from the MSE formula. We randomly generate k = 400 i.i.d. tuples of random variables (xi, θi, εi, Xi, Wi), 1 i k, from their assumed distributions (we assume X and x have i.i.d. N(0, 1) entries in numerical simulations), and estimate the MSE by calculating:... |