Accelerating Adaptive Cubic Regularization of Newton's Method via Random Sampling
Authors: Xi Chen, Bo Jiang, Tianyi Lin, Shuzhong Zhang
JMLR 2022 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results on the regularized logistic regression problems demonstrate a clear effect of acceleration on several real data sets. (...) 6. Numerical Experiments We shall demonstrate the efficacy of the proposed method by presenting some computational results on different genres of real data. Experimental results on regularized logistic regression confirm that our algorithm is suitable for solving large-scale statistical learning problems and at least competitive with other algorithms. |
| Researcher Affiliation | Academia | Xi Chen EMAIL Stern School of Business New York University New York, NY 10012, USA Bo Jiang EMAIL Research Institute for Interdisciplinary Sciences School of Information Management and Engineering Shanghai University of Finance and Economics Shanghai 200433, China Tianyi Lin EMAIL Department of Electrial Engineering and Computer Science University of California, Berkeley Berkeley, CA 94720, USA Shuzhong Zhang EMAIL Department of Industrial and Systems Engineering University of Minnesota Minneapolis, MN 55455, USA |
| Pseudocode | Yes | Algorithm 1 Accelerated Subsampling Adaptive Cubic Regularized Newton s Method (...) Algorithm 2 SSAS(x0, σ0, ϵ0, ϵ, γ1, γ2, δ0, κθ) (...) Algorithm 3 ASAS(x0, σ0, σmin, ϵ0, ϵ, ς0, γ1, γ2, γ3, η, δ, κθ) |
| Open Source Code | No | The paper provides a GitHub link (https://github.com/dalab/subsampled_cubic_regularization) in footnote 1, referencing a 'variant of SCR in (Kohler and Lucchi, 2017) denoted as SCR-KL'. This code is for a baseline algorithm used for comparison, not explicitly for the authors' proposed SACR algorithm described in the paper. There is no explicit statement or link confirming the availability of the SACR code itself. |
| Open Datasets | Yes | Table 1: The Statistics of Eight LIBSVM Datasets Name Instances No. Features No. Processing SUSY 5,000,000 18 Done by Baldi et al. (2014) (...) all eight data sets are selected from the LIBSVM collection1 in which their statistics are summarized in Table 1 (...) 1. The LIBSVM collection is available at https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets |
| Dataset Splits | Yes | Problem. Given a collection of data samples {(wi, yi)}n i=1 in which yi { 1, 1}, the model of regularized logistic regression is given by min x Rd 1 n i=1 log 1 + e yiw i x + λ x 2 , where the regularization term 2 promotes smoothness and λ > 0 balances smoothness with goodness-of-fit and generalization and is chosen by five-fold cross validation. |
| Hardware Specification | Yes | all algorithms are implemented using Python 3.5 on a Mac Book Pro running with Mac OS High Sierra 10.13.6 and 16GB memory. |
| Software Dependencies | No | The paper states 'all algorithms are implemented using Python 3.5'. While Python 3.5 is a specific version of a programming language, the criteria specifies that a programming language version alone is insufficient without listing any versioned libraries or solvers. |
| Experiment Setup | Yes | We implement Algorithm 1 with η = 0.1, γ1 = γ2 = γ3 = 2, σmin = 10 16, σ0 = 1 and κθ = 0.1, denoted as SACR, in a hybrid manner. (...) In our experiment, we stop the second phase when |f(xi+1) f(xi)|/|f(xi)| 10 1 and the final stopping criterion as f(x) 10 7. (...) The sample size is chosen inversely proportional to the square norm of the gradient (cf. Lemmas 3 and 4) with proportional ratio 0.2 log(100d) and are bounded both below and above by some constants. (...) For LBFGS and MLBFGS implementations, we set an initial matrix as identity matrix and the line search criterion with strong Wolfe condition. The ratio for measuring the progress is set as 0.9, the maximum number of line search is set as 5 and the memory size is set as 30. (...) Finally, we set the stopping criterion for subproblem solving as (15) with κθ = 0.1. |