A Unified q-Memorization Framework for Asynchronous Stochastic Optimization
Authors: Bin Gu, Wenhan Xian, Zhouyuan Huo, Cheng Deng, Heng Huang
JMLR 2020 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The experimental results on various large-scale datasets confirm the fast convergence of our Asy SGHT-q M, Asy SPG-q M and Asy SGD-q M through concrete realizations of SVRG and SAGA. In this section, we first give the experimental setup, then present our experimental results and discussion. |
| Researcher Affiliation | Collaboration | Bin Gu EMAIL School of Computer & Software, Nanjing University of Information Science & Technology, China Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA, 15261, USA JD Finance America Corporation, Mountain View, CA, 94043, USA |
| Pseudocode | Yes | Algorithm 1 HSAG({αt i}l i=1, J, m, t ) ; Algorithm 2 Generalized Variance Reduction Asynchronous Stochastic Gradient Hard Thresholding Algorithm (Asy SGHT-q M) ; Algorithm 3 Asynchronous Stochastic Proximal Gradient Algorithm with Generalized Variance Reduction (Asy SPG-q M) ; Algorithm 4 Generalized Variance Reduction Asynchronous Stochastic Gradient Descent Algorithm (Asy SGD-q M) |
| Open Source Code | No | We implement our Asy SGHT-q M, Asy SPG-q M and Asy SGD-q M (including SVRG and SAGA) using C++, where the shared memory parallel computation is handled via Open MP (Chandra, 2001). The paper describes implementation but does not state that the source code is publicly available or provide a link. |
| Open Datasets | Yes | Table 3 summarizes the six large-scale real-world binary classification datasets (i.e., the A1a, Covtype, Phishing, Criteo, Kdd2012, and URL datasets) used in our experiments. They are from LIBSVM website(Chang and Lin, 2011)2. 2. The datasets are available at: https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/. |
| Dataset Splits | No | For the experiments to the Asy SPG-q M algorithm, we consider the sparse logistic regression (26) with cardinality constraint for binary classification problem. The number of iterations for each epoch is the number of training samples. The paper does not provide specific details on training, validation, or test dataset splits. |
| Hardware Specification | Yes | Our experiments are performed on a 32-core two-socket Intel Xeon E5-2699 machine where each socket has 16 cores. |
| Software Dependencies | No | We implement our Asy SGHT-q M, Asy SPG-q M and Asy SGD-q M (including SVRG and SAGA) using C++, where the shared memory parallel computation is handled via Open MP (Chandra, 2001). The paper mentions C++ and Open MP but does not specify their version numbers. |
| Experiment Setup | Yes | In the experiments, the steplength γ for all compared methods is selected from {102, 10, 1, 10 1, 10 2, 10 3, 10 4, 10 5} according to the optimal value of objective funtion reached in a fixed number of iterations, and the number of iterations for each epoch is the number of training samples. For the experiments of Asy SPG-q M, we fix the parameter of λ1 to 10 6. q is set as 10. k is set as 10, 30 and 50 respectively. |