A Class of Parallel Doubly Stochastic Algorithms for Large-Scale Learning
Authors: Aryan Mokhtari, Alec Koppel, Martin Takac, Alejandro Ribeiro
JMLR 2020 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | RAPSA and its extensions are then numerically evaluated on a linear estimation problem and a binary image classification task using the MNIST handwritten digit dataset. 6. Numerical analysis In this section we study the numerical performance of the doubly stochastic approximation algorithms developed in Sections 2-4 by first considering a linear regression problem. We then use RAPSA to develop a visual classifier to distinguish between distinct hand-written digits. Figures 2-23 show the evolution of training error as a function of iterations and wall clock time for various values of I (number of workers used) for both synchronous and asynchronous settings. |
| Researcher Affiliation | Academia | Aryan Mokhtari EMAIL Department of Electrical and Computer Engineering The University of Texas at Austin Austin, TX 78712, USA Alec Koppel EMAIL Computational and Information Sciences Directorate U.S. Army Research Laboratory Adelphi, MD 20783, USA Martin Tak aˇc EMAIL Industrial and Systems Engineering Lehigh University, Bethlehem, PA 18015, USA Alejandro Ribeiro EMAIL Department of Electrical and Systems Engineering University of Pennsylvania Philadelphia, PA 19104, USA |
| Pseudocode | Yes | Algorithm 1 Random Parallel Stochastic Algorithm (RAPSA) ... Algorithm 2 Computation of the ARAPSA step ... Algorithm 3 Accelerated Random Parallel Stochastic Algorithm (ARAPSA) ... Algorithm 4 Asynchronous RAPSA at processor i ... Algorithm 5 Asynchronous Accelerated RAPSA at processor i |
| Open Source Code | No | The paper states: We wrote the code in C++ and compiled using Intel C++ compiler (version 16.0.2). However, there is no explicit statement or link indicating that this code has been made publicly available for the methodology described in the paper. |
| Open Datasets | Yes | RAPSA and its extensions are then numerically evaluated on a linear estimation problem and a binary image classification task using the MNIST handwritten digit dataset... We use the MNIST dataset... All datasets are available for download at https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/. |
| Dataset Splits | Yes | We run RAPSA on LMMSE estimation problem instances where q = 1, p = 1024, and N = 104 samples are given... We use the MNIST dataset, in which feature vectors zn Rp are p = 282 = 784 pixel images... a training set T = {zn, yn}N n=1 with N = 1.76 104 sample points... classification accuracy on a test subset of size N = 5.88 103... a binary training subset of size N = 105 samples. |
| Hardware Specification | Yes | In this Section, we report numerical experiments from a real high-performance computing (HPC) cluster. We wrote the code in C++ and compiled using Intel C++ compiler (version 16.0.2). Each node was equipped with two Intel Xeon Processor E7. |
| Software Dependencies | Yes | We wrote the code in C++ and compiled using Intel C++ compiler (version 16.0.2). Each node was equipped with two Intel Xeon Processor E7. The communication between nodes was achieved by MPI (we have used Intel s MPI implementation). |
| Experiment Setup | Yes | We first consider the performance of RAPSA (Algorithm 1) when using a constant step-size γt = γ = 10 2. The size of mini-batch is set as L = 10... We run RAPSA with a hybrid step-size scheme γt = min(ϵ, ϵ T0/t) which is a constant ϵ = 10 1.5 for the first T0 = 400 iterations... ARAPSA uses the curvature memory level s = 10... mini-batch size L = 10 and the level of curvature information set as τ = 10. We further select regularizer λ = 1/N = 7.5 10 3... Asynchronous RAPSA... no mini-batching L = 1 for both the case that the algorithm step-size is diminishing and constant step-size regimes... algorithm is initialized as x0 = 1031. |