Towards Faster Decentralized Stochastic Optimization with Communication Compression
Authors: Rustem Islamov, Yuan Gao, Sebastian Stich
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide numerical experiments to validate our theoretical findings and confirm the practical superiority of Mo TEF. ... 4 NUMERICAL EXPERIMENTS In this section, we complement the theoretical results on the convergence of Algorithm 1 with numerical evaluations. |
| Researcher Affiliation | Academia | Rustem Islamov1 Yuan Gao2,3 Sebastian U. Stich3 1Universität Basel, 2Universität des Saarlandes, 3CISPA Helmholtz Center for Information Security EMAIL, EMAIL |
| Pseudocode | Yes | Algorithm 1 Mo TEF ... Algorithm 2 Mo TEF-VR |
| Open Source Code | Yes | Our implementation is based on open-source code from (Zhao et al., 2022) https://github.com/ liboyue/beer and is available at https://anonymous.4open.science/r/Mo TEF-0DCF. The code to reproduce our synthetic experiment is available at https:// anonymous.4open.science/r/decentralized-exp-A3C6 |
| Open Datasets | Yes | We set λ = 0.05, n = 100 and use Lib SVM datasets (Chang & Lin, 2011). ... Finally, we consider training MLP on MNIST dataset (Deng, 2012) with 1 hidden layer of size 32. |
| Dataset Splits | No | We do not shuffle datasets to have a more heterogeneous setting. Besides, each dataset is equally distributed among all clients. ... Finally, we consider training MLP on MNIST dataset (Deng, 2012) with 1 hidden layer of size 32. The paper does not provide explicit details about training/test/validation splits (e.g., percentages, sample counts, or citations to specific standard splits used in their experimental setup) for the datasets mentioned. |
| Hardware Specification | Yes | We run our experiments on AMD EPYC 9554 64-Core Processor. |
| Software Dependencies | Yes | To find a suitable choice of constants we use Symbolic Math Toolbox in MATLAB (Inc., 2023). Our code can be found at https://anonymous.4open.science/r/dec-symb-verification. |
| Experiment Setup | Yes | We fix the parameters γ = 0.1, η = 0.0005, λ = 0.005, and n = 16, if the opposite is not stated. ... For Mo TEF we tune stepsize as follows η {0.001, 0.01, 0.05}, γ {0.1, 0.2, 0.5, 0.9}, λ {0.005, 0.01, 0.05, 0.1}. |