Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations
Authors: Shaocong Ma, Heng Huang
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through empirical evaluations on both synthetic problems and practical tasks, we demonstrate that DAPs outperform traditional methods under specific conditions. |
| Researcher Affiliation | Academia | Shaocong Ma and Heng Huang Department of Computer Science University of Maryland College Park, MD 20742, USA EMAIL |
| Pseudocode | Yes | Algorithm 1: The algorithm for sampling from a hyper-plane Algorithm 2: A practical implementation of gradient estimator using DAPs |
| Open Source Code | Yes | All source codes, including visualization scripts, are provided with our submission. |
| Open Datasets | Yes | We conducted experiments using the OPT-1.3b model (Zhang et al., 2022) for sentiment classification on the Stanford Sentiment Treebank (SST-2) dataset (Socher et al., 2013). |
| Dataset Splits | Yes | We conducted experiments using the OPT-1.3b model (Zhang et al., 2022) for sentiment classification on the Stanford Sentiment Treebank (SST-2) dataset (Socher et al., 2013). |
| Hardware Specification | Yes | We conducted our experiments on a cluster running RHEL8, equipped with Dual AMD EPYC 9124 processors and eight NVIDIA RTX 6000 Ada Generation graphics cards. |
| Software Dependencies | No | Our code was tested using Python version 3.10.10. Additional dependencies are specified in the supplementary requirements.txt file. |
| Experiment Setup | Yes | Learning rate η: 10 4; Perturbation size µ: 10 5; Zeroth-order gradient estimation batch size b: 2; Stochastic gradient updates batch size: 16. |