Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variance
Authors: Dun Zeng, Zenglin Xu, Yu Pan, Xu Luo, Qifan Wang, Xiaoying Tang
TMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This section evaluates the convergence benefits of utilizing FL client samplers. Our experiment evaluation aligns with previous works (Li et al., 2020b; Chen et al., 2020) on synthetic and Federated EMNIST datasets. And, we additionally evaluate our method on language model and text datasets. |
| Researcher Affiliation | Collaboration | Dun Zeng EMAIL University of Electronic Science and Technology of China Peng Cheng Lab Zenglin Xu EMAIL Fudan University Shanghai Academy of AI for Science Yu Pan EMAIL Harbin Institute of Technology, Shenzhen Xu Luo EMAIL University of Electronic Science and Technology of China Qifan Wang EMAIL Meta AI Xiaoying Tang EMAIL The Chinese University of Hong Kong, Shenzhen |
| Pseudocode | Yes | Algorithm 1 Fed Avg with Unbiased Client Sampler Require: Client set S, where |S| = N, client weights λ, times T, local steps R 1: Initialize sample distribution p0 and model x0 2: for time t in [T] do 3: Server runs sampling procedure to create St pt ... Algorithm 2 K-Vib Sampler Require: N, K, T, γ, and θ. Initialize client feedback storage ω(i) = 0 for all i [N]. for time t in [T] do ... |
| Open Source Code | No | The experiment implementations are supported by Fed Lab framework (Zeng et al., 2023b). For the case that N > T in FEMNIST tasks, we set θ = 0.3 following the official source code3. (footnote 3: https://github.com/zalanborsos/online-variance-reduction) |
| Open Datasets | Yes | Our experiment evaluation aligns with previous works (Li et al., 2020b; Chen et al., 2020) on synthetic and Federated EMNIST datasets. We evaluate the proposed sampler on the Federated EMNIST (FEMNIST) following Chen et al., 2020 for image classification. We evaluate the efficacy of the K-Vib sampler on two large sizes of models and datasets, including a fine-tuning task on AGNews (Zhang et al., 2015) and a pre-training task on CCNews (Mackenzie et al., 2020). |
| Dataset Splits | No | We generate N = 100 clients of each has a synthetic dataset, where the size of each dataset follows the power law. FEMNIST v1 (2,231 clients in total, 10% clients hold 82% training images), FEMNIST v2 (1,231 clients in total, 20% client hold 90% training images) and FEMNIST v3 (462 clients in total, 50% client hold 98% training images). For both tasks, we partition the datasets into N = 1, 000 clients with three different levels of heavy long tails (Charles et al., 2024). |
| Hardware Specification | No | The paper does not explicitly describe the hardware used for running its experiments. It mentions using 'Fed Lab framework' but provides no specific details on GPUs, CPUs, or other hardware components. |
| Software Dependencies | No | We use vanilla SGD optimizers for client-side optimization. Distill Bert (Sanh et al., 2019) on the AGNews task. GPT2 model called Pythia-70M (Biderman et al., 2023) on CCNews. |
| Experiment Setup | Yes | We set local learning rate ηl = 0.02, local epoch 1, and batch size 64. We set ηg = 1 on the server and tune ηl for different tasks. For K-Vib sampler, we set θ = ( N T K ) 1/3 , which aligns with Theorem 5.2. Then, we guarantee the stability of designed probability via setting γ G2 N Kθ. We set batch size 20, local epochs 3, ηl = 0.01, and K = 111, 62, 23 as 5% of total clients. We set communication round T = 300 and budget K = 25. We set local learning rate 1e 4, batch size 16, and epoch 1 for the local SGD optimizer. |