Rethinking the Temperature for Federated Heterogeneous Distillation
Authors: Fan Qi, Daxu Shi, Chuokun Xu, Shuai Li, Changsheng Xu
ICML 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Evaluations across diverse benchmarks with varying model/data heterogeneity demonstrate that the Re T-FHD achieves significant accuracy improvements over baseline methods while substantially reducing communication costs compared to existing approaches. Section 5 Experiment 5.1 Experimental Setup Datasets and Heterogeneity Setting Datasets: Cifar10 (Krizhevsky & Hinton, 2009), Cifar100 (Krizhevsky & Hinton, 2009), Tiny Image Net (Chrabaszcz et al., 2017), and Flower102 (Nilsback & Zisserman, 2008). Table 1. Testing accuracy (%) in FL model heterogeneous (a.) and homogeneous (b.) settings. Table 2. Compared to decentralized methods in four datasets. |
| Researcher Affiliation | Academia | 1School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, China 2School of Computer Science and Engineering, Southeast University, Nanjing, China 3Institute of Automation, Chinese Academy of Sciences, Beijing, China. |
| Pseudocode | Yes | Algorithm 1 Algorithm of our Re T-FHD Input: Dataset{D1, D2, . . . , DK}, T T : Maximum communication rounds 1: Initialization: W Worker, V Validator, M Miner, Global logits zg, Blacklist B, Client neighbors set N, Honest server S, Client reward set R 2: for t = 1 T do 3: each k K Randomly{W, V, M} , each k K {W} 4: for each w [W] do 5: w downloads zg t 1 from blockchain do local update with Eq.6 |
| Open Source Code | No | The paper does not contain any explicit statements about releasing source code, nor does it provide a link to a code repository. |
| Open Datasets | Yes | Datasets: Cifar10 (Krizhevsky & Hinton, 2009), Cifar100 (Krizhevsky & Hinton, 2009), Tiny Image Net (Chrabaszcz et al., 2017), and Flower102 (Nilsback & Zisserman, 2008). |
| Dataset Splits | Yes | All datasets are taken as 75% as training set and 25% as test set. Data Heterogeneity: For each dataset, we apply two main types of Non-IID setting (Zhang et al., 2024b): 1) Dirichlet distribution, we set α = {0.1, 0.5}, it s worth noting that α is the concentration parameter and smaller α corresponds to stronger data heterogeneity; 2) Pathological Setting, each client contains a fixed number of sample categories, but each client has a different sample category. |
| Hardware Specification | Yes | All experiments are implemented in Pytorch (i.e. Intel(R) Xeon(R) Platinum 8176 CPU @ 2.10GHz with NVIDIA Ge Force RTX 3090 GPU). |
| Software Dependencies | No | All experiments are implemented in Pytorch (i.e. Intel(R) Xeon(R) Platinum 8176 CPU @ 2.10GHz with NVIDIA Ge Force RTX 3090 GPU). The paper mentions Pytorch but does not provide a specific version number for it or any other key libraries. |
| Experiment Setup | Yes | 5.1 Experimental Setup Datasets and Heterogeneity Setting Datasets: Cifar10 (Krizhevsky & Hinton, 2009), Cifar100 (Krizhevsky & Hinton, 2009), Tiny Image Net (Chrabaszcz et al., 2017), and Flower102 (Nilsback & Zisserman, 2008). All datasets are taken as 75% as training set and 25% as test set. Data Heterogeneity: For each dataset, we apply two main types of Non-IID setting (Zhang et al., 2024b): 1) Dirichlet distribution, we set α = {0.1, 0.5}, it s worth noting that α is the concentration parameter and smaller α corresponds to stronger data heterogeneity; 2) Pathological Setting, each client contains a fixed number of sample categories, but each client has a different sample category. Model Heterogeneity: We adopt the heterogeneous model setup from (Zhang et al., 2021), including Alex Net (Krizhevsky et al., 2017), Shuffle Net V2 (Ma et al., 2018), Res Net18 (He et al., 2016) and Google Net (Szegedy et al., 2015), which are randomly and evenly assigned to clients. For the De L experimental setup, we use only 2-layer CNNs and apply two-stage elastic distillation for homogeneous models (Li et al., 2022a). |