CoDeC: Communication-Efficient Decentralized Continual Learning
Authors: Sakshi Choudhary, Sai Aparna Aketi, Gobinda Saha, Kaushik Roy
TMLR 2024 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide empirical evidence by performing experiments over various standard image-classification datasets, networks, graph sizes, and topologies. We also extend our analysis by designing and evaluating a decentralized continual learning benchmark Med MNIST-5 using biomedical image-classification datasets from Med MNIST-v2 (Yang et al., 2021). This imitates a practical real-world application where multiple healthcare organizations aim to learn a global generalized model without sharing the locally accessible patients data. |
| Researcher Affiliation | Academia | Sakshi Choudhary EMAIL Department of Electrical and Computer Engineering Purdue University; Sai Aparna Aketi EMAIL Department of Electrical and Computer Engineering Purdue University; Gobinda Saha EMAIL Department of Electrical and Computer Engineering Purdue University; Kaushik Roy EMAIL Department of Electrical and Computer Engineering Purdue University |
| Pseudocode | Yes | Algorithm 1 Communication-Efficient Decentralized Continual Learning (Co De C); Algorithm 2 Decentralized Elastic Weight Consolidation (D-EWC); Algorithm 3 Decentralized Synaptic Intelligence (D-SI) |
| Open Source Code | Yes | 1The PyTorch implementation can be found at https://github.com/Sakshi09Ch/Co De C |
| Open Datasets | Yes | We evaluate Co De C on three well-known continual learning benchmark datasets: 10-Split CIFAR-100 (Krizhevsky, 2009), 20-Split Mini Image Net (Vinyals et al., 2016) and a sequence of 5-Datasets (Ebrahimi et al., 2020). 10-Split CIFAR-100 is constructed by splitting CIFAR-100 into 10 tasks, where each task comprises of 10 classes. The sequence of 5-Datasets includes CIFAR-10, MNIST, SVHN (Netzer et al., 2011), not MNIST (Bulatov, 2011) and Fashion MNIST (Xiao et al., 2017). We design Med MNIST-5, a biomedical decentralized continual learning benchmark based on the datasets in Med MNIST-v2 (Yang et al., 2021). |
| Dataset Splits | Yes | For each task, the data distribution is IID across the agents. [...] For instance, for a graph size of 4 agents, each agent has 5000/4 = 1250 training samples for a particular task in Split CIFAR-100. [...] Table 8: Dataset Statistics for Split CIFAR-100 and Split-mini Image Net; Split CIFAR-100: # Training samples/tasks 5,000, # Test samples/tasks 1,000; Split mini Image Net: # Training samples/tasks 2,500, # Test samples/tasks 500 |
| Hardware Specification | Yes | We perform our experiments on a single machine with 4 NVIDIA GeForce GTX 1080 Ti GPUs. |
| Software Dependencies | No | The PyTorch implementation can be found at https://github.com/Sakshi09Ch/Co De C. The paper mentions PyTorch but does not provide a specific version number. |
| Experiment Setup | Yes | All our experiments were run for three randomly chosen seeds. We decay the learning rate by a factor of 10 after 50% and 75% of the training, unless mentioned otherwise. For Split CIFAR-100, we use a mini-batch size of 22 per agent, and we run all our experiments for a total of 100 epochs for each task. For Split Mini Image Net, we use a mini-batch size of 10 per agent and 10 epochs for each task. For 5-Datasets and Med MNIST-5, we use a mini-batch size of 32 per agent, and 50 epochs for each task. We list additional hyperparameters in Table 10. |