A Snapshot of the Frontiers of Client Selection in Federated Learning
Authors: Gergely Dániel Németh, Miguel Angel Lozano, Novi Quadrianto, Nuria M Oliver
TMLR 2022 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Finally, we believe that there is an opportunity to improve the empirical evaluation of existing and newly proposed incentive mechanisms for FL. There are 3 key components depicted in Figure 3 that we consider necessary to include in such an evaluation: first, a clear explanation as to why the proposed scoring satisfies the requirements of being an incentive mechanism; second, a performance and efficiency analysis on benchmark datasets; and third, a systematic comparison with other state-of-the-art client selection methods. |
| Researcher Affiliation | Academia | Gergely Dániel Németh EMAIL ELLIS Alicante Miguel Ángel Lozano EMAIL University of Alicante Novi Quadrianto EMAIL University of Sussex Nuria Oliver EMAIL ELLIS Alicante |
| Pseudocode | No | The paper does not contain any clearly labeled pseudocode or algorithm blocks. Figure 1 shows a diagram of a federated learning training round, but it is a flow chart, not a pseudocode algorithm. |
| Open Source Code | No | We would also encourage the community to leverage rapidly maturing Federated Learning frameworks such as, TFF2, Py Shift3, and Flower4 to ease the reproducibility of the results and enable the integration of novel client selection strategies into other parts of the federated pipeline, such as FL optimizers, or differential privacy frameworks. The paper recommends using existing open-source frameworks but does not state that its own methodology's code is released. |
| Open Datasets | Yes | Table 2: Commonly used datasets in client selection experiments in FL. Clients are either split by classes, or more naturally along a feature of the data for example, by writers of social media posts. Dataset Split Type Methods MNIST (Le Cun, 1998) Classes Image S-Fed Avg, FAVOR, CI-MR, FMore, CBIM, Fashion MNIST (Xiao et al., 2017) Classes Image pow-d, FAVOR, Fed CS, q-FFL, AFLM, FMore, DDa BA EMNIST (Cohen et al., 2017) Classes Image FD FEMNIST (Caldas et al., 2018a) Writers Image k-FED, DDa BA CIFAR10 (Krizhevsky, 2009) Classes Image FL-CIR, FAVOR, Fed CS, FD, FMore, DDa BA Shakespeare (Caldas et al., 2018a) Roles Text k-FED, q-FFL Sent140 (Go et al., 2009) Writers Text q-FFL Reddit (multiple variations) Writers Text q-FFL, Oort, AFLG UCI Adult Dataset (Blake, 1998) Ph D or not Tabular q-FFL, AFLM London Low Carbon (Marantes & Openshaw, 2012; Schofield et al., 2015) |
| Dataset Splits | No | Note that creating an artificially distributed dataset from an originally centralized one requires designer choices to be made. In the current client selection literature even if two papers use the same dataset, the results are in most cases not comparable due to different choices, such as a different distribution of the data. Moreover, the models that are used (e.g. deep neural networks) vary in different evaluations, adding another layer of difficulty to make comparisons. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used to conduct experiments. As a survey paper, it describes methodologies and findings from other works but does not present its own experimental results with hardware specifications. |
| Software Dependencies | No | We would also encourage the community to leverage rapidly maturing Federated Learning frameworks such as, TFF2, Py Shift3, and Flower4 to ease the reproducibility of the results and enable the integration of novel client selection strategies into other parts of the federated pipeline, such as FL optimizers, or differential privacy frameworks. The paper mentions several Federated Learning frameworks (TFF, Py Shift, Flower) but does not provide specific version numbers for any software used in the context of its own work or analysis. |
| Experiment Setup | No | The paper is a survey and taxonomy of client selection methods in Federated Learning. It describes experimental setups and hyperparameters used in other research papers but does not present its own experimental setup with specific hyperparameters or training configurations. |