What’s Left After Distillation? How Knowledge Transfer Impacts Fairness and Bias
Authors: Aida Mohammadshahi, Yani Ioannou
TMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Using two common fairness metrics, Demographic Parity Difference (DPD) and Equalized Odds Difference (EOD) on models trained with the Celeb A, Trifeature, and Hate Xplain datasets, our results suggest that increasing the distillation temperature improves the distilled student model s fairness, and the distilled student fairness can even surpass the fairness of the teacher model at high temperatures. Additionally, we examine individual fairness, ensuring similar instances receive similar predictions. Our results confirm that higher temperatures also improve the distilled student model s individual fairness. |
| Researcher Affiliation | Academia | Aida Mohammadshahi EMAIL Yani Ioannou EMAIL Department of Electrical and Software Engineering Schulich School of Engineering, University of Calgary Calgary, AB, Canada |
| Pseudocode | Yes | Algorithm 1 Fairness Metrics Calculation for Trifeature Dataset |
| Open Source Code | No | The paper does not explicitly state that source code for the methodology is provided, nor does it include a direct link to a code repository. |
| Open Datasets | Yes | We use the SVHN (Netzer et al., 2011), CIFAR-10 (Krizhevsky, 2009), CIFAR100 (Krizhevsky, 2009), Tiny Image Net (Le and Yang, 2015), and Image Net (ILSVRC 2012) (Deng et al., 2009) datasets... One of the datasets we use is Celeb Faces Attributes dataset (Celeb A) (Liu et al., 2015)... We also apply our analysis to the Trifeature (Hermann and Lampinen, 2020) synthetic dataset... In addition, we extend our analysis to the Hate Xplain text dataset (Mathew et al., 2021)... |
| Dataset Splits | Yes | To mitigate the effects of this imbalance w.r.t. smiling/not smiling on potential bias and enhance prediction accuracy, we randomly undersample the over-represented class, resulting in the distribution of training data shown in Table 5 across our label attribute and demographic groups after balancing the training data. ... Tiny Image Net is a medium-scale image classification datasets categorized into 200 classes, containing 100,000 training and 10,000 test images from the Image Net dataset, but downsized to 64 64 pixels. Each class has 500 training images, 50 validation images and 50 test images. |
| Hardware Specification | Yes | We trained Le Net-5, Res Net-56, Res Net-20, and Dense Net models using an NVIDIA RTX A4000 GPU with 16 GB of RAM. ... For training the Res Net-50 and Res Net-18 models on Tiny Image Net and Image Net, we utilized four NVIDIA Ge Force RTX 3090 GPUs, each with 24 GB of RAM. ... For the Vi T-Base and Tiny Vi T models, we employed four NVIDIA A100 GPUs with 40 GB of RAM. |
| Software Dependencies | No | The paper mentions using SGD optimizer and various model architectures (Res Net, Dense Net, Le Net-5, BERT-Base, Distil BERT) but does not specify any software libraries or frameworks with version numbers (e.g., PyTorch, TensorFlow, scikit-learn versions). |
| Experiment Setup | Yes | We use α = 0.8 for all experiments to have consistency. ... We utilize data augmentation of random crop and horizontal flip, and adopt SGD optimizer with weight decay of 0.001, and decreasing learning rate schedules with a starting learning rate of 0.1 and a factor of 10 drop-off at epochs 30, 60, and 90 to train Le Net-5, Res Net and Dense Net models for 100 epochs. ... We train on SVHN, CIFAR-10, and CIFAR-100 with a batch size of 128 and on Tiny Image Net and Image Net with a batch size of 512. ... We use the batch size of 256 for the Celeb A dataset and batch size of 64 for Trifeature dataset. |