Debiased Distillation for Consistency Regularization
Authors: Lu Wang, Liuchi Xu, Xiong Yang, Zhenhua Huang, Jun Cheng
AAAI 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments on the CIFAR-100, Image Net-1K, and Tiny-Image Net datasets validate the superiority of IKD. |
| Researcher Affiliation | Academia | Lu Wang 1, Liuchi Xu 1, Xiong Yang 2, Zhenhua Huang* 3, Jun Cheng* 4,5 1 School of Computer Science and Engineering, Northeastern University, Shenyang, China; 2 Institute of Applied Artificial Intelligence of the Guangdong-Hong Kong-Macao Greater Bay Area, Shenzhen Polytechnic University, Shenzhen, China; 3 School of Computer Science, South China Normal University, Guangzhou, China; 4 Guangdong Provincial Key Laboratory of Robotics and Intelligent System, Shenzhen Institute of Advanced Technology, CAS, China; 5 The Chinese University of Hong Kong, Hong Kong, China 1 {wanglu@mail,xuliuchi@stumail}.neu.edu.cn, 2 EMAIL, 3 EMAIL, 4,5 EMAIL |
| Pseudocode | Yes | Algorithm 1 presents the pseudo-code of the IKD. Please refer to Supplementary Materials for a detailed algorithm description. |
| Open Source Code | Yes | Code https://github.com/yema-web/IKD |
| Open Datasets | Yes | CIFAR-100 (Krizhevsky, Hinton et al. 2009) comprises 100 classes, with each image having a resolution of 32 32 pixels. The CIFAR-100 dataset contains 50k training images and 10k validation images. Image Net-1K (ILSVRC2012) (Deng et al. 2009) is a large-scale dataset comprising 1k classes. The dataset comprises 1.2 million training images and 50k validation images. Tiny-Image Net (Le and Yang 2015) is a subset of the Image Net-1K dataset, consisting of 200 classes, and the image is 64 64 pixels. The training set contains 100k images, and the validation set contains 10k images. |
| Dataset Splits | Yes | CIFAR-100 (Krizhevsky, Hinton et al. 2009) ... The CIFAR-100 dataset contains 50k training images and 10k validation images. Image Net-1K (ILSVRC2012) (Deng et al. 2009) ... The dataset comprises 1.2 million training images and 50k validation images. Tiny-Image Net (Le and Yang 2015) ... The training set contains 100k images, and the validation set contains 10k images. |
| Hardware Specification | No | Implementations details are provided in Supplementary Materials due to page constraints. |
| Software Dependencies | No | Implementations details are provided in Supplementary Materials due to page constraints. |
| Experiment Setup | Yes | We conduct extensive ablation studies on the intra-class temperature T and the weight of IKD λavg. ... as shown in Fig. 4 (a), choosing a temperature parameter of 1.0 is appropriate for suppressing noise. ... Fig. 4 (b) demonstrates that for DKD+IKD, the optimal performance is achieved with a λavg of 6.5, whereas for NKD+IKD, a λavg of 6.0 yields the best results. For other parameter details, refer to Supplementary Materials. |