Robust and Efficient Quantization-aware Training via Coreset Selection
Authors: Xijie Huang, Zechun Liu, Shih-Yang Liu, Kwang-Ting Cheng
TMLR 2024 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate our method on various networks (Res Net-18, Mobile Net V2, Retina Net), datasets(CIFAR-10, CIFAR-100, Image Net-1K, COCO), and under different quantization settings. Specifically, our method can achieve an accuracy of 68.39% of 4-bit quantized Res Net-18 on the Image Net-1K dataset with only a 10% subset, which has an absolute gain of 4.24% compared to the baseline. Our method can also improve the robustness of QAT by removing noisy samples in the training set. |
| Researcher Affiliation | Collaboration | 1Hong Kong University of Science and Technology (HKUST), 2Meta Reality Lab EMAIL, EMAIL, EMAIL, EMAIL |
| Pseudocode | Yes | The pseudo-code for our ACS algorithm is shown in Alg. 1. |
| Open Source Code | No | The paper does not explicitly state that source code for the methodology is openly available, nor does it provide a link to a code repository. |
| Open Datasets | Yes | The efficiency experiments are conducted on CIFAR-100 (Krizhevsky et al., 2009) and Image Net-1K dataset (Deng et al., 2009). We evaluate Mobile Net V2 (Howard et al., 2017) on CIFAR-100 and evaluate Res Net-18 (He et al., 2016) on the Image Net-1K dataset. We further provide experimental results of quantized Retina Net Lin et al. (2017) on MS COCO object detection benchmark (Lin et al., 2014). |
| Dataset Splits | Yes | For 2-bit weights-only quantization of Mobile Net V2 on the CIFAR-100 dataset, QAT based on our ACS can achieve a mean accuracy of 67.19% with only 50% training data used for training per epoch. For 4-bit quantization of Res Net-18 on the Image Net-1K dataset, our ACS can achieve top-1 accuracy of 68.39% compared to the 64.15% of random selection when only 10% training data is selected for the training of every epoch. We investigate two coreset fractions: 10% and 50% and use R=10. When full data are selected (S = 100%), the mean accuracy and standard deviation is 68.1 0.9%. When full data are selected (S = 100%), the accuracy is 72.46%. |
| Hardware Specification | Yes | All the experiments were carried out on 2 NVIDIA RTX 3090 GPUs. |
| Software Dependencies | No | The paper mentions using SGD and Adam optimizers, and the LSQ+ quantization method, but it does not specify version numbers for any software libraries or dependencies (e.g., PyTorch, TensorFlow, CUDA). |
| Experiment Setup | Yes | For Mobile Net V2, we train the network for 200 epochs using a learning rate of 0.01, weight decay of 5e-4, batch size of 512, R = 20, and SGD optimizer. For Res Net-18 on Image Net-1K, we train the network for 120 epochs using a learning rate of 1.25e-3, no weight decay, batch size of 512, R = 10, and Adam optimizer. |