A Quantum Circuit-Based Compression Perspective for Parameter-Efficient Learning
Authors: Chen-Yu Liu, Chao-Han Huck Yang, Hsi-Sheng Goan, Min-Hsiu Hsieh
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our objective is to assess the effectiveness of QPA, as outlined in Sec. 3, to determine whether the proposed QPA can effectively reduce the number of parameters while maintaining or surpassing the performance of existing PEFT methods. This evaluation is based on one hypothesis that the high-dimensional Hilbert space enables efficient representation for adaptation. The experiment is conducted using quantum circuit simulation via Py Torch and Torch Quantum (Wang et al., 2022). At this stage, noise effects on the quantum system are ignored, and the quantum state amplitudes (probabilities) are obtained exactly. A discussion on the impact of finite measurement shots and noise is provided in Appendix G. We assess the text generation perplexity of Gemma-2 (2B) and GPT-2 (80M), fine-tuned on the Wiki Text-2 dataset, using several well-known PEFT methods, including Lo RA, Do RA, PT, and Feed-Forward Adapter (FFA). |
| Researcher Affiliation | Collaboration | Chen-Yu Liu1,2 Chao-Han Huck Yang3 Hsi-Sheng Goan1,4,5,6 Min-Hsiu Hsieh2 1 Graduate Institute of Applied Physics, National Taiwan University, Taipei, Taiwan 2 Hon Hai (Foxconn) Research Institute, Taipei, Taiwan 3 Georgia Institute of Technology, USA 4 Department of Physics and Center for Theoretical Physics, National Taiwan University, Taipei, Taiwan 5 Center for Quantum Science and Engineering, National Taiwan University, Taipei, Taiwan 6 Physics Division, National Center for Theoretical Sciences, Taipei, Taiwan {d10245003@g,goan@phys}.ntu.edu.tw, EMAIL, EMAIL |
| Pseudocode | No | The paper includes an "IMPLEMENTATION DETAIL" section in Appendix J with Python code snippets demonstrating how QPA is applied. However, it does not contain a dedicated section or figure explicitly labeled "Pseudocode" or "Algorithm", nor does it present structured steps in a pseudocode-like format separate from actual code. |
| Open Source Code | No | The paper includes an "IMPLEMENTATION DETAIL" section in Appendix J with Python code snippets. However, it does not contain an explicit statement indicating that the source code for the methodology is released or available, nor does it provide a direct link to a code repository. |
| Open Datasets | Yes | We assess the text generation perplexity of Gemma-2 (2B) and GPT-2 (80M), fine-tuned on the Wiki Text-2 dataset, using several well-known PEFT methods, including Lo RA, Do RA, PT, and Feed-Forward Adapter (FFA). ... we extend our analysis to showcase the broader applicability of QPA across additional datasets. In this section, we report results of QPA applied to Lo RA, Prefix Tuning (PT), and Feed-Forward Adapter (FFA) on the Penn Treebank dataset. |
| Dataset Splits | No | The paper mentions using the "Wiki Text-2 dataset" and the "Penn Treebank dataset" for fine-tuning and assessing testing perplexity. However, it does not explicitly provide details regarding the train/validation/test splits (e.g., percentages, absolute counts, or references to specific predefined splits) used for these datasets. |
| Hardware Specification | Yes | All experiments were conducted on NVIDIA V100S and NVIDIA H100 GPUs. |
| Software Dependencies | No | The experiment is conducted using quantum circuit simulation via Py Torch and Torch Quantum (Wang et al., 2022). ... This implementation demonstrates how QPA is applied to generate low-rank matrices using quantum parameter genreation, which can be seamlessly integrated into LLMs to improve parameter efficiency. |
| Experiment Setup | Yes | C TRAINING HYPERPARAMETER CONFIGURATION In this section, we provide the training hyperparameter configuration used for the results presented in the main text. Notably, α represents the scaling factor in the low-rank adaptation methods. All experiments were conducted on NVIDIA V100S and NVIDIA H100 GPUs. Table 4: Hyperparameter configurations of Lo RA and QPA Lo RA for fine-tuning GPT-2 and Gemma-2 with Wiki Text-2 dataset. Table 5: Hyperparameter configurations of Do RA and QPA Do RA for fine-tuning GPT-2 and Gemma-2 with Wiki Text-2 dataset. Table 6: Hyperparameter configurations of PT and QPA PT for fine-tuning GPT-2 and Gemma-2 with Wiki Text-2 dataset. Table 7: Hyperparameter configurations of FFA and QPA FFA for fine-tuning GPT-2 and Gemma-2 with Wiki Text-2 dataset. |