Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation
Authors: Yi-Chen Li, Fuxiang Zhang, Wenjie Qiu, Lei Yuan, Chengxing Jia, Zongzhang Zhang, Yang Yu, Bo An
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments based on the Llama-3.1 model on the DSP dataset and HH-RLHF dataset illustrate the superior effectiveness of QAdapter on both retaining existing knowledge and learning new preferences. Our code is available at https://github.com/LAMDA-RL/Q-Adapter. |
| Researcher Affiliation | Collaboration | 1 National Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China, 2 School of Artificial Intelligence, Nanjing University, Nanjing, China, 3 Nanyang Technological University, Singapore, 4 Polixir Technologies, Nanjing, China, 5 Skywork AI, Singapore EMAIL, EMAIL, EMAIL, EMAIL, EMAIL |
| Pseudocode | Yes | Algorithm 1 Q-Adapter |
| Open Source Code | Yes | Our code is available at https://github.com/LAMDA-RL/Q-Adapter. |
| Open Datasets | Yes | Experiments based on the Llama-3.1 model on the DSP dataset and HH-RLHF dataset illustrate the superior effectiveness of QAdapter on both retaining existing knowledge and learning new preferences. |
| Dataset Splits | No | We choose the Domain Specific Preference (DSP) (Cheng et al., 2023) dataset that contains preference data from four domains including academy, business, entertainment and literature. For the second one, we choose the HH-RLHF (Bai et al., 2022) dataset that has two folds of preference data: helpful and harmless. The helpful data will first be used to post-train a Llama-3.1 model. Then we will use the harmless data to further customize it. |
| Hardware Specification | Yes | We train QAdapter in one single machine with 4x NVIDIA Ge Force RTX 4090 GPUs. |
| Software Dependencies | No | We adopt the peft library (Mangrulkar et al., 2022) to modify the linear layers of the model with a rank of 8. We adopt the Hugging Face trainer to control the training procedure with a rewritten loss function. We use the TRL (von Werra et al., 2020) implementation of the DPO trainer to train our DPO model with Lo RA. |
| Experiment Setup | Yes | In Table 3, we list the basic hyper-parameters of Q-Adapter. Hyper-parameter Value α 1.0 α0 0.01 β 0.1 γ 1.0 optimizer Adam W learning rate 3e-4 batch size 512 max sequence length T 512 |