ComLoRA: A Competitive Learning Approach for Enhancing LoRA
Authors: Qiushi Huang, Tom Ko, Lilian Tang, Yu Zhang
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments demonstrate the superiority of Com Lo RA over Lo RA, without increasing inference overhead. |
| Researcher Affiliation | Collaboration | 1Southern University of Science and Technology, 2University of Surrey, 3Byte Dance |
| Pseudocode | No | The paper describes the methodology using natural language and mathematical equations in Section 3, but does not include any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | The Com Lo RA s code is available at https://github.com/hqsiswiliam/comlora. |
| Open Datasets | Yes | We evaluate our models using a comprehensive commonsense reasoning dataset, which includes eight sub-tasks: Bool Q (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), Hella Swag (Zellers et al., 2019), Wino Grande (Sakaguchi et al., 2021), ARC-c (Clark et al., 2018), ARC-e (Clark et al., 2018), and OBQA (Mihaylov et al., 2018). MMLU. The MMLU (Massive Multitask Language Understanding) benchmark (Hendrycks et al., 2021b;a)... Personalized Conversation Task. We assess model performance on personalized conversational understanding using the CONVAI2 dataset (Dinan et al., 2019; Zhang et al., 2018). |
| Dataset Splits | Yes | We aggregate the training sets from all sub-tasks into a single corpus of 170,420 entries, from which we randomly select 120 entries for validation to identify the optimal model. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for running the experiments. It only mentions the LLa MA-3-8B model as the base model. |
| Software Dependencies | No | The paper mentions using the Adam W optimizer and the LLa MA-3-8B model, but does not provide specific version numbers for any key software components or libraries. |
| Experiment Setup | Yes | The Adam W optimizer (Loshchilov & Hutter, 2019) is used to train Lo RA and Com Lo RA. The learning rates for both Lo RA and Com Lo RA methods are selected from [1e-3, 1e-4], and finetuning is conducted for 3 epochs for all the tasks. Each experiment runs with 5 different seeds, and the average results are reported. |