RepLoRA: Reparameterizing Low-rank Adaptation via the Perspective of Mixture of Experts
Authors: Tuan Truong, Chau Nguyen, Huy Nguyen, Minh Le, Trung Le, Nhat Ho
ICML 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments across multiple domains demonstrate that Rep Lo RA consistently outperforms vanilla Lo RA. Notably, with limited data, Rep Lo RA surpasses Lo RA by a margin of up to 40.0% and achieves Lo RA s performance with only 30.0% of the training data, highlighting both the theoretical and empirical robustness of our PEFT method. |
| Researcher Affiliation | Collaboration | 1Qualcomm AI Research, Qualcomm Vietnam Company Limited 2Work was completed while an employee at Qualcomm 3The University of Texas at Austin, USA 4Monash University, Australia. Correspondence to: Tuan Truong <EMAIL>. |
| Pseudocode | No | The paper describes the methods in prose and mathematical formulations, particularly in Sections 4, 5, and 6, but does not include any explicitly labeled 'Pseudocode' or 'Algorithm' blocks or structured code-like procedures. |
| Open Source Code | No | The paper does not contain any explicit statements about the release of source code for the methodology described, nor does it provide a link to a code repository. |
| Open Datasets | Yes | We extend our evaluation of Rep Lo RA to the image domain and fine-tune the Vi TB/16 architecture (Dosovitskiy, 2021), pre-trained on the Image Net-21K dataset (Deng et al., 2009), on two challenging benchmarks: the VTAB-1K dataset suite (Zhai et al., 2019) and the FGVC dataset collection (Jia et al., 2022). |
| Dataset Splits | Yes | The VTAB-1K benchmark is a diverse suite of 19 datasets spanning various domains designed to test image classification and prediction capabilities. These datasets cover a wide range of tasks involving distinct semantics and object categories, organized into Natural, Specialized, and Structured domains. Each dataset includes 1,000 training examples, with an official 80/20 train-validation split, making it a rigorous test for generalization across different domains. |
| Hardware Specification | No | The paper describes various experimental setups and models used (e.g., LLa MA-7B/13B, Vi TB/16, Video Swin Transformer), but does not specify the underlying hardware (e.g., GPU models, CPU types) used for conducting these experiments. |
| Software Dependencies | No | The paper mentions models and optimizers like 'LLa MA-7B/13B', 'Vi TB/16', 'Video Swin Transformer', 'VL-BART', and 'Adam W', but does not provide specific version numbers for any software libraries or dependencies used (e.g., PyTorch, TensorFlow versions). |
| Experiment Setup | Yes | For the vision tasks, we use grid search to tune the learning rate in the range of {0.001, 0.005, 0.01, 0.05, 0.1}, and the weight decay in the range of {0.0001, 0.0005, 0.001, 0.01, 0.1}. Other hyperparameters are reported in the tables below: Table 7. Hyperparameter configurations of Rep Lo RA for Vi T-B/16 on the vision tasks. Table 8. Hyperparameter configurations of Rep Lo RA for LLa MA-7B/13B on the commonsense reasoning tasks. Table 9. Hyperparameter configurations of Rep Lo RA for VL-BART on the Image/Video-Text Understanding tasks. |