In-Context Meta LoRA Generation
Authors: Yihua Shao, Minxi Yan, Yang Liu, Siyu Chen, Wenjie Chen, Xinwei Long, Ziyang Yan, Lei Li, Chenyu Zhang, Nicu Sebe, Hao Tang, Yan Wang, Hao Zhao, Mengzhu Wang, Jingcai Guo
IJCAI 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate our method for both text and visual tasks on different models. For visual tasks, we select target detection tasks and use the COCO dataset [Lin et al., 2014] to classify subclasses based on the detection task labels for the experiments. For language tasks, we choose The Pile [Gao et al., 2020] as the training corpus and use five different subsets to simulate multi-class training tasks and validate the model on the validation set. The results indicate that CVAE can successfully generate different tasks Lo RA parameters. |
| Researcher Affiliation | Academia | 1Department of COMP/LSGI, The Hong Kong Polytechnic University, Hong Kong SAR 2School of Computer Science, Peking University, China 3Institute for AI Industry Research (AIR), Tsinghua University, China 4Beijing Institute for General Artificial Intelligence, China 5Department of Information Engineering and Computer Science, University of Trento, Italy 6Department of Computer Science, University of Copenhagen, Denmark |
| Pseudocode | No | The paper describes the methodology in prose and uses diagrams (Figure 3) to illustrate the overall framework, but does not include any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | The code is available at https://github.com/Yihua Jerry/ICM-Lo RA |
| Open Datasets | Yes | For visual tasks, we select target detection tasks and use the COCO dataset [Lin et al., 2014] to classify subclasses based on the detection task labels for the experiments. For language tasks, we choose The Pile [Gao et al., 2020] as the training corpus. |
| Dataset Splits | No | For visual tasks, we select target detection tasks for to experiment. We choose the COCO [Lin et al., 2014] dataset and divide it into different subclasses based on the detection task labels. For the language modelling task, we employ The Pile [Gao et al., 2020] as the training corpus. To simulate the multi-category training tasks, we pick five various subsets from the Pile corpus and validate our method on the test sets. |
| Hardware Specification | Yes | All experiments were conducted on a single NVIDIA A800 GPU, with each experiment taking approximately 3 hours to complete. |
| Software Dependencies | No | The CVAE model employs a 12-layer 1D CNN architecture for both the encoder and decoder. The loss function for the CVAE combines the Kullback-Leibler divergence (KLD) and reconstruction loss, with the KLD weight set to 0.005. The task vector is extracted from the last token of the last layer in the CLIP [Radford et al., 2021]. |
| Experiment Setup | Yes | The CVAE model employs a 12-layer 1D CNN architecture for both the encoder and decoder. The loss function for the CVAE combines the Kullback-Leibler divergence (KLD) and reconstruction loss, with the KLD weight set to 0.005. We fine-tuned the model on a specific task using Lo RA (Low-Rank Adaptation) for a total of 150 epochs, saving the Lo RA parameters from the final 50 epochs. ... Subsequently, the CVAE model is trained for 2,000 epochs to ensure robust learning of the latent space. |