PaCA: Partial Connection Adaptation for Efficient Fine-Tuning
Authors: Sunghyeon Woo, Sol Namkung, SunWoo Lee, Inho Jeong, Beomseok Kim, Dongsuk Jeon
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In experiments with various scenarios, Pa CA demonstrates substantial reductions in both training time and memory compared to prior PEFT schemes while maintaining comparable accuracy on NVIDIA A100 GPU (Choquette et al., 2021) and Intel Gaudi2 HPU (Intel Corporation, 2023). We experimentally show that Pa CA effectively reduces memory consumption and improves training speed compared to prior PEFT algorithms across various fine-tuning scenarios on different types of GPUs. |
| Researcher Affiliation | Academia | Sunghyeon Woo, Sol Namkung, Sunwoo Lee, Inho Jeong, Beomseok Kim, Dongsuk Jeon Seoul National University EMAIL |
| Pseudocode | No | The paper describes the algorithm's forward and backward operations using mathematical equations (Eqs. 7-9) but does not present a structured pseudocode block or algorithm steps labeled explicitly as 'Pseudocode' or 'Algorithm'. |
| Open Source Code | Yes | The code is available at https://github.com/Woo Sunghyeon/paca. |
| Open Datasets | Yes | fine-tuning on the MMLU dataset (Hendrycks et al., 2021) and instruction tuning on the Oasst1 dataset (K opf et al., 2023). fine-tuning Vi T-B/16 (Dosovitskiy et al., 2021) and Efficient Net V2-L (Tan & Le, 2021) using various datasets such as CIFAR-10 (Krizhevsky & Hinton, 2009), CIFAR-100 (Krizhevsky & Hinton, 2009), Oxford-IIIT Pets (Parkhi et al., 2012), and Oxford-Flowers 102 (Nilsback & Zisserman, 2008) |
| Dataset Splits | No | The paper mentions using specific datasets like MMLU, Oasst1, CIFAR-10, CIFAR-100, Oxford-IIIT Pets, and Oxford-Flowers 102, which typically have standard splits. However, it does not explicitly provide the percentages or sample counts for training, validation, and test splits within the text or tables. |
| Hardware Specification | Yes | Pa CA demonstrates substantial reductions in both training time and memory compared to prior PEFT schemes while maintaining comparable accuracy on NVIDIA A100 GPU (Choquette et al., 2021) and Intel Gaudi2 HPU (Intel Corporation, 2023). All experiments were conducted on a single NVIDIA A100 GPU. fine-tuning LLa MA3-8B using a single NVIDIA A100 GPU fine-tuning LLa MA3.1-70B on a single NVIDIA A100 GPU |
| Software Dependencies | No | We have implemented Pa CA using Py Torch (Paszke et al., 2019), a widely used deep learning framework, and integrated it into the PEFT library in Huggingface (Wolf et al., 2019) to ensure easy reproducibility. However, specific version numbers for PyTorch or the Huggingface PEFT library are not provided. |
| Experiment Setup | Yes | Table 8: Hyperparameters used for analyzing the number of operations and the average training time per iteration, averaged over 100 iterations, for fine-tuning LLa MA3-8B. Table 9: Hyperparameters when fine-tuning LLa MA2-7B/13B and LLa MA3-8B using PEFT algorithms on the MMLU dataset. Table 10: Hyperparameters used when fine-tuning LLa MA3-8B using PEFT algorithms on the Oasst1 dataset. Table 11: Hyperparameters used when fine-tuning LLa MA3.1-70B using QLo RA and QPa CA on the Oasst1 dataset. Table 12: Hyperparameters used for verifying the maximum sequence length on a single GPU for fine-tuning LLa MA3-8B. Table 13: Hyperparameters for comparing training throughput when increasing batch size on a single GPU for fine-tuning LLa MA3-8B. |