Zero-Shot Adaptation of Parameter-Efficient Fine-Tuning in Diffusion Models
Authors: Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse, Fatih Porikli
ICML 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This section describes our experiments to evaluate the effectiveness of Pro Lo RA in transferring a Lo RA from a source to a target diffusion model. We first train the Lo RA from scratch for a specific task on both source and target models and then compare the performance of the Lo RA trained on the target model with the one transferred from the source model using Pro Lo RA. We analyze and quantify Pro Lo RA through text-to-image generation experiments in the following sections, with additional text-generation experiments presented in Appendix E. Metrics: To quantify the quality of images generated by Lo RA and its transferred version using Pro Lo RA, we report the DINOv2 (Oquab et al., 2024), HPSv2.1 (Wu et al., 2023), and LPIPS (Zhang et al., 2018) diversity scores, as well as CSD-MMD. 5.3. Ablation Studies |
| Researcher Affiliation | Industry | 1Qualcomm AI Research, San Diego, CA 92121, USA. Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. Correspondence to: Farzad Farhadzadeh <EMAIL>. |
| Pseudocode | No | The paper describes methods using mathematical equations (e.g., Equations 1, 2, 3) and prose, but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not contain any explicit statements about releasing source code, links to repositories, or mentions of code in supplementary materials. |
| Open Datasets | Yes | Datasets: For style transfer, we are using datasets from public domains, such as Blue Fire, Origami Styles, and Paintings. We follow the same setup as described in (Borse et al., 2024; Farhadzadeh et al., 2025). For concept adapter, we use Dream Booth dataset (Ruiz et al., 2022). For transferring acceleration adapter we only use the off-the-shelf LCM-Lo RA (Luo et al., 2023b). |
| Dataset Splits | No | The paper mentions datasets with their sizes (e.g., "Blue Fire (900 images)", "Paintings (630 images)", "Origami (900 images)"), but it does not specify explicit training, validation, or test dataset splits (e.g., percentages or sample counts). |
| Hardware Specification | Yes | Table 11. Evaluation of Lo RA trained from scratch on SDXL versus training-free transferred Pro Lo RA from SSD-1B into SDXL and X-adapter from SD-v1.5 to SDXL using Blue Fire dataset. Wall clock inference time is measured on A100 GPU. Table 13. Wall clock time comparison when different adapters are trained on the source model SDXL and transferred to the target model SSD-1B. Measurements are done on 1 A100 GPU |
| Software Dependencies | No | The paper does not specify any software dependencies with version numbers. |
| Experiment Setup | Yes | Table 1. Comparison of text-to-image generation using Lo RAs trained from scratch on target diffusion models versus training-free transfer using Pro Lo RA. Lo RA rank is 32 for all cases. 5.2. Performance of Lo RA Transfer: To enable training-free adapter transfer between source and target base models, we first identify correlated modules using equation 1. Following (Farhadzadeh et al., 2025), a threshold of 0.8 is applied to select the most relevant modules. The source Lo RA is then projected onto its corresponding target module using equation 2. Table 9. Pro Lo RA performance on transferring style Do RA from SDXL to SSD-1B. Do RA rank is 8. Table 10. Pro Lo RA performance on transferring style Fou RA from SD-v1.5 to RV3.0. Fou RA rank is 64. |