Multi-concept Model Immunization through Differentiable Model Merging
Authors: Amber Yijia Zheng, Raymond A. Yeh
AAAI 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In our experiments, we demonstrate the effectiveness of multi-concept immunization by generalizing prior work s experiment setup of re-learning and personalization adaptation to multiple concepts. Empirically, we experiment with several adaptation methods, including, Textual Inversion (Gal et al. 2023), Dream Booth (Ruiz et al. 2023), Lo RA (Hu et al. 2022), and Custom Diffusion (Kumari et al. 2023b) over two applications: (a) restoring erased concepts such as artistic styles or object categories, and (b) learning personalized concepts. We found that MIMA successfully immunizes a model against multiple malicious concepts and outperforms IMMA-inspired baselines. Our contributions are summarized as follows: We conduct experiments over two tasks and four adaptation methods demonstrating the efficacy of MIMA. |
| Researcher Affiliation | Academia | Amber Yijia Zheng, Raymond A. Yeh Department of Computer Science, Purdue University EMAIL |
| Pseudocode | Yes | Algorithm 1: MIMA (Our method) Input: pre-trained model θp, images X = n X[n] concepts C = {c[n]}N n=1, learning rates α and β, modified parameters set Sl and Su in lower and upper tasks, loss function L, Merge layer, training epochs K Output: Immunized model θ 1: Initialize θ0 = θp 2: for k = 1 to K do 3: Sample batches of each concept {(xu [n], c[n])}N n=1 from X and C 4: # Solve the lower-level tasks for one step. 5: for n = 1 to N do 6: Sample batch xl [n] from X[n] 7: θ [n], Sl θk 1 Sl α θL(xl [n], c[n]; θk 1) 8: end for 9: # Each θ [n] is a function of θk 1 10: θ Merge({θ [1], . . . , θ [N]}) 11: θk Su θk 1 Su + β θL(xu [n], c[n]; θ ) 12: end for 13: θ θK 14: return θ |
| Open Source Code | No | Project page https://www.amberyzheng.com/mima |
| Open Datasets | Yes | Following IMMA (Zheng and Yeh 2024), we perform experiments on eight artistic styles and ten classes spanning various categories from a subset of Image Net (Deng et al. 2009). We conduct the experiments on thirteen unique concepts from Kumari et al. (2023b), including pets, furniture, scenes, decor items, etc. |
| Dataset Splits | Yes | For immunization, we generate 20 images for each target concept from Stable Diffusion V1-4 (SD V1-4) with the prompts of the target artistic styles and objects. Specpficially, the prompts are an artwork of {artist name} and a photo of {object name} rescptively. As in IMMA, we consider the risk of re-learning the concept using the efficient adaptation method of Lo RA (Hu et al. 2022). We generate another 20 images to be used as the training images for Lo RA. To maintain the model capability of being finetuned to learn other concepts, we generate 200 regularization images for Merge using either the prompt artwork or object for each of the corresponding settings. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for the experiments, such as GPU models or CPU specifications. |
| Software Dependencies | No | The paper mentions several adaptation methods and models like Stable Diffusion V1-4, Lo RA, Textual Inversion, Dream Booth, and Custom Diffusion. However, it does not specify version numbers for these or any other software libraries or dependencies used in the implementation. |
| Experiment Setup | Yes | All the numbers are reported at the 400th step of Lo RA with a batch size of four. All the results in the tables are reported at the 40th step for all adaptations. |