Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate
Authors: Byung Hyun Lee, Sungjin Lim, Seunggyu Lee, Dong Un Kang, Se Young Chun
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments on the erasure of celebrities, artistic styles, and explicit contents demonstrated that the proposed CPE outperforms prior arts by keeping diverse remaining concepts while deleting the target concepts with robustness against attack prompts. |
| Researcher Affiliation | Academia | Byung Hyun Lee1, , Sungjin Lim2, , Seunggyu Lee1, Dong Un Kang1, Se Young Chun1,2,3, 1Department of ECE, 2IPAI, 3INMC, Seoul National University EMAIL |
| Pseudocode | Yes | Algorithm 1 Framework of training Res AG in CPE |
| Open Source Code | Yes | Code is available at https://github.com/Hyun1A/CPE. |
| Open Datasets | Yes | Extensive experiments on the erasure of celebrities, artistic styles, and explicit contents demonstrated that the proposed CPE outperforms prior arts by keeping diverse remaining concepts while deleting the target concepts with robustness against attack prompts. ... We used CS and FID for COCO-30K, or KID for the other remaining concepts. ... We fine-tuned them on SD v1.4 (Rombach et al., 2022b) ... For remaining concepts, we considered three domains: 100 celebrities and 100 artistic styles from (Lu et al., 2024), 64 characters from Word2Vec by Church (2017). ... We evaluated explicit contents erasure on Inappropriate Image Prompts (I2P) and used the Nude Net detector ... we utilized the top-1 accuracy of the GIPHY Celebrity Detector (GCD) to specifically evaluate the effectiveness of celebrity concept erasure and preservation as an additional metric. |
| Dataset Splits | No | The paper describes how images were generated for evaluation purposes (e.g., "We generated 1250 images using 5 prompt templates with 5 random seeds" for target celebrities, and "25 images using 5 prompt templates with 5 random seeds, resulting in 2,500, 2,500, and 1,600 images for each remaining domain" for other concepts). It also mentions "randomly selected 1,000 captions from COCO dataset" for an ablation study. However, it does not provide explicit training, validation, or test dataset splits in the traditional sense for the primary experiments, nor does it specify how existing datasets like COCO-30K are split for their experimental evaluation beyond being used as "remaining concepts". |
| Hardware Specification | Yes | The training time for each celebrity took 7 minutes on an A6000 GPU, Thus, erasing all 50 celebrities took about 6 A6000 GPU hours. |
| Software Dependencies | No | The paper mentions several libraries, models, and optimizers such as "Adam optimizer (Kingma & Ba, 2015)", "Gensim Word2Vec library (Church, 2017)", "Stable Diffusion (SD) v1.4", "CLIP", and "U-Net". However, it does not provide specific version numbers for any of these software dependencies, which would be necessary for precise replication. |
| Experiment Setup | Yes | For training Res AG, we used Adam optimizer (Kingma & Ba, 2015) with learning rate 3.0 10 4 for initial erasing stage and 3.0 10 5 for the other erasing stages. For training the adversarial embeddings, we used Adam optimizer with learning rate 0.01. We scheduled the learning rate with cosine-with-restart for all cases (Loshchilov & Hutter, 2016). We set 1800 iterations for initial erasing stage and 450 iterations for subsequent erasing stages. For adversarial embeddings learning, 16 adversarial embeddings were trained for 450 iterations for each stage. ... The rank of the shared attention gate s1 was set to 16, and the rank s2 was set to 1. η for erasing loss and λ for attention the attention anchor loss were configured to 0.3 and 1.0 105, respectively. |