A Closer Look at Machine Unlearning for Large Language Models
Authors: Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, Min Lin
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results across three scenarios, i.e., fictitious unlearning, continual unlearning, and real-world unlearning, demonstrate the effectiveness of our approaches. The code is available at https://github.com/sail-sg/closer-look-LLM-unlearning. |
| Researcher Affiliation | Collaboration | 1University of Science and Technology of China 2Sea AI Lab, Singapore EMAIL; EMAIL |
| Pseudocode | No | The paper describes methods using mathematical formulations and textual explanations but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | The code is available at https://github.com/sail-sg/closer-look-LLM-unlearning. |
| Open Datasets | Yes | Experimentally, we consider the widely adopted TOFU benchmark (Maini et al., 2024; Zhang et al., 2024a; Jia et al., 2024; Ji et al., 2024; Huang et al., 2024; Liu et al., 2024a) for fictitious unlearning, and extend it to the continual unlearning scenario. Besides, we also conduct evaluations in a more realistic real-world unlearning scenario. |
| Dataset Splits | Yes | It has three levels of tasks, namely forget01, forget05 and forget10, to forget 1%, 5%, and 10% of the constructed data. The complement of each forget set serves as the retain set (i.e., neighbor set). |
| Hardware Specification | Yes | All experiments are conducted on two NVIDIA A100 GPUs with 40GB of memory. |
| Software Dependencies | No | The paper mentions using Llama2-chat-7B and Llama-3-8B-Instruct models, Deep Speed with Ze RO3, and Sentence-BERT and NLI models, but does not provide specific version numbers for any of these software components or libraries. |
| Experiment Setup | Yes | We employ the Adam W optimizer with a weight decay of 0.01, a learning rate of 1 10 5, and an effective batch size of 32 for all experiments, consistent with the settings in (Maini et al., 2024; Zhang et al., 2024a). During unlearning, we fine-tune for 5 epochs, using a linear warm-up learning rate in the first epoch and a linearly decaying learning rate in the subsequent epochs. Following the setup in (Maini et al., 2024; Zhang et al., 2024a), we randomly sample up to 300 question-answer pairs from the dataset for evaluation to improve efficiency. The β in NPO and AP is set to 0.1. The parameter α in MG+GD is set to 0.1 in Section 5.1 and is set to 1.0 in Section 5.2. |