Resolving Lexical Bias in Model Editing
Authors: Hammad Rizwan, Domenic Rosati, Ga Wu, Hassan Sajjad
ICML 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In our empirical study, we show that our method, Projector Editor Networks for Model Editing PENME, achieves state-of-the-art model editing results while being computationally efficient during inference compared to previous methods and adaptable across different architectures. |
| Researcher Affiliation | Academia | 1Department of Computer Science, Dalhousie University, Halifax, Canada. Correspondence to: Hammad Rizwan <EMAIL>. |
| Pseudocode | Yes | Algorithm 1 Data Construction for Projector Network |
| Open Source Code | Yes | We provide the codebase of PENME here: https://github. com/hammadrizwan/PENME.git |
| Open Datasets | Yes | The zs RE dataset (Levy et al., 2017) and the Counterfact dataset (Meng et al., 2022) are the most commonly used model editing datasets. zs RE consists of an edit prompt along with several paraphrased versions of that prompt. Irrelevant prompts are sourced from the NQ dataset (Kwiatkowski et al., 2019)... Specifically, we evaluate using three tasks: sentiment classification using the DAi R-Emotions dataset (Saravia et al., 2018), summarization using the CNN/Daily Mail dataset (Hermann et al., 2015), and natural language inference (NLI) using the RTE dataset (Dagan et al., 2005). |
| Dataset Splits | Yes | We sample 1500 tuples (ei, pi, p i ) of edits denoted ei, paraphrases pi, and their unrelated irrelevant prompts p i from the Counterfact dataset with accompanying input prompts xi and split them into train and test sets of 1000 and 500 samples respectively. |
| Hardware Specification | Yes | Training for all projector networks is conducted on an NVIDIA P100 GPU with 16GB VRAM. A larger VRAM or RAM capacity is only necessary for the initial extraction of layer representations from the pre-trained language models. For the evaluation of approaches from relevant literature, some of which demanded greater computational resources, we employed NVIDIA A100 GPUs with 40GB and 80GB VRAM. |
| Software Dependencies | No | The paper mentions 'Adam optimiser' and 'Easy-Editor library (Wang et al., 2023)' but does not specify version numbers for general software dependencies such as Python, PyTorch/TensorFlow, or CUDA. |
| Experiment Setup | Yes | For training the projector networks, we utilise the Adam optimiser. we experiment with various learning rates 1e1 2, 2e1 2, 3e1 2. we find that a moderate learning rate is required to learn faster while not overfitting, hence we choose 1e1 2, with a learning rate decay rate of 0.01. All projection networks are trained for 200 epochs using a batch size of 8192 and an early stopping patience of 8 epochs. For selecting the margin m in the contrastive learning cost function we ablate on the hyperparameter m for the GPT2-XL model. The table 6 shows the margin m along with the adjustment to τ for balanced results for generalization and locality. It can be observed from the table to achieve high-performance minimum value of 30 needs to be utilized. The higher the the value for m the better the score for localization. The value chosen is 40 which has the most balanced results. |