The Same but Different: Structural Similarities and Differences in Multilingual Language Modeling
Authors: Ruochen Zhang, Qinan Yu, Matianyu Zang, Carsten Eickhoff, Ellie Pavlick
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We employ new tools from mechanistic interpretability to ask whether the internal structure of large language models (LLMs) shows correspondence to the linguistic structures which underlie the languages on which they are trained. In particular, we ask (1) when two languages employ the same morphosyntactic processes, do LLMs handle them using shared internal circuitry? and (2) when two languages require different morphosyntactic processes, do LLMs handle them using different internal circuitry? In a focused case study on English and Chinese multilingual and monolingual models, we analyze the internal circuitry involved in two tasks. We find evidence that models employ the same circuit to handle the same syntactic process independently of the language in which it occurs, and that this is the case even for monolingual models trained completely independently. Moreover, we show that multilingual models employ language-specific components (attention heads and feed-forward networks) when needed to handle linguistic processes (e.g., morphological marking) that only exist in some languages. |
| Researcher Affiliation | Academia | Ruochen Zhang1 Qinan Yu2 Matianyu Zang1 Carsten Eickhoff3 Ellie Pavlick1 1Brown University 2Stanford University 3University of T ubingen |
| Pseudocode | No | The paper describes methods like 'Path Patching' and 'Information Flow Routes' in prose and illustrates circuits with diagrams, but it does not contain any clearly labeled pseudocode or algorithm blocks with structured steps. |
| Open Source Code | No | The paper does not provide an unambiguous statement of releasing its own source code, nor does it include any links to a code repository for the methodology described. |
| Open Datasets | Yes | We set up our data following the original IOI paper (Wang et al., 2023)... To collect the verbs, we start with the English verb tense dataset from Big-bench (Srivastava et al., 2022) and filter the verbs to only retain those with regular inflection rules. |
| Dataset Splits | No | The paper mentions using '50 examples respectively in English and Chinese' for the information flow routes technique and '62 English verb instances' for the past tense task. It also provides accuracy and zero-rank rates. However, it does not specify explicit training, validation, or test splits for these datasets, nor does it reference predefined splits with clear citations for data partitioning within the scope of this paper. |
| Hardware Specification | No | The paper uses specific models such as BLOOM-560M, GPT2-small, CPM-distilled, and Qwen2-0.5B-instruct. However, it does not provide any specific details about the hardware (e.g., GPU models, CPU types, memory) used to run the experiments. |
| Software Dependencies | No | The paper mentions the use of 'transformer-lens library (Nanda & Bloom, 2022)' for extracting IOI circuits. However, it does not specify a version number for this or any other software used in their experimental setup. |
| Experiment Setup | Yes | Following Ferrando & Voita (2024), we set the threshold τ = 0.03. We consider a head is highly activated if its contribution value surpasses τ. The ablation experiments carried out in the paper are done via zero-ablation, which refers to zeroing out the contribution of a certain head. |