The 2024 Foundation Model Transparency Index
Authors: Rishi Bommasani, Kevin Klyman, Sayash Kapoor, Shayne Longpre, Betty Xiong, Nestor Maslej, Percy Liang
TMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To characterize the status quo, the Foundation Model Transparency Index was launched in October 2023 to measure the transparency of leading foundation model developers. The October 2023 Index (v1.0) assessed 10 major foundation model developers (e.g. Open AI, Google) on 100 transparency indicators (e.g. does the developer disclose the wages it pays for data labor?). At the time, developers publicly disclosed very limited information with the average score being 37 out of 100. To understand how the status quo has changed, we conduct a follow-up study (v1.1) after 6 months: we score 14 developers against the same 100 indicators. We find that developers now score 58 out of 100 on average, a 21 point improvement over v1.0. Our findings demonstrate that transparency can be improved in this nascent ecosystem. |
| Researcher Affiliation | Academia | Rishi Bommasani* EMAIL Stanford University Kevin Klyman* EMAIL Stanford University Sayash Kapoor EMAIL Princeton University Shayne Longpre EMAIL Massachusetts Institute of Technology Betty Xiong EMAIL Stanford University Nestor Maslej EMAIL Stanford University Percy Liang EMAIL Stanford University |
| Pseudocode | No | The paper describes its methodology in prose in Section 3 "FMTI v1.1 involves four steps: indicator selection, developer selection, information gathering, and scoring." It does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | We publish the transparency reports for all developers to enable further research.3 Footnote 3: https://www.github.com/stanford-crfm/fmti |
| Open Datasets | Yes | We publish the transparency reports for all developers to enable further research.3 Footnote 3: https://www.github.com/stanford-crfm/fmti |
| Dataset Splits | No | The paper analyzes transparency scores of foundation model developers. It does not involve traditional machine learning datasets or their splits (training, validation, test) for experimental reproduction. |
| Hardware Specification | No | The paper describes a study to score the transparency of foundation model developers and does not describe any computational experiments involving specific hardware for model training or evaluation by the authors. It mentions "compute" as one of the indicators it assesses in foundation models, but not the hardware used for its own analysis. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers used for its analysis (e.g., Python version, specific libraries like Pandas or SciPy with versions). |
| Experiment Setup | No | The paper details a methodology for scoring transparency, which includes indicator selection, developer selection, information gathering, and scoring. However, it does not describe experimental setup details such as hyperparameters, learning rates, batch sizes, or model initialization, as these are not relevant to the nature of this research. |