Position: AI Should Not Be An Imitation Game: Centaur Evaluations
Authors: Andreas Haupt, Erik Brynjolfsson
ICML 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | This position paper argues that the machine learning community should increasingly use centaur evaluations, in which humans and AI jointly solve tasks. Centaur Evaluations refocus machine learning development toward human augmentation instead of human replacement, they allow for direct evaluation of human-centered desiderata, such as interpretability and helpfulness, and they can be more challenging and realistic than existing evaluations. |
| Researcher Affiliation | Academia | 1Digital Economy Lab, Stanford University, CA, USA. Correspondence to: Andreas Haupt <EMAIL>. |
| Pseudocode | No | The paper includes a mathematical formalization in Section 4.3 to highlight how centaur evaluations allow for a formalization of human augmentation, which includes a production function and a definition, but it does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper is a position paper proposing a new evaluation methodology and does not describe or provide code for its own methodology. It mentions Open AI's Gym as a framework used by others, but not its own code. |
| Open Datasets | No | The paper discusses various existing evaluation datasets (e.g., MMLU, Imagenet) as examples within its argument, but it does not use a dataset for experiments or provide access information for a new or existing dataset that it has utilized in its own research. |
| Dataset Splits | No | The paper is a theoretical position paper and does not conduct experiments with datasets, therefore, it does not provide any dataset split information. |
| Hardware Specification | No | The paper is a theoretical position paper and does not conduct experiments, therefore, it does not specify any hardware used for running experiments. It mentions 'K' for compute in a theoretical production function, but not actual hardware specifications. |
| Software Dependencies | No | The paper is a theoretical position paper and does not implement or run any software. It mentions 'Open AI's Gym' as an example of an interface in the context of other research, but not as a dependency for its own work. |
| Experiment Setup | No | The paper is a theoretical position paper and does not conduct experiments, therefore, it does not provide any experimental setup details such as hyperparameters or training configurations. |