Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Energy-based Automated Model Evaluation
Authors: Ru Peng, Heming Zou, Haobo Wang, Yawen Zeng, Zenan Huang, Junbo Zhao
ICLR 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide extensive experiments across modalities, datasets and different architectural backbones to validate MDE s validity, together with its superiority compared with prior approaches. |
| Researcher Affiliation | Collaboration | Ru Peng1 Heming Zou1 Haobo Wang1 Yawen Zeng2 Zenan Huang1 Junbo Zhao1 1Zhejiang University 2Byte Dance EMAIL EMAIL |
| Pseudocode | Yes | Algorithm 1 Automated Model Evaluation via Meta-distribution Energy |
| Open Source Code | Yes | Code and data are available: https://github.com/pengr/Energy_Auto_Eval |
| Open Datasets | Yes | In this work, we evaluate each method on the image classification tasks CIFAR-10, CIFAR100 (Krizhevsky et al., 2009), Tiny Image Net (Le & Yang, 2015), Image Net-1K (Deng et al., 2009), WILDS (Koh et al., 2021a) and the text inference task MNLI (Williams et al., 2018). |
| Dataset Splits | Yes | Table 5: Details of the datasets considered in our work. Train (Source) Valid (Source) Evaluation (Target)... (ii)-Synthetic Shift. We use CIFAR-10-C benchmark (Hendrycks & Dietterich, 2019)...applied to the CIFAR-10 validation set. |
| Hardware Specification | No | The paper mentions training models and experiments, but does not provide specific details on the hardware used, such as exact GPU or CPU models, or cloud computing instance types. |
| Software Dependencies | No | The paper mentions software like 'pytorch-cifar-models', 'timm library', and 'Hugging Face library' but does not specify their version numbers, which are required for reproducibility. |
| Experiment Setup | No | The paper states 'Following the practice in Deng et al. (2023), we train models...' and 'we use the same training settings as (Yu et al., 2022)', deferring specific experimental setup details and hyperparameters to external sources rather than providing them explicitly within the text. |