Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1]
Deletion and Insertion Tests in Regression Models
Authors: Naofumi Hama, Masayoshi Mase, Art B. Owen
JMLR 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this paper we make a careful study of the properties of those measures and we illustrate their use on two datasets. Insertion and deletion tests were used by Petsiuk et al. (2018) to compare variable importance methods for black box functions. In this section we illustrate insertion and deletion tests for regression. We compare our variance importance measures on two tabular datasets. The first one is about predicting the value of houses in India. The second dataset is from CERN and it computes the invariant mass produced from some electron collisions. The ABCs and their differences are summarized in Table 1. |
| Researcher Affiliation | Collaboration | Naofumi Hama EMAIL Hitachi, Ltd. Research & Development Group Kokubunji, Tokyo, 185-8601, Japan Masayoshi Mase EMAIL Hitachi, Ltd. Research & Development Group Kokubunji, Tokyo, 185-8601, Japan Art B. Owen EMAIL Department of Statistics Stanford University Stanford, CA 94305, USA |
| Pseudocode | No | The paper describes methods and derivations in prose and mathematical notation but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not contain any explicit statement about releasing source code, nor does it provide a link to a code repository or mention code in supplementary materials. |
| Open Datasets | Yes | The dataset we use lists the value in Indian rupees (INR) for houses in India. The data are from Kaggle at this URL: www.kaggle.com/ruchi798/housing-prices-in-metropolitan-areas-of-india. The CERN Electron Collision Data (Mc Cauley, 2014) is a dataset about dielectron collision events at CERN. |
| Dataset Splits | Yes | We selected 80% of the data points at random to train a multilayer perceptron (MLP). randomly select 80% of the complete observations (79,931 data points) to construct an MLP. |
| Hardware Specification | No | The paper describes training models and running experiments but does not provide any specific hardware details such as GPU or CPU models, or memory specifications. |
| Software Dependencies | No | The paper mentions using Optuna for hyperparameter optimization and Captum for XAI method implementations, but it does not provide specific version numbers for these or any other software dependencies. |
| Experiment Setup | Yes | The hyperparameters such as number of layers and ratio of dropouts are determined from a search described in Appendix A.2. Appendix A.2 Table 6: 'Hyperparameter Value Dropout Ratio 0.10031 Learning Rate 1.7389 10 2 Number of Neurons [333 465 86 234] Huber Parameter 1.0'. Appendix A.4 Table 7: 'Hyperparameter Value Dropout Ratio 0.11604 Learning Rate 1.9163 10 4 Number of Neurons [509 421 65 368 122 477] Huber Parameter 1.0'. |