V-statistics and Variance Estimation
Authors: Zhengze Zhou, Lucas Mentch, Giles Hooker
JMLR 2021 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirical studies in Section 7 corroborate the developed asymptotic theories and the effectiveness of variance estimation procedure; 7. Empirical Studies Here, we conduct two suites of experiments. All simulations are implemented in Python. For building random forests, we apply Random Forest Classifier and Random Forest Regressor from scikit-learn Friedman (1991). Unless otherwise noted, default parameter values are used. 7.1 Predictive Performance In this section, we evaluate the predictive performance for different sampling strategies. ... There are six datasets taken from UCI Machine learning Repository... 7.2 Asymptotic Normality and Variance Estimation In this section, we illustrate empirically the asymptotic normality property and variance estimation algorithms for V -statistics. |
| Researcher Affiliation | Academia | Zhengze Zhou EMAIL Department of Statistics and Data Science Cornell University Ithaca, NY 14850, USA Lucas Mentch EMAIL Department of Statistics University of Pittsburgh Pittsburgh, PA 15260, USA Giles Hooker EMAIL Department of Statistics University of California, Berkeley Berkeley, CA 94720, USA |
| Pseudocode | Yes | Algorithm 1 Internal Variance Estimation Method for i in 1 to n OUT do Select initial fixed point z(i) for j in 1 to n IN do Select subsample S z(i),j of size kn from training set that includes z(i) Build base learner and evaluate hkn(S z(i),j) end for Record average of the n IN predictions end for Compute the variance of the n OUT averages to estimate ζ1,kn Compute the variance of all predictions to estimate ζkn,kn Compute the mean of all predictions to obtain final ensemble prediction |
| Open Source Code | Yes | Code accompanying this paper can be found at https://github.com/ Zhengze Zhou/V-statistics-and-Variance-Estimation. |
| Open Datasets | Yes | 1.1 A Motivating Example We will illustrate the practical importance of our contribution through a motivating example using Boston Housing Data1. 1. https://archive.ics.uci.edu/ml/machine-learning-databases/housing/ and There are six datasets taken from UCI Machine learning Repository (see Appendix K for details) and Appendix K. Datasets Information Six of the seven datasets in Table 7.1 are taken from UCI Machine Learning Repository5: boston: https://archive.ics.uci.edu/ml/machine-learning-databases/housing/. ... diabetes: https://archive.ics.uci.edu/ml/datasets/diabetes. ... iris: https://archive.ics.uci.edu/ml/datasets/Iris. ... digits: https://archive.ics.uci.edu/ml/datasets/optical+recognition+of+handwritten+ digits. ... retinopathy: https://archive.ics.uci.edu/ml/datasets/Diabetic+Retinopathy+ Debrecen+Data+Set. ... breast cancer: https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+ (Diagnostic). and 4. https://archive.ics.uci.edu/ml/datasets/Physicochemical+Properties+of+Protein+Tertiary+ Structure |
| Dataset Splits | Yes | 1.1 A Motivating Example To simulate practical use cases, we leave 20% of the data as test set and train on the remaining samples. and 7.1 Predictive Performance Each model is built using 100 trees and to full depth until a leaf is pure or contain fewer than 2 data points. 20% of samples are left as test set. and 7.2 Asymptotic Normality and Variance Estimation In order to see how our results work in real world settings, we pick a relatively large scale dataset: Physicochemical Properties of Protein Tertiary Structure Data Set4. The data set contains 45730 samples with 9 covariates and the target is the size of the residue. To simulate the situation where one can attain alternative training data drawn from the same data generating distribution to quantify sampling uncertainty, we randomly select 45000 samples and partition them to 45 sets with 1000 in each. |
| Hardware Specification | No | The paper does not provide specific hardware details used for running its experiments. |
| Software Dependencies | No | All simulations are implemented in Python. For building random forests, we apply Random Forest Classifier and Random Forest Regressor from scikit-learn Friedman (1991). and implemented by scipy.stats.normaltest3 in Python. |
| Experiment Setup | Yes | Each model is built using 100 trees and to full depth until a leaf is pure or contain fewer than 2 data points. 20% of samples are left as test set. For classification, p of features are considered when searching for best splits, and p/3 for regression. and In each iteration we generate n = 500 training observations and train random forests with subsample size k = 100, 250, 500 and the number of trees B = 500, 1000, 2500, 5000. and We make evaluation on three test points: p1 = [0.5, 0.5, 0.5, 0.5, 0.5] and p2, p3 are randomly drawn from U([0, 1]5). |