Online Selective Conformal Inference: Errors and Solutions
Authors: Yusuf Sale, Aaditya Ramdas
TMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our theoretical findings are supported by experimental evidence examining trade-offs between valid methods. |
| Researcher Affiliation | Academia | Yusuf Sale EMAIL Institute of Computer Science Ludwig-Maximilians Universität München Munich Center for Machine Learning (MCML) Aaditya Ramdas EMAIL Departments of Statistics and Machine Learning Carnegie Mellon University |
| Pseudocode | Yes | Algorithm 1 LORD-CI procedure (Weinstein & Ramdas, 2020, modified) |
| Open Source Code | No | No explicit statement or link to source code for the methodology described in this paper was found. The paper discusses other methods' code or refers to papers where code might be available, but not for its own proposed methods. |
| Open Datasets | No | In each iteration of the simulation, we generate data of size N = Noff + Non. For the following results, we set Noff = 10 and Non = 20. We generate a univariate feature X Unif[0, 2] and model the response as Y = µ(X) + ϵ with µ(X) = Xβ, where we assume a heterogeneous noise distribution, i.e., ϵ | X N(0, X/2). |
| Dataset Splits | No | The paper describes generating synthetic data of size N = Noff + Non for simulations but does not refer to traditional dataset splits (training, validation, test) for a pre-existing dataset. The data arrives sequentially. |
| Hardware Specification | No | The paper does not specify any particular hardware (e.g., GPU, CPU models, or cloud resources) used for running the experiments or simulations. |
| Software Dependencies | No | The paper does not explicitly mention specific software dependencies with version numbers (e.g., programming languages, libraries, or frameworks) used in the implementation or simulations. |
| Experiment Setup | Yes | In each iteration of the simulation, we generate data of size N = Noff + Non. For the following results, we set Noff = 10 and Non = 20. We generate a univariate feature X Unif[0, 2] and model the response as Y = µ(X) + ϵ with µ(X) = Xβ, where we assume a heterogeneous noise distribution, i.e., ϵ | X N(0, X/2). For simplicity, we assume β = 1. Since we are in a synthetic setting, we have direct access to the true function and use it as our model; specifically, we have bµ( ) = µ( ). Further, the selection rules are defined as... where we choose τ0 = 20 and τ1 = 16. We then perform online selective conformal prediction with both existing and novel strategies. All reported metrics are averaged over N = 1 * 10^6 runs. |