Volume Optimality in Conformal Prediction with Structured Prediction Sets
Authors: Chao Gao, Liren Shan, Vaidehi Srinivas, Aravindan Vijayaraghavan
ICML 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We complement our theoretical guarantees with an evaluation of our methods for both the unsupervised setting of Section 2 and the supervised setting of Section 3. [...] Figure 1, Tables 1 and 2 summarize the results using data generated from a mixture of Gaussians [...] The two methods are also evaluated on two real-world datasets (Acidity and Enzyme) [...] The algorithms for the supervised setting are compared against conformalized quantile regression (CQR) [...] against benchmark simulated datasets [...] |
| Researcher Affiliation | Academia | 1Department of Statistics, University of Chicago, Chicago, Illinois, USA 2Toyota Technological Institute at Chicago, Chicago, Illinois, USA 3Department of Computer Science, Northwestern University, Evanston, Illinois, USA. |
| Pseudocode | Yes | Algorithm 1 Dynamic Programming Solving (7) 1: Input: data points Y1, . . . , Yn R, coverage level 1 α (0, 1) and slack γ (0, α), number of intervals k 2: Output: k intervals that cover (1 α)n points with minimum volume 3: Sort data in non-decreasing order Y(1) Y(n) 4: Set DP(i, j, l) = for all i [k], j [n], l [1/γ] 5: for i = 1 to k, j = 1 to n, l = 1 to 1/γ do 6: for j = i to j do 7: for j = i 1 to j 1 do 8: Set l = l (j j + 1)/(γn) 9: if l < 0 and i = 1 then 10: DP(i, j, l) = min{DP(i, j, l), Y(j) Y(j )} 11: end if 12: if DP(i 1, j , l ) = then 13: DP(i, j, l) = min{DP(i, j, l), Y(j) Y(j )+ DP(i 1, j , l )} 14: end if 15: end for 16: end for 17: end for 18: Find the minimum volume among all DP(k, j, (1 α)/γ ) for j = 1, . . . , n and backtrack to construct the prediction set b CDP. 19: Return the set b CDP. |
| Open Source Code | No | The paper does not explicitly state that source code for the methodology described is made available, nor does it provide a link to a code repository. It mentions using third-party packages like 'sklearn-quantile' and 'FlexCoDE' but not the authors' own implementation code. |
| Open Datasets | Yes | Figure 1, Tables 1 and 2 summarize the results using data generated from a mixture of Gaussians [...] The two methods are also evaluated on two real-world datasets (Acidity and Enzyme) used in density estimation literature (Richardson & Green, 1997). [...] The algorithms for the supervised setting are compared [...] against benchmark simulated datasets in Romano et al. (2019); Izbicki et al. (2020) (Figures 2 and 3). [...] We evaluate our method along with several baselines on two real-world regression datasets from the UCI repository: Air Foil and Wine Quality. These two datasets were previously considered in the conformal prediction study by Dhillon et al. (2024). |
| Dataset Splits | Yes | Simulated Dataset (Romano et al., 2019): [...] We generate 2000 training examples, and 5000 test examples, as in the work of (Romano et al., 2019). [...] Simulated Dataset (Izbicki et al., 2020): [...] We generate 2000 training examples and 5000 test examples. The same training and test sets are used consistently across all experiments to ensure reproducibility. [...] All methods are implemented within the split conformal framework, where the training data is randomly divided into two equal parts. Specifically, half of the data is allocated for model training, while the remaining half is used as the calibration set to ensure valid coverage guarantees. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., CPU/GPU models, memory) used for running the experiments. |
| Software Dependencies | Yes | For simulated dataset (Romano et al., 2019) with single dimensional predictor variable, we use the package sklearn-quantile (Roebroek, 2023) to implement the quantile regression, which implements the method of Quantile Regression Forests, due to Meinshausen (2006). The CD-Split and HPD-Split methods require the conditional density estimation, which is achieved by the R package Flex Co DE (Izbicki & Lee, 2017). |
| Experiment Setup | Yes | The construction involves the following steps: 1. Generate Sj by Dynamic Programming. For j = (1 α + n 1 + 3δ)m , we generate Sj by applying Algorithm 1 with coverage level 1 j /m and slack γ = 1/m. The discretization level m and statistical tolerance δ are chosen as m = 50 and δ = p (k + log n)/n,4 where k is the number of intervals in the prediction set. [...] For the conformalized DP method, the conformity score is constructed based on the nested system described in Section B.1 with m = 50 and δ = p (k + log n)/n. The conformalized KDE is also in the form of (9), with the conformity score given by q KDE(x) = 1 nρ Pn i=1 K y Yi ρ , where K( ) is the standard Gaussian kernel and ρ is the bandwidth parameter. Both methods involve a single tuning parameter, k for conformalized DP and ρ for conformalized KDE. |