TorchCP: A Python Library for Conformal Prediction
Authors: Jianguo Huang, Jianqing Song, Xuanning Zhou, Bingyi Jing, Hongxin Wei
JMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This paper introduces Torch CP, a PyTorch-native library designed to integrate state-of-the-art CP algorithms into DL techniques, including DNN-based classifiers/regressors, GNNs, and LLMs... Notably, Torch CP enables CP-specific training algorithms, online prediction, and GPU-accelerated batch processing, achieving up to 90% reduction in inference time on large datasets. |
| Researcher Affiliation | Academia | 1Southern University of Science and Technology 2Nanyang Technological University 3Nanjing University 4The Chinese University of Hong Kong, Shenzhen Corresponding author: Hongxin Wei (EMAIL) |
| Pseudocode | No | The paper includes Python code snippets in Section 2.2 for demonstrating usage and discusses interfaces for extensibility in Appendix B, but does not present any formal pseudocode blocks or algorithms labeled as such. |
| Open Source Code | Yes | This paper introduces Torch CP, a PyTorch-native library designed to integrate state-of-the-art CP algorithms into DL techniques... Released under the LGPL-3.0 license... Torch CP provides a user-friendly framework for conformal prediction in deep learning with a detailed documentation2. The library contains illustrative code showcasing functionality for common applications (See the examples folder 3). Footnote 3 links to: https://github.com/ml-stat-Sustech/Torch CP/tree/master/examples |
| Open Datasets | Yes | D.1 Evaluation results DNN Classifier. In Figure 2, we present the experimental results of Split CP on Image Net... DNN Regressor. In Figure 4, we present an example of experimental results on the Community dataset (Redmond, 2002)... GNN. In Figure 5, we present experimental results on the Cora ML dataset (Mc Callum et al., 2000)... LLM. In Table 5, we present an example prediction set, generated by conformal LLM, of the Trivia QA dataset (Joshi et al., 2017)... MNIST. We compared the computational efficiency of five libraries... on MNIST (Le Cun et al., 2010)... We evaluate the computational efficiency and predictive performance of Torch CP and an R-based CP package, Adaptive Conformal (Susmann et al., 2023) on a time series regression task, using the France electricity forecasting dataset14. Footnote 14 links to: https://cran.r-project.org/web/packages/opera/index.html |
| Dataset Splits | Yes | For the dataset, we use 48,000 samples for training, 6,000 for calibration, and 6,000 for testing. Image Net. ...we select 25,000 images from the validation set for calibration and another 25,000 for testing. Regression Task ...we use the first 350 time steps to train a SARIMA model with order (1, 0, 0) and seasonal order (1, 1, 0), and construct prediction intervals for the remaining time steps. |
| Hardware Specification | Yes | We use NVIDIA H100 for GPU computation and AMD EPYC 9654 for CPU computation. |
| Software Dependencies | No | The paper mentions that Torch CP is a "PyTorch-native library" and uses "Py Torch on the GPU" for training and inference, and mentions "JAX" for Fortuna. However, it does not specify any version numbers for PyTorch, JAX, or other relevant software libraries. |
| Experiment Setup | Yes | Throughout all experiments, we used Split CP with APS as the conformal prediction method... In our experiments, we set the batch size to 128 for batch processing... For DAPS, the diffusion parameter is set to 0.5. In CF-GNN, the topology-aware correction model is implemented using a two-layer GCN. For SNAPS, the hyperparameters are set as follows: λ = 1/3, and k = 20... We use a pre-trained Res Net-50 model from torchvision... we use the first 350 time steps to train a SARIMA model with order (1, 0, 0) and seasonal order (1, 1, 0) |