Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1]
Fortuna: A Library for Uncertainty Quantification in Deep Learning
Authors: Gianluca Detommaso, Alberto Gasparin, Michele Donini, Matthias Seeger, Andrew Gordon Wilson, Cedric Archambeau
JMLR 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We present Fortuna, an open-source library for uncertainty quantification in deep learning. Fortuna supports a range of calibration techniques, such as conformal prediction that can be applied to any trained neural network to generate reliable uncertainty estimates, and scalable Bayesian inference methods that can be applied to deep neural networks trained from scratch for improved uncertainty quantification and accuracy. By providing a coherent framework for advanced uncertainty quantification methods, Fortuna simplifies the process of benchmarking and helps practitioners build robust AI systems. |
| Researcher Affiliation | Collaboration | 1AWS, Berlin, Germany 2Amazon, Berlin, Germany 3AWS & New York University |
| Pseudocode | Yes | Example. We wish to calibrate credible intervals with coverage error given by error . We assume to be given credible intervals ( test lower bounds and test upper bounds ) corresponding to different test input variables, and prediction intervals for several validation inputs ( val lower bounds and val upper bounds ), along with corresponding validation targets ( val targets ). The following code produces conformal prediction intervals as calibrated versions of the test prediction intervals. from fortuna.conformal.regression import Quantile Conformal Regressor conformal_intervals = Quantile Conformal Regressor (). conformal_interval ( val_lower_bounds =val_lower_bounds , val_upper_bounds =val_upper_bounds , test_lower_bounds =test_lower_bounds , test_upper_bounds =test_upper_bounds , val_targets=val_targets , error=error) |
| Open Source Code | Yes | We present Fortuna, an open-source library for uncertainty quantification in deep learning. Fortuna documentation and Git Hub repository. |
| Open Datasets | No | The paper does not provide concrete access information (specific link, DOI, repository name, formal citation, or reference to established benchmark datasets) for any publicly available or open datasets. It mentions using 'val_lower_bounds', 'val_upper_bounds', 'val_targets', 'calib_outputs', 'calib_targets', 'train_data_loader', 'val_data_loader', 'test_data_loader' as abstract examples of data inputs for the library, but does not specify actual datasets. |
| Dataset Splits | No | The paper does not provide specific dataset split information (exact percentages, sample counts, citations to predefined splits, or detailed splitting methodology). It mentions 'train_data_loader', 'val_data_loader', and 'test_data_loader' as inputs for the library, implying data splits exist, but does not specify how these splits are defined or created. |
| Hardware Specification | No | The paper does not provide specific hardware details (exact GPU/CPU models, processor types with speeds, memory amounts, or detailed computer specifications) used for running experiments. |
| Software Dependencies | No | The paper mentions "Fortuna is written in JAX (Bradbury et al., 2018), a fast growing Num Py-like framework... and adopts Flax (Heek et al., 2020)" but does not provide specific version numbers for JAX or Flax. |
| Experiment Setup | No | The paper provides code examples demonstrating how to use the Fortuna library but does not include specific experimental setup details such as concrete hyperparameter values, training configurations, or system-level settings for any particular experiment. |