Calibrated Probabilistic Forecasts for Arbitrary Sequences
Authors: Charles Marx, Volodymyr Kuleshov, Stefano Ermon
TMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, our algorithms improve calibration and downstream decision-making for energy systems. We perform three experiments to evaluate the ability of ORCA to recalibrate forecasts on regression tasks. In the first experiment, we test whether ORCA can improve calibration without worsening predictive performance for four different online learning models on two regression tasks. In the second experiment, we apply ORCA to data that is adversarially selected to maximize miscalibration. In the third experiment, we simulate a decision task for a wind farm operator to test whether ORCA improves downstream decision-making. |
| Researcher Affiliation | Academia | Charles Marx EMAIL Department of Computer Science Stanford University Volodymyr Kuleshov EMAIL Department of Computer Science Cornell Tech Stefano Ermon EMAIL Department of Computer Science Stanford University |
| Pseudocode | Yes | Algorithm 1 Blackwell Forecasting ... Algorithm 2 ORCA: Gradient-Based Blackwell Forecasting |
| Open Source Code | No | The paper mentions the use of the 'River online learning library (Montiel et al., 2021)' as an implementation for expert forecasters but does not state that the authors' own code for ORCA or their methodology is open-source or publicly available. There is no explicit statement of code release or a link to a code repository for the work described in this paper. |
| Open Datasets | Yes | The wind dataset consists of hourly wind energy generation in ERCOT for the year of 2022 (of Texas, 2022). ... The sunspot dataset (Clette et al., 2014) is a standard forecasting benchmark where the task is to predict the total monthly sunspots. |
| Dataset Splits | Yes | For each dataset, we forecast for 1000 time steps, using lag features from the previous 24 steps. |
| Hardware Specification | Yes | Experiments were run on CPU on a Mac Book Pro with an M2 chip and 64 GB of memory. |
| Software Dependencies | No | The paper mentions the use of 'Adam optimizer (Kingma & Ba, 2014)' and the 'River online learning library (Montiel et al., 2021)'. However, it does not provide specific version numbers for these or other software dependencies like programming languages (e.g., Python), deep learning frameworks (e.g., PyTorch, TensorFlow), or other libraries used in their implementation of ORCA. |
| Experiment Setup | Yes | We parameterize the action spaces of Forecaster and Nature for the minimax optimization task both as piecewise constant densities over 50 evenly-spaced subsets of the outcome space. ... We perform 400 update steps with the Adam optimizer (Kingma & Ba, 2014) to solve the minimax optimization task. |