Learned Reference-based Diffusion Sampler for multi-modal distributions
Authors: Maxence Noble, Louis Grenioux, Marylou Gabrié, Alain Oliviero Durmus
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We experimentally demonstrate that LRDS best exploits prior knowledge on the target distribution compared to competing algorithms on a variety of challenging distributions. To validate our approach, we compare GMM-LRDS and EBM-LRDS on a variety of multi-modal distributions against the following annealed methods: (a) annealed MCMC methods Sequential Monte Carlo (SMC) and Replica Exchange (RE); (b) variational diffusion-based methods, implemented with the LV loss LV-PIS (Zhang & Chen, 2022), LV-DDS (Vargas et al., 2023a), LV-DIS (Berner et al., 2023) and LV-CMCD (Vargas et al., 2024); (c) adaptive diffusion-based approaches i DEM (Akhound-Sadegh et al., 2024) and PDDS (Phillips et al., 2024). |
| Researcher Affiliation | Academia | Maxence Noble , Louis Grenioux , Marylou Gabri e & Alain Oliviero Durmus CMAP, CNRS Ecole polytechnique, Institut Polytechnique de Paris 91120 Palaiseau, France. Corresponding authors: EMAIL |
| Pseudocode | Yes | A PSEUDO-CODES OF RDS-BASED ALGORITHMS We respectively give sampling procedures and training procedures of a general version of RDS in Algorithm 1 and Algorithm 2. Relying on this, we derive the complete training schemes for GMM-LRDS (Algorithm 3) and EBM-LRDS (Algorithm 4). |
| Open Source Code | Yes | Our codebase is available at https://github.com/h2o64/sde_sampler_lrds. |
| Open Datasets | Yes | Finally, we evaluate the performance of a Bayesian logistic model... Following Blessing et al. (2024), we consider four real-world settings of binary classification problem: Ionosphere (dim = 35, M = 351), Sonar (dim = 61, M = 208), German Credit (dim = 25, M = 1000), Breast Cancer (dim = 31, M = 569). Each of these datasets is randomly split into a training subset Dtrain of size 0.8M and a test subset Dtest of size 0.2M. |
| Dataset Splits | Yes | Each of these datasets is randomly split into a training subset Dtrain of size 0.8M and a test subset Dtest of size 0.2M. |
| Hardware Specification | Yes | The computations were all ran on the same V100 GPU. |
| Software Dependencies | No | For GMM-LRDS, the EM algorithm is taken from Pedregosa et al. (2011). - While this implies scikit-learn, it does not provide a specific version number for the library or any other software dependencies. |
| Experiment Setup | Yes | For all of them, we perform 4096 optimization steps with a batch of size 2048. The neural network at stake is a Fourier MLP, as in Zhang & Chen (2022), with 4 layers of width 64. In the case of PIS, DDS and DIS, we use a target-informed parameterization by adding NN(t) log π(x) (where NN is a time-dependent scalar neural network) to the Fourier MLP, as suggested by the respective authors. As recommended by Vargas et al. (2024), we do not consider this extra-parameterization in CMCD, since the drift of the generative process is already informed by π. We highlight that, by default, we do not use this target-informed parameterization in LRDS, since the reference process is specifically designed to be it-self target-informed, hence avoiding useless evaluations of the target score. Additionally, as recommended by Zhang & Chen (2022), we design the LRDS guidance network such that gθ0 = 0. This ensures that the very first sampling phase is driven solely using the reference process. |