Differentially Private Diffusion Models
Authors: Tim Dockhorn, Tianshi Cao, Arash Vahdat, Karsten Kreis
TMLR 2023 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We validate our novel DPDMs on image generation benchmarks and achieve state-of-the-art performance in all experiments. Moreover, on standard benchmarks, classifiers trained on DPDM-generated synthetic data perform on par with task-specific DP-SGD-trained classifiers, which has not been demonstrated before for DP generative models. Project page and code: https://nv-tlabs.github.io/DPDM. |
| Researcher Affiliation | Collaboration | Tim Dockhorn EMAIL Stability AI Tianshi Cao EMAIL NVIDIA University of Toronto Vector Institute Arash Vahdat EMAIL NVIDIA Karsten Kreis EMAIL NVIDIA |
| Pseudocode | Yes | For an algorithm for DPDM training with noise multiplicity see Alg. 1. For the sake of completeness we also formally prove the DP of DPDMs (DP of releasing sanitized training gradients Gbatch): |
| Open Source Code | Yes | Project page and code: https://nv-tlabs.github.io/DPDM. |
| Open Datasets | Yes | Datasets. We focus on image synthesis and use MNIST (Le Cun et al., 2010), Fashion-MNIST (Xiao et al., 2017) (28x28), and Celeb A (Liu et al., 2015) (downsampled to 32x32). These datasets are standard benchmarks in the DP generative modeling literature. |
| Dataset Splits | Yes | For downstream classifier training, we follow the DP generation literature and use 60k synthesized samples. We follow Cao et al. (2021) and split the 60k samples into a training set (90%) and a validation set (remaining 10%). |
| Hardware Specification | Yes | For all experiments, we use an in-house GPU cluster of V100 NVIDIA GPUs. |
| Software Dependencies | Yes | We implement DPDMs in Py Torch (Paszke et al., 2019) and use Opacus (Yousefpour et al., 2021), a DP-SGD library in Py Torch, for training and privacy accounting. |
| Experiment Setup | Yes | In all settings, we use 300 epochs and clipping constant C=1. We use batch size B=4096 for MNIST and Fashion-MNIST and decrease the batch size of Celeb A to B=2048 for the sole purpose of fitting the entire batch into GPU memory. The DP noise σDP values for each setup can be found in Tab. 9 |