Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1]
Formulating Robustness Against Unforeseen Attacks
Authors: Sihui Dai, Saeed Mahloujifar, Prateek Mittal
NeurIPS 2022 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically demonstrate that using VR can lead to improved generalization to unforeseen attacks during test-time, and combining VR with perceptual adversarial training (Laidlaw et al., 2021) achieves state-of-the-art robustness on unforeseen attacks. |
| Researcher Affiliation | Academia | Sihui Dai Princeton University EMAIL Saeed Mahloujifar Princeton University EMAIL Prateek Mittal Princeton University EMAIL |
| Pseudocode | No | The paper describes the training objective with equations and text, but does not provide any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our code is publicly available at https://github.com/inspire-group/variation-regularization. |
| Open Datasets | Yes | Datasets We train models on CIFAR-10, CIFAR-100, (Krizhevsky et al., 2009) and Image Nette (Howard). |
| Dataset Splits | No | The paper mentions training on datasets like CIFAR-10 and CIFAR-100 but does not explicitly provide the specific training, validation, and test split percentages or sample counts in the main text or appendix. While it refers to 'standard practices', explicit split details are not stated. |
| Hardware Specification | Yes | Training an AT-VR ResNet-18 on CIFAR-10 takes 12-15 hours on a single NVIDIA A100 GPU. |
| Software Dependencies | No | The paper mentions implementing models 'using PyTorch' but does not specify a version number for PyTorch or any other software dependencies. |
| Experiment Setup | Yes | For CIFAR-10 and CIFAR-100, we used an ℓ adversarial budget of 8/255 and ℓ2 budget of 0.5. For Image Nette, we use an ℓ adversarial budget of 4/255 and ℓ2 budget of 1.0. We use a batch size of 128. For CIFAR-10 and CIFAR-100, we use an initial learning rate of 0.1 with a cosine learning rate schedule that decays to 0.001 at the last epoch. For Image Nette, we use an initial learning rate of 0.01 with cosine learning rate schedule that decays to 0.001. We train for 100 epochs on all datasets. We use a momentum of 0.9 and a weight decay of 0.0005. We use PGD to generate adversarial examples with 10 steps and step size of 2/255 for ℓ and 0.01 for ℓ2. |