Combining Machine Learning Defenses without Conflicts
Authors: Vasisht Duddu, Rui Zhang, N. Asokan
TMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show that Def\Con achieves 90% accuracy on eight combinations from prior work, and 86% in 30 unexplored combinations which we empirically evaluated. |
| Researcher Affiliation | Academia | Vasisht Duddu EMAIL University of Waterloo Rui Zhang EMAIL Zhejiang University N. Asokan EMAIL University of Waterloo |
| Pseudocode | Yes | We now present our algorithm to evaluate conflict for multi-way combinations, and assume that we have a set of defenses partitioned into three ordered stages. We iterate through each stage in order, and skip stages with no defenses: M1 If stage has single defense: For the stage s, we consider the defense as Ds and check its compatibility with defenses in other stages. Ds is D1, D2 or D3 depending on s. M2 If stage has multiple defenses: For each stage with multiple defenses: M2.1 Permutation Checking: Consider all permutations of the defenses. For each permutation, check every consecutive pair of defenses using Def\Con flowchart to detect conflicts for two defenses. M2.2 Pruning and Selection: If a pair in a permutation causes a conflict, discard that permutation and prune others containing that conflicting pair (e.g., for defenses: A, B, and C, if AB conflict, then no need to check for ABC or CAB ). If a conflict-free permutation is found, treat the entire sequence as a single composite defense, Ds (e.g., let us say CAB does not conflict). M3 Sequential Composition: Treat the resulting composite defense as an atomic unit (e.g., D1 = CAB ) and check for conflicts with the next stage s resolved defenses (e.g., D2 = DE ), using Def\Con flowchart between D1 and D2. In other words, we check whether any of the constituent defenses in D1 use a risk which is mitigated by constituent defenses in D2 (S3 in Figure 1). M4 Termination: If any invocation of Def\Con for pairwise check indicates a conflict, terminate and report as a conflict (marked as ). Else, indicate as alignment (marked as ). |
| Open Source Code | Yes | 1Link to Code: https://github.com/ssg-research/combining-defenses. |
| Open Datasets | Yes | We use two image datasets: FMNIST and UTKFACE. FMNIST consists of 28x28 grayscale images of ten clothing types, with 60,000 training and 10,000 testing images. ... UTKFACE includes 48x48 RGB images, classifying individuals as young (under 30), with 11,852 training and 10,667 testing images. |
| Dataset Splits | Yes | FMNIST consists of 28x28 grayscale images of ten clothing types, with 60,000 training and 10,000 testing images. ... UTKFACE includes 48x48 RGB images, classifying individuals as young (under 30), with 11,852 training and 10,667 testing images. |
| Hardware Specification | No | The paper does not explicitly state the hardware used for running the experiments. It only generally mentions 'GPUs' in a discussion about practical considerations. |
| Software Dependencies | No | The paper mentions several software libraries like TRADES (Zhang et al., 2019), Auto Attack (Croce & Hein, 2020) in Sec ML library (Pintor et al., 2022b), Opacus library (Yousefpour et al., 2021), fair fairness benchmark (Han et al., 2023), and Captum library (Kokhlikyan et al., 2020) for Deep Lift (Shrikumar et al., 2017), but does not provide specific version numbers for any of them. |
| Experiment Setup | Yes | When combining defenses, we use the same hyperparameters, but revisit hyperparameter tuning for defenses in combination (see 7.4). For Poisn Rob.Post (Pruning), we use the implementation from Zheng et al. (2022b). We sweep pruning thresholds from 0.6 to 1.5, in increments of 0.05, to get a model with highest ϕu and lowest ϕASR. We use the step size of 1.0 for L1 attack, 0.01 for L2 attack, and 0.001 for Linf, and 50 samples for computing p-value from the confidence regressor model. We use the implementation from Opacus library (Yousefpour et al., 2021) with a noise multiplier of 1.0 and gradient norm clipping of 1.0 as used in their tutorial for MNIST. We set the regularization hyperparameter λ = 1 which was sufficient to reduce ϕeqodds with 2% drop in ϕu. Table 6: Configurations for hyperparameter tuning of defenses in conflicting combinations. |