Unveiling Multiple Descents in Unsupervised Autoencoders
Authors: Kobi Rahimi, Yehonathan Refael, Tom Tirer, Ofir Lindenbaum
TMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this study, we first demonstrate analytically that double descent does not occur in linear unsupervised autoencoders (AEs). In contrast, we show for the first time that both double and triple descent can be observed with nonlinear AEs across various data models and architectural designs. We examine the effects of partial sample and feature noise and highlight the critical role of bottleneck size in shaping the double descent curve. Through extensive experiments on both synthetic and real datasets, we uncover model-wise, epoch-wise, and sample-wise double descent across several data types and architectures. |
| Researcher Affiliation | Academia | Kobi Rahimi EMAIL Faculty of Engineering, Bar-Ilan University, Ramat-Gan 5290002, Israel. Yehonathan Refael EMAIL Faculty of Engineering, Tel Aviv University, Tel Aviv 6997801, Israel. Tom Tirer EMAIL Faculty of Engineering, Bar-Ilan University, Ramat-Gan 5290002, Israel. Ofir Lindenbaum EMAIL Faculty of Engineering, Bar-Ilan University, Ramat-Gan 5290002, Israel. |
| Pseudocode | No | The paper describes methodologies and theoretical analyses in prose and mathematical formulations. It does not contain any explicitly labeled "Pseudocode" or "Algorithm" sections, nor any structured code-like blocks. |
| Open Source Code | No | The paper does not contain an explicit statement about releasing source code for the described methodology, nor does it provide a link to a code repository. It mentions being "Reviewed on Open Review: https: // openreview. net/ forum? id= Fqf HDs6unx", which pertains to the review process, not code availability. |
| Open Datasets | Yes | Our experiments focus on synthetic and real-world datasets under various contamination scenarios, including noise, domain shifts, and outliers. Additionally, we identify model, epoch, and sample-wise double descent, highlighting the bottleneck size s role in shaping the double descent curve. We used single-cell RNA data from (Tran et al., 2020) to demonstrate our findings in a challenging, highdimensional real-world setting. The Celeb A dataset (Liu et al., 2015) was selected to evaluate the impact of model complexity on unsupervised anomaly detection performance (Han et al., 2022). We employed the MNIST (Le Cun et al., 1998) and CIFAR-10 (Krizhevsky et al., 2009) datasets to evaluate our findings on standard image benchmarks, demonstrating double descent under sample noise, feature noise, and domain shift scenarios. |
| Dataset Splits | Yes | For the subspace data model, nonlinear subspace data model, MNIST, and CIFAR-10 datasets, we generate 5000 samples for training and 10000 for testing across all scenarios (sample noise, feature noise, domain shift, and anomaly detection). Regarding the single-cell RNA data,... We allocate 5000 samples for training and introduce noise to specific samples and features, as described in subsection 3.1. The reserved 3569 samples are for testing. For the celeb A dataset, including over 200K samples and 4547 anomalies, each characterized by 40 binary attributes, we sub-sample 3000 clean samples and replace 3000 p of them with anomalies... Due to the limited availability of anomaly data (4547 samples), the test set includes (1 p) 4547 anomalies along with an equal number of clean samples. |
| Hardware Specification | Yes | All experiments were conducted on NVIDIA RTX 6000 Ada Generation with 47988 Mi B, NVIDIA Ge Force RTX 3080 with 10000 Mi B, Tesla V100-SXM2-32GB with 34400 Mi B, and NVIDIA Ge Force GTX 1080 Ti with 11000 Mi B. |
| Software Dependencies | No | The training optimizer utilized was Adam (Kingma & Ba, 2014), and the loss function for reconstruction is the mean squared error, which is mentioned in this Section. The paper mentions software names like "Adam" but does not specify version numbers for any software libraries or frameworks used in the implementation. |
| Experiment Setup | Yes | Table 1: Parameters and hyper-parameters PARAMETERS LINEAR/ NONLINEAR SUBSPACE RNA CELEBA MNIST/ CIFAR-10 Model FCN FCN FCN CNN Learning rate 0.001 0.001 0.001 0.001 Optimizer Adam Adam Adam Adam Epochs 200 1000 200 1000 Batch size 10 128 10 128 Data s latent features size (d) 20 Number of features (D) 50 1000 40 784/ 3072 Train dataset size 5000 5000 3000 5000 SNR/ SAR [d B] -20, -18, -17, -15, -10, -7, -5, -2, 0, 2 Contamination percentage (p) 0, 0.1, 0.2,...,1 Domain shift scale (s) 1, 2, 3, 4 Embedding layer size 25, 30, 45 20, 100, 300 25 10, 30, 50, 500 Hidden layer size 4 500 10 3000 4-400 Channels 1-64 |