Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination
Authors: Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, Qing Qu
JMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Moreover, our extensive experiments not only validate our theoretical results but also reveal a similar pattern in deep nonlinear networks, which aligns well with recent empirical studies. Finally, we demonstrate the practical value of our results in transfer learning. Our code is available at https://github.com/Heimine/PNC_DLN. ... In this section, we conduct various numerical experiments to validate our assumptions, verify our theoretical results, and investigate the implications of our results on both synthetic and real data sets. |
| Researcher Affiliation | Academia | Department of Electrical Engineering & Computer Science, University of Michigan, Ann Arbor Department of Computer Science & Engineering, Ohio State University, Columbus |
| Pseudocode | No | The paper describes methods and theoretical analysis but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our code is available at https://github.com/Heimine/PNC_DLN. |
| Open Datasets | Yes | We train two networks with different architectures on the CIFAR-10 dataset (Krizhevsky and Hinton, 2009)... We train these networks on the Fashion MNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky and Hinton, 2009) datasets... We use the SST-5 text dataset (Socher et al., 2013)... We use the PyTorch pretrained VGG11 network (Simonyan and Zisserman, 2015) on Image Net (Deng et al., 2009). |
| Dataset Splits | Yes | We employ the SGD optimizer to train the networks by minimizing the MSE loss on the CIFAR-10 dataset... We train these networks on the Fashion MNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky and Hinton, 2009) datasets... We use the CIFAR-100 and CIFAR-10 dataset in the pre-training and fine-tuning tasks, respectively. |
| Hardware Specification | Yes | All of our experiments are conducted on a PC with 8GB memory and an Intel(R) Core i5 1.4GHz CPU, except for those involving large datasets such as CIFAR and Fashion MNIST, which are conducted on a server equipped with NVIDIA A40 GPUs. |
| Software Dependencies | No | Our code is implemented in Python and made available at https://github.com/Heimine/PNC_DLN. The paper mentions Python but does not provide specific version numbers for Python or any libraries used. |
| Experiment Setup | Yes | We employ the SGD optimizer to train the networks by minimizing the MSE loss on the CIFAR-10 dataset (Krizhevsky and Hinton, 2009). For the settings of the SGD optimizer, we use a momentum of 0.9, a weight decay of 10-4, and a dynamically adaptive learning rate ranging from 10-3 to 10-5, modulated by a Cosine Annealing learning rate scheduler as detailed in Loshchilov and Hutter (2017). We use the orthogonal initialization in (18) with ΞΎ = 0.1 to initialize the network weights. The neural networks are trained for 400 epochs with a batch size of 128. |