Eigen Analysis of Conjugate Kernel and Neural Tangent Kernel
Authors: Xiangchao Li, Xiao Han, Qing Yang
ICML 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The existence and asymptotic positions of such isolated eigenvalues are rigorously analyzed. Furthermore, we provide a precise characterization of the entrywise limit of the projection matrix onto the eigenspace associated with these isolated eigenvalues. Our findings reveal that the eigenspace captures inherent group features present in X. This study offers a quantitative analysis of how group features from the input data evolve through hidden layers in randomly weighted neural networks. [...] The techniques employed in this paper are grounded in random matrix theory, and from a theoretical perspective, our analysis falls within the framework of finite-rank deformation models in this field. [...] In this subsection, we present additional simulations on both synthetic GMM and real data to further support our theoretical findings. [...] Figures 4(a)-(b) visualize the eigenvalues of f KCK,3 under different activation function settings. [...] Next, we turn to real data analysis. The input data consists of 1600 randomly selected images from each of the digit classes 1 and 7 in the MNIST dataset, with the class-specific mean subtracted from each group. Figure 5(a) displays the spectrum of the CK, obtained using a three-layer neural network. The eigenvectors associated with the isolated eigenvalues are shown in Figure 5(b). |
| Researcher Affiliation | Academia | 1School of Management, University of Science and Technology of China 2International Institute of Finance, School of Management, University of Science and Technology of China. Correspondence to: Qing Yang <EMAIL>. |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any explicit statements about the release of source code, nor does it include links to a code repository. |
| Open Datasets | Yes | The input data consists of 1600 randomly selected images from each of the digit classes 1 and 7 in the MNIST dataset, with the class-specific mean subtracted from each group. |
| Dataset Splits | No | The paper mentions using 1600 randomly selected images from each of the digit classes 1 and 7 in the MNIST dataset, but it does not specify any training/test/validation splits or percentages. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., CPU, GPU models, memory) used for running the experiments or simulations. |
| Software Dependencies | No | The paper does not specify any software dependencies (e.g., programming languages, libraries, frameworks) with version numbers. |
| Experiment Setup | Yes | Figure 1. Spectrum of f KCK,3 under the parameter settings: n = 1200, p = 600, c1 = c3 = 0.3, c2 = 0.4 and Ca = (1 + 2(a - 1)/ p)Ip for a = 1, 2, 3. The activation functions for the three layers are σ1 = σ2 = σ3 = Poly, where Poly(t) = 0.2t2 + t. The weights W 1 Rd1 p, W 2 Rd2 d1 and W 3 Rd3 d2 consist of i.i.d. standard normal entries, where d1 = d2 = 2000, d3 = 1000. [...] Figure 4. Eigenvalue histograms ((a)-(b)) and eigenvectors ((c)-(d)) corresponding to the isolated (largest) eigenvalues of f KCK,3 obtained with Poly and Re LU activation functions. Parameters settings: n = 2000, p = 3600, c1 = c3 = 0.2, c2 = c4 = 0.3 and Ca = (1 + 8(a - 1)/ p)Ip for a = 1, 2, 3, 4. The width d1, d2, d3 are identical to those in Figure 1. [...] Figure 5. [...] Activations=[Sin, Re LU/10, Sin]. The weights W 1 Rd1 p, W 2 Rd2 d1, W 3 Rd3 d2 consist of i.i.d. standard normal entries, where d1 = 2000, d2 = d3 = 1000. |