On the expressiveness and spectral bias of KANs
Authors: Yixuan Wang, Jonathan Siegel, Ziming Liu, Thomas Hou
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We theoretically consider a single KAN layer and show that it does not suffer from the spectral bias by analyzing gradient descent for minimizing a least squares objective. Although our analysis is necessarily highly simplified, we argue that it provides some evidence and intuition showing that KANs will have a reduced spectral bias compared with MLPs. Finally, we study the spectral bias of KANs experimentally on a wide variety of problems, including 1D frequency fitting, fitting samples from a higher-dimensional Gaussian kernel, and solving the Poisson equation with a high-frequency solutions. |
| Researcher Affiliation | Academia | Yixuan Wang1,+, Jonathan W. Siegel2,+ Ziming Liu3,4 Thomas Y. Hou1 1 California Institute of Technology 2 Texas A&M University 3 Massachusetts Institute of Technology 4 The NSF Institute for Artificial Intelligence and Fundamental Interactions |
| Pseudocode | No | The paper describes algorithms and methods but does not provide any structured pseudocode or algorithm blocks. |
| Open Source Code | No | We implement these numerical experiments using the pykan package version 0.2.5. |
| Open Datasets | No | In the first example, we take the same setting as in Rahaman et al. (2019) and study the regression of a linear combination of waves of different frequencies. Consider the function prescribed as f(x) = X Ai sin (2πkiz + φi) , k = (5, 10, , 45, 50). ... In this example, we consider fitting functions sampled from a Gaussian random field. The target function f is sampled from a d-dimensional Gaussian random field with mean zero and covariance exp( |x y|2/(2σ2)). ... In this example, we solve the 1D Poisson equation with a high-frequency solution, similar to Xu et al. (2019a). |
| Dataset Splits | Yes | For f with different scales σ and dimensions d, we split the points into 80% training and 20% testing points. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, memory) are provided for running the experiments. |
| Software Dependencies | Yes | We implement these numerical experiments using the pykan package version 0.2.5. |
| Experiment Setup | Yes | For MLPs, we train with 80000 iterations as in Rahaman et al. (2019); for KANs, we only train with 8000 iterations. Normalized magnitudes of discrete Fourier transform at frequencies ki are computed as | fki/Ai| and averaged over 10 runs of different phases. ... For MLPs, we use 500 iterations of LBFGS iteration, and for KANs, we use the grid extension technique, with grid sizes (10, 20, 30, 40, 50), each trained with 100 iterations of LBFGS. ... We choose the hyperparameter λ = 0.01 balancing the energy and boundary loss and perform LBFGS iterations. For MLPs, we use 200 iterations, and for KANs, we use grid sizes (20, 40), each trained with 100 iterations. |