Fundamental Limits of Visual Autoregressive Transformers: Universal Approximation Abilities
Authors: Yifang Chen, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song
ICML 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | Our primary contributions establish that, for single-head VAR transformers with a single self-attention layer and single interpolation layer, the VAR Transformer is universal. From the statistical perspective, we prove that such simple VAR transformers are universal approximators for any word-to-image Lipschitz functions. Furthermore, we demonstrate that flow-based autoregressive transformers inherit similar approximation capabilities. Our results provide important design principles for effective and computationally efficient VAR Transformer strategies that can be used to extend their utility to more sophisticated VAR models in image generation and other related areas. |
| Researcher Affiliation | Academia | 1The University of Chicago 2University of New South Wales 3The University of Hong Kong 4University of Wisconsin Madison 5University of California, Berkeley. Correspondence to: Zhao Song <EMAIL>. |
| Pseudocode | Yes | Algorithm 1 High-Order Flow AR Training (Liang et al., 2025b) ... Algorithm 2 High-Order Flow AR Inference (Liang et al., 2025b) |
| Open Source Code | No | The paper makes no explicit mention of providing open-source code for the methodology described. |
| Open Datasets | No | The paper focuses on theoretical analysis and does not conduct empirical studies using specific datasets that would require public access information. |
| Dataset Splits | No | The paper is theoretical and does not involve empirical experiments with datasets, thus no dataset splits are mentioned. |
| Hardware Specification | No | The paper is theoretical and does not describe any experimental setup or mention specific hardware used. |
| Software Dependencies | No | The paper is theoretical and does not describe any software implementation or list specific software dependencies with version numbers. |
| Experiment Setup | No | The paper focuses on theoretical proofs and analysis, without detailing any experimental setup, hyperparameters, or training configurations. |