Generalized Orders of Magnitude for Scalable, Parallel, High-Dynamic-Range Computation
Authors: Franz A. Heinsen, Leo Kozachkov
TMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate that our implementation of GOOMs outperforms traditional approaches with three representative experiments... (1) compounding real matrix products... (2) estimating spectra of Lyapunov exponents... (3) capturing long-range dependencies in deep recurrent neural networks..." and Section 4 "Representative Experiments" |
| Researcher Affiliation | Industry | Franz A. Heinsen EMAIL Glass Room Software LLC Leo Kozachkov EMAIL Thomas J. Watson Research Center, IBM Research |
| Pseudocode | Yes | Section 5: Selective-Resetting Method for Parallel Scans of Linear Recurrences describes a method including a structured block: '// first, selective resetting: If S A prev = 1 and B prev = {0}d d : B prev R A prev A prev {0}d d // then, ordinary recurrence: A curr A curr A prev B curr A curr B prev + B curr' |
| Open Source Code | Yes | Source code for replicating our experiments is available at github.com/glassroom/generalized_orders_of_magnitude. |
| Open Datasets | Yes | We test our parallel algorithm on all dynamical systems from a dataset spanning multiple scientific disciplines, including astrophysics, climatology, and biochemistry (Gilpin, 2023a;b)... We train and test instances of our RNN on several tasks, including (but not limited to) generative language modeling on The Pile (Gao et al., 2020), classification and generation of pixel sequences from MNIST (Le Cun et al., 2010), and a Copy Memory task. |
| Dataset Splits | No | The paper mentions using datasets like 'The Pile (Gao et al., 2020)', 'MNIST (Le Cun et al., 2010)', and 'a dataset spanning multiple scientific disciplines (Gilpin, 2023a;b)', but does not explicitly provide specific details on how these datasets were split into training, validation, and test sets within the main text. |
| Hardware Specification | No | The paper mentions running experiments on 'a recent Nvidia GPU' and 'a single Nvidia GPU' (Figures 1, 3, Section 4.1, 4.2), and also on 'a recent multi-core CPU' (Section 4.1). However, it does not provide specific model numbers or detailed specifications for these hardware components. |
| Software Dependencies | No | The paper states, 'We implement GOOMs for Py Torch, a widely used software framework for parallel computation...' and mentions 'Triton (Tillet et al., 2019)'. However, it does not provide specific version numbers for PyTorch, Triton, or any other software dependencies. |
| Experiment Setup | Yes | For matrix product experiments: 'sampling matrix elements independently from N(0, 1)' and 'chains of up to 1M random normal square matrices of size ranging from 8x8 to 1024x1024' (Section 4.1). For RNN experiments: 'a 124M-parameter RNN incorporating a 50257 token-id vocabulary and 24 layers; we stopped training at 10B tokens' and 'a 12.8M-parameter RNN incorporating a 256 token-id vocabulary and 8 layers' (Figure 4). |