Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1]
Locally Connected Echo State Networks for Time Series Forecasting
Authors: Filip Matzner, František Mráz
ICLR 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate LCESN s performance on the NARMA10 benchmark task and compare it to stateof-the-art models on nine real-world datasets. Despite the simplicity of our model and its one-shot training approach, LCESN achieves competitive results, even surpassing several state-of-the-art models. |
| Researcher Affiliation | Academia | Filip Matzner František Mráz EMAIL Charles University, Faculty of Mathematics and Physics |
| Pseudocode | No | The paper describes mathematical recurrence equations (e.g., equations 1 and 2) and presents a diagram of the prediction procedure (Figure 4), but it does not include any explicitly labeled 'Pseudocode' or 'Algorithm' block, nor does it present structured steps in a code-like format. |
| Open Source Code | Yes | We have released our C++/CUDA implementation of LCESN under a permissive license at our Git Hub repository (https://github.com/Floop CZ/echo-state-networks), together with logs, network checkpoints, and fixed random seeds to ensure reproducibility. |
| Open Datasets | Yes | We use the NARMA10 dataset, widely used in ESN literature, along with nine real-world datasets commonly utilized in state-of-the-art sequence forecasting studies. From longest to shortest, these datasets are ETTm1, ETTm2 (Electricity Transformer Temperature), Weather, Solar Energy, Electricity, Traffic, ETTh1, ETTh2, and Exchange. ... We use nine publicly available datasets ETTh1, ETTh2, ETTm1, and ETTm2 proposed by Zhou et al. (2021), weather by Wu et al. (2021), and electricity, traffic, solar-energy, and exchange datasets used by Lai et al. (2018). |
| Dataset Splits | Yes | To be comparable to the state-of-the-art feedforward models in TSF benchmarks, we have used the same training/validation/testing split and the same data normalization technique (Nie et al., 2023). ... See Table 2 for the dataset sizes and consult the source code of the original papers (Python) or ours (C++) for more details. Table 2: Sizes of the real-world datasets measured in the number of time steps. ETTm1/2 Training 34560 Validation 11520 Test 11520 |
| Hardware Specification | Yes | To enable building our model from scratch on consumer hardware, we limit each hyperparameter optimization run to 2000 evaluations, ensuring it fits within a 24-hour deadline on an older NVIDIA GTX 2080 Ti GPU (released in 2018) and a single core of an average desktop CPU. ... The measurements were performed on an NVIDIA GTX 1080 Ti GPU (released in 2017) supported by a single core of an Intel i7-4770 CPU. ... Liu et al. (2024) (i Transformer) provided a quality code, and we were able to reproduce their results on the same hardware (NVIDIA GTX 1080 Ti GPU supported by a single core of an Intel i7-4770 CPU). |
| Software Dependencies | No | The paper mentions that the implementation is in C++/CUDA and refers to the Arrayfire CUDA library (Yalamanchili et al., 2022). However, it does not specify explicit version numbers for C++, CUDA, or Arrayfire, which would be necessary for full reproducibility. |
| Experiment Setup | Yes | We use the hyperparameter tuning framework by Matzner (2022), based on the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) (Hansen and Ostermeier, 2001). ... The optimized hyperparameters include the reservoir scaling parameters σres, µres, the bias term µb, the input scaling parameters σi in for each input, the feedback scaling parameters σj fb for each output (only for sequence-to-sequence tasks), and the ridge regression regularization coefficient λ2 R+. ... Table 5 summarizes the optimized hyperparameters, the corresponding transformations, and their upper and lower bounds. ... The maximum memory length H is fixed at 100 ... Leakage is fixed at 1 ... Sparsity is fixed at 0 ... σnoise is set to zero ... The learning rate for NLMS weight updates µlms is set arbitrarily at 1 10 3. ... We use a reservoir of 40 50 with a 7 7 kernel. |