Exploring the Limitations of Layer Synchronization in Spiking Neural Networks
Authors: Roel Koopman, Amirreza Yousefzadeh, Mahyar Shahsavari, Guangzhi Tang, Manolis Sifalakis
TMLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We present and quantify this problem, and show that models trained with layer synchronization either perform poorly in absence of the synchronization, or fail to benefit from any energy and latency reduction, when such a mechanism is in place. We then explore a potential solution direction, based on a generalization of backpropagation-based training that integrates knowledge about an asynchronous execution scheduling strategy, for learning models suitable for asynchronous processing. We experiment with 2 asynchronous neuron execution scheduling strategies in datasets that encode spatial and temporal information, and we show the potential of asynchronous processing to use less spikes (up to 50%), complete inference faster (up to 2x), and achieve competitive or even better accuracy (up to 10% higher). |
| Researcher Affiliation | Collaboration | Roel Koopman EMAIL Machine Learning Group, Centrum Wiskunde & Informatica (CWI) Amsterdam, The Netherlands Amirreza Yousefzadeh EMAIL Faculty of Electrical Engineering, Mathematics & Computer Science, University of Twente Enschede, The Netherlands Mahyar Shahsavari EMAIL Donders Centre for Cognition, Radboud University Nijmegen, The Netherlands Guangzhi Tang EMAIL Department of Advanced Computing Sciences, Maastricht University Maastricht, The Netherlands Manolis Sifalakis EMAIL Innatera Nanosystems BV Rijswijk, The Netherlands |
| Pseudocode | Yes | Algorithm 1 Vectorized network asynchrony forward pass Input: input spikes sin NNin, previous forward pass time t0 R, current forward pass time t1 R, neuron state u RN at time t0, forward group size F N>0 Output: spike count c NN |
| Open Source Code | Yes | (Source code available at: https://github.com/Roel MK/asynctorch) |
| Open Datasets | Yes | We carried out our experiments on SNN models trained primarily in three common benchmarking datasets, each of them has a different structure: N-MNIST (Orchard et al., 2015), SHD (Cramer et al., 2020), and DVS gestures (Amir et al., 2017). ... We also repeated some of the experiments with a fourth dataset, CIFAR-10 (Krizhevsky, 2012) |
| Dataset Splits | Yes | N-MNIST (Orchard et al., 2015)... It consists of 60000 training samples and 10000 test samples. The Spiking Heidelberg Digits (SHD) dataset (Cramer et al., 2020)... It consists of 8156 training samples and 2264 test samples. The DVS gesture dataset (Amir et al., 2017)... It consists of 1176 training samples and 288 test samples. |
| Hardware Specification | Yes | We also tried to confirm the results in a scaled up setup, namely with a deeper VGG-7 like network (details in A.10), trained on the CIFAR-10 dataset. ... make it feasible on an NVIDIA RTX 5000 |
| Software Dependencies | No | The paper mentions using backpropagation, Adam optimizer, and specific activation functions but does not specify version numbers for any software libraries (e.g., PyTorch, TensorFlow) or their specific versions. |
| Experiment Setup | Yes | Table 6: Network architecture and hyperparameters. The architecture is given as [neurons in hidden layers number of hidden layers] [neurons in output layer]. N-MNIST: Timestep size 10 ms, Batch size 256, Epochs 50, Learning rate 5e-4, Membrane threshold Uthr 0.3, Weight decay constant λW 1e-5, Membrane time constant τm 1 ms, Surrogate steepness α 2, Input spike dropout 0.25, Forward group size F 8, Refractory dropout 0.8, Momentum noise λMS 1e-6 |