Benchmarking Predictive Coding Networks -- Made Simple
Authors: Luca Pinchetti, Chang Qi, Oleh Lokshyn, Cornelius Emde, Amine M'Charrak, Mufeng Tang, Simon Frieder, Bayar Menzat, Gaspard Oliviers, Rafal Bogacz, Thomas Lukasiewicz, Tommaso Salvatori
ICLR 2025 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | BENCHMARKING PREDICTIVE CODING NETWORKS MADE SIMPLE In this work, we tackle the problems of efficiency and scalability for predictive coding networks (PCNs) in machine learning. To do so, we propose a library, called PCX, that focuses on performance and simplicity, and use it to implement a large set of standard benchmarks for the community to use for their experiments. All this has allowed us to (i) test architectures much larger than commonly used in the literature, on more complex datasets; (ii) reach new state-of-the-art results in all of the tasks and datasets provided; (iii) clearly highlight what the current limitations of PCNs are, allowing us to state important future research directions. 4 EXPERIMENTS AND BENCHMARKS The benchmark that we propose is a standardized set of models, datasets, and testing procedures that have been consistently used to evaluate predictive coding, but in a non-uniform way. Here, for a comprehensive evaluation, we test models of increasing complexity on multiple computer vision datasets, with both feedforward and convolutional/de-convolutional layers; and multiple learning algorithms present in the literature. |
| Researcher Affiliation | Collaboration | 1Department of Computer Science, University of Oxford, Oxford, UK 2Institute of Logic and Computation, Vienna University of Technology, Vienna, Austria 3MRC Brain Network Dynamics Unit, University of Oxford, UK 4VERSES AI Research Lab, Los Angeles, US Corresponding author: EMAIL |
| Pseudocode | Yes | Figure 1: (a): Generative and discriminative modes; (b): Pseudocode of PC in supervised learning, where both the latent variables hl and the weight parameters θl are updated to minimize the variational free energy F. In the colored boxes, informal description of the different algorithms considered in this work. |
| Open Source Code | Yes | With the hope of galvanizing community efforts towards one of the main open problems in the field, scalability, we release code, tests, and benchmarks.1 1Link to the library: https://github.com/liukidar/pcx Tool. We release an open-source library for accelerated training for predictive coding called PCX. This library runs in JAX (Bradbury et al., 2018), and offers a user-friendly interface with a minimal learning curve through familiar syntax inspired by Pytorch. We also provide extensive tutorials. |
| Open Datasets | Yes | To provide a comprehensive evaluation, we have tested on multiple computer vision datasets, MNIST (Le Cun & Cortes, 2010), Fashion MNIST (Xiao et al., 2017), CIFAR10/100 (Krizhevsky et al., 2009), Celeb A (Liu et al., 2018), and Tiny Image NET (Le & Yang, 2015); on models of increasing complexity, and multiple learning algorithms present in the literature. |
| Dataset Splits | Yes | For data augmentation on the training sets of CIFAR-10, CIFAR-100, and Tiny Image Net, we apply random horizontal flipping with a probability of 50%. Additionally, we employ random cropping with different settings for each dataset. For CIFAR-10 and CIFAR-100, images are randomly cropped to 32 32 resolution with a padding of 4 pixels on each side. In the case of Tiny Image Net, random cropping is performed to obtain 56 56 resolution images without any padding. And on the testing set of Tiny Image Net, we use center cropping to extract 56 56 resolution images, also without padding, since the original resolution of Tiny Image Net is 64x64. |
| Hardware Specification | Yes | We measured the wall-clock time of our PCNs implementation against another existing open-source library (Song, 2024) used in many PC works (Song et al., 2024; Salvatori et al., 2021; 2022; Tang et al., 2023), as well as comparing it with equivalent BP-trained networks (developed also with PCX for a fair comparison). Tab. 4 reports the measured time per epoch, averaged over 5 trials, using a A100 GPU. Training takes 0.40 0.005 seconds per epoch on an Nvidia V100 GPU. Results were obtained on a GTX TITAN X, showing that parallelization is potentially achievable also on consumer GPUs. |
| Software Dependencies | No | Tool. We release an open-source library for accelerated training for predictive coding called PCX. This library runs in JAX (Bradbury et al., 2018), and offers a user-friendly interface with a minimal learning curve through familiar syntax inspired by Pytorch. We also provide extensive tutorials. It is also fully compatible with Equinox (Kidger & Garcia, 2021), a popular deep-learningoriented extension of JAX, ensuring reliability, extendability, and compatibility with ongoing research developments. It also supports JAX s Just-In-Time (JIT) compilation, making it efficient and allowing both easy development and execution of PC networks, gaining efficiency with respect to existing libraries. Compatibility. PCX shares the same philosophy of equinox (Kidger & Garcia, 2021), according to which models are just Py Trees. Consequently, it is fully compatible, using a complete functional approach, with both libraries and many other tools developed for JAX, such as diffrax (Kidger, 2021) and optax (Deep Mind et al., 2020). |
| Experiment Setup | Yes | For every class of experiments, we have performed a large hyperparameter search, and the details needed to reproduce the experiments, as well as a discussion about lessons learned during such a large search, are in the Appendix B and C. Table 7: Hyperparameters search configuration Parameter PC i PC BP Epoch (MLP) 25 Epoch (VGG and Res Net) 50 Batch Size 128 Activation [leaky relu, gelu, hard tanh] [leaky relu, gelu, hard tanh, relu] β [0.0, 1.0], 0.051 βir [0.02, 0.0] lrh (1e-2, 5e-1)2 (1e-2, 1.0)2 lrθ (1e-5, 3e-4)2 (3e-5, 3e-4)2 momentumh [0.0, 1.0], 0.051 weightdecayθ (1e-5, 1e-2)2 (1e-5, 1e-1)2 (1e-5, 1e-2)2 T (MLP and VGG-5) [4,5,6,7,8] T (VGG-7) [8,9,10,11,12] T (VGG-9) [9,10,12,15,18] T (Res Net-18) [6,10,12,18,24] - |