Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1]
TopoNets: High performing vision and language models with brain-like topography
Authors: Mayukh Deb, Mainak Deb, Apurva Murty
ICLR 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We validate our method on both vision (Res Net-18, Res Net-50, Vi T) and language models (GPT-Neo-125M, Nano GPT), collectively Topo Nets. Topo Nets are the highest performing supervised topographic models to date, exhibiting brain-like properties such as localized feature processing, lower dimensionality, and increased efficiency. |
| Researcher Affiliation | Academia | Mayukh Deb1,2, Mainak Deb3, N. Apurva Ratan Murty1,2 1Cognition and Brain Science, School of Psychology, Georgia Tech 2Center for Excellence in Computational Cognition, Georgia Tech 3Independent Contributor EMAIL |
| Pseudocode | No | The paper defines the Topo Loss mathematically in Section 2.2 but does not present it or any other procedure in a structured pseudocode or algorithm block format. Steps are described in narrative text. |
| Open Source Code | Yes | Our project page: https://toponets.github.io. The components used for implementing the sheet can be found in our source code on github: Determining the height and width of the cortical sheet: link Obtaining the cortical sheets for convolutional and linear layers: link |
| Open Datasets | Yes | For language, we trained GPT-Neo-125M models (Black et al., 2021) on the Wikipedia Dataset (Wikimedia Foundation) and Nano GPT models Karpathy (2022) on 10 billion tokens from Fine Web Edu (Lozhkov et al., 2024). For vision, we trained topographic Res Net-18... All vision models were trained on a supervised 1000-way classification task on Image Net (Deng et al., 2009). |
| Dataset Splits | Yes | For Res Nets, we followed the procedure outlined in (Margalit et al., 2024). We chose 20,000 images from the Image Net validation set calculated the effective dimensionality of the features for all the convolutional layers. For language models, we chose 8192 samples from the openwebtext dataset and measured dimensionality of the representations from the topographic (c fc) layers. |
| Hardware Specification | No | The paper mentions "GPU-accelerated data augmentation" and "efficient binary format and leveraging multiprocessing" facilitated by ffcv, but does not specify any exact GPU models, CPU models, or other detailed hardware specifications used for their experiments. |
| Software Dependencies | No | We trained our models using the ffcv (Leclerc et al., 2023) training recipe. ffcv (Fast Forward Computer Vision) significantly accelerates model training by replacing traditional data loaders with an efficient binary format and leveraging multiprocessing and GPU-accelerated data augmentation to optimize data pipelines. We followed the recipe provided by Torch Vision maintainers & contributors (2016). Our model can incorporated in only 2 to 3 lines of additional code (pip install topoloss). |
| Experiment Setup | Yes | We trained 8 distinct Res Net-18 (He et al., 2016) models from scratch on the Image Net (Deng et al., 2009) dataset across various topographic configurations: one baseline model (no topography), six Topo Nets with different topographic scaling factors: τ = 0.5, 1, 5, 10, 20, 50. We applied Topo Loss with τ = 10 on the last MLP module i.e the mlp.3 module in each transformer block. We applied Lo RA fine-tuning on our Topo Net-Nano GPT models (τ = 1, 50) and compared the Lo RA fine-tuning performance with the baseline Nano GPT model (without topography). The experiments were conducted on multiple Lo RA ranks (2, 4, and 8) to ensure generalizability. |