LEA: Learning Latent Embedding Alignment Model for fMRI Decoding and Encoding
Authors: Xuelin Qian, Yikai Wang, Xinwei Sun, Yanwei Fu, Xiangyang Xue, Jianfeng Feng
TMLR 2024 | Venue PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through extensive experimentation on BOLD5000 and GOD benchmarks, we demonstrate that LEA exhibits both efficiency and effectiveness in f MRI decoding and encoding. |
| Researcher Affiliation | Academia | Xuelin Qian EMAIL School of Data Science Fudan University Yikai Wang EMAIL School of Data Science Fudan University Xinwei Sun EMAIL School of Data Science Fudan University Yanwei Fu EMAIL School of Data Science Fudan University Xiangyang Xue EMAIL School of Computer Science Fudan University Jianfeng Feng EMAIL Institute of Science and Technology for Brain-inspired Intelligence Fudan University |
| Pseudocode | No | The paper describes the methodology in prose, detailing the architecture and alignment process, but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | The codes are available at https://github.com/naiq/LEA. |
| Open Datasets | Yes | We validate our LEA on Brain, Object, Landscape Dataset (BOLD5000) (Chang et al., 2019) and Generic Object Decoding Dataset (GOD) (Horikawa & Kamitani, 2017). |
| Dataset Splits | Yes | (1) BOLD5000 encompasses a substantial f MRI-image dataset with a total of 5254 f MRI-image stimulus trials, involving four subjects. This dataset includes 1000, 2000, and 1916 images sourced from the Scene, COCO, and Image Net datasets, respectively. Following the protocol in (Chen et al., 2022), we choose 4, 803 images presented on a single trial for training, and the remaining 113 images for testing. (2) The GOD comprises data from 5 subjects, each viewing 1,250 images spanning 200 categories, resulting in a total of 1,250 f MRI-image pairs. We split the training set with 1200 pairs from 150 categories, and the other non-overlapping 50 classes are used for testing. |
| Hardware Specification | No | The computations in this research were performed using the CFFF platform of Fudan University. |
| Software Dependencies | No | LEA is implemented using Py Torch. |
| Experiment Setup | Yes | For fine-tuning both models, we use the Adam W optimizer with hyper-parameters β1 = 0.9, β1 = 0.95, and a batch size of 8. The initial learning rate is set to 5e-5 with a weight decay of 0.01. We apply a linear learning rate schedule, gradually reducing the learning rate until it reaches the minimum value. The total number of training iterations is 100K and 300K for the f MRI and image reconstruction models, respectively. The encoder and decoder have a depth of 24/8 with dimensions of 1024/512, and there are 16 multi-heads. For fitting linear models, we adopt ridge regression from the Scipy library. The coefficient of L2 regularization term α is set to 500 for both image-to-f MRI and f MRI-to-image regressions. During the inference of f MRI decoding, the number of steps for Mask GIT is 11 as default, and we generate 5 samples for all evaluations. |