Language Models and Brain Alignment: Brain Encoding and Decoding 

The past two decades has seen a steady increase in studies investigating the brain encoding and decoding models. 

• Encoding models aim at how brain represents stimulus information. They have several practical applications in evaluating and diagnosing neurological conditions and thus also help design therapies for brain damage.

• Decoding models solve the inverse problem of reconstructing the stimuli given the brain recordings. They are useful for designing brain-machine or brain-computer interfaces. Inspired by the effectiveness of deep learning models for natural language processing, computer vision, and speech, recently several neural encoding and decoding models have been proposed. 

• The latest ones leverage GPT-3, Wav2Vec2.0 and Stable Diffusion for processing text, speech and images respectively. 

In this tutorial, we plan to discuss different kinds of stimulus representations, and popular encoding and decoding architectures in detail. The tutorial will provide a working knowledge of the state of the art methods for encoding and decoding, a thorough understanding of the literature, and a better understanding of the benefits and limitations of encoding/decoding with deep learning.

[1] Wehbe et al. 2014, Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses, PlusOne 2014. 

[2] Deniz et al. 2019, The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality, Journal of Neuroscience 2019. 

[3] Toneva et al. 2019, Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain), NeurIPS 2019. 

[4] Schrimpf et al. 2022, The neural architecture of language: Integrative modeling converges on predictive processing, PNAS 2022.  

[5] Caucheteux et al. 2022, Brains and algorithms partially converge in natural language processing, Communications biology 2022.

[6] Oota et al. 2022, Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?, NAACL 2022. 

[7] Millet et al. 2023, Toward a realistic model of speech processing in the brain with self-supervised learning, NeurIPS 2023. 

[8] Antonello et al. 2024, Scaling laws for language encoding models in fMRI, NeurIPS 2024. 

[9] Oota et al. 2024, Joint processing of linguistic properties in brains and language models, NeurIPS 2024.