Unveiling functions of the visual cortex
using task-specific deep neural networks
Kshitij Dwivedi, Michael F. Bonner, Radoslaw Martin Cichy, Gemma Roig
PLOS Computational Biology 2021
Paper/ Code / Data
Abstract
The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.
Our Approach
We generate a functional map of the visual cortex following the below steps:
Extract activations from multiple DNNs and responses of a cortical region for all the images in the stimulus set
Create representational dissimilarity matrices (RDMs) by computing pairwise distance between DNN activations (or fMRI responses) of all the images
Predicting fMRI RDM from DNN RDM using a linear regression
Highlighting the cortical region by color code corresponding to the best predicting DNN
Functional map of visual cortex using DNNs performing distinct functions related to scene perception
Color map according to task type
Map highlighting best predicting DNN for different regions in the visual cortex. Regions with low signal-to-noise ratio are not visualized.
Use your mouse to zoom in or rotate the map.
To view a larger interactive map click here
Diverging color map for better visualization
Map highlighting best predicting DNN for different regions in the visual cortex using a diverging color map for better visualization. Regions with low signal-to-noise ratio are not visualized.
Use your mouse to zoom in or rotate the map.
To view a larger interactive map click here
Functional map of visual cortex using 2D, 3D, and semantic DNNs
Map highlighting grouped DNN that explained significantly higher variance of a region uniquely than others. Regions with low signal-to-noise ratio are not visualized
Use your mouse to zoom in or rotate the map.
To view a larger interactive map click here
Anatomical ROIs
Map highlighting anatomical ROIs from Kastner Atlas investigated in this paper.
Reference:
Wang, L., Mruczek, R.E., Arcaro, M.J. and Kastner, S., 2015. Probabilistic maps of visual topography in human cortex. Cerebral cortex, 25(10), pp.3911-3931.
Tasks from Taskonomy (Zamir et al., CVPR 2018)
A demonstration of Taskonomy Tasks
Original source: http://taskonomy.stanford.edu/
Reference:
Zamir, A. R., Sax, A., Shen, W., Guibas, L. J., Malik, J., & Savarese, S. (2018). Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3712-3722).