Home

Unveiling functions of the visual cortex

using task-specific deep neural networks

Kshitij Dwivedi, Michael F. Bonner, Radoslaw Martin Cichy, Gemma Roig

PLOS Computational Biology 2021

Paper/ Code / Data

Abstract

Approach

Functional Map (Individual)

Functional Map (Grouped)

Tasks

Abstract

The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.

Our Approach

We generate a functional map of the visual cortex following the below steps:

Extract activations from multiple DNNs and responses of a cortical region for all the images in the stimulus set
Create representational dissimilarity matrices (RDMs) by computing pairwise distance between DNN activations (or fMRI responses) of all the images
Predicting fMRI RDM from DNN RDM using a linear regression
Highlighting the cortical region by color code corresponding to the best predicting DNN

Functional map of visual cortex using DNNs performing distinct functions related to scene perception

Color map according to task type

Map highlighting best predicting DNN for different regions in the visual cortex. Regions with low signal-to-noise ratio are not visualized.

Use your mouse to zoom in or rotate the map.

To view a larger interactive map click here

Diverging color map for better visualization

Map highlighting best predicting DNN for different regions in the visual cortex using a diverging color map for better visualization. Regions with low signal-to-noise ratio are not visualized.

Use your mouse to zoom in or rotate the map.

To view a larger interactive map click here

Functional map of visual cortex using 2D, 3D, and semantic DNNs

Map highlighting grouped DNN that explained significantly higher variance of a region uniquely than others. Regions with low signal-to-noise ratio are not visualized

Use your mouse to zoom in or rotate the map.

To view a larger interactive map click here

Anatomical ROIs

Map highlighting anatomical ROIs from Kastner Atlas investigated in this paper.

Reference:

Wang, L., Mruczek, R.E., Arcaro, M.J. and Kastner, S., 2015. Probabilistic maps of visual topography in human cortex. Cerebral cortex, 25(10), pp.3911-3931.

Tasks from Taskonomy (Zamir et al., CVPR 2018)

A demonstration of Taskonomy Tasks

Original source: http://taskonomy.stanford.edu/

Reference:

Zamir, A. R., Sax, A., Shen, W., Guibas, L. J., Malik, J., & Savarese, S. (2018). Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3712-3722).

Page updated

Google Sites

Report abuse