Yizhen Zhang

Postdoctoral Scholar

About Me

I am a postdoctoral scholar in the Chang lab at the University of California, San Francisco (UCSF), supervised by Dr. Edward Chang.
I received my Ph.D. from the University of Michigan in June, 2021. During my Ph.D., I was a member of Laboratory of Integrated Brain Imaging (LibiLab), advised by Prof. Zhongming Liu. My research interests are in the neural encoding/decoding of naturalistic stimuli, multimodal learning, and brain-inspired artificial intelligence.

News

2024 July: I received Postdoctoral Abstract Merit Award from the SNL 2024 for abstract entitled "Neural encoding of word forms from continuous speech in human cortex".
2021 July: I joined Chang lab as a postdoctoral scholar.
2021 June: I successfully defended my Ph.D. dissertation titled "Grounding Language Learning in Vision for Artificial Intelligence and Brain Research" in electrical and computer engineering at the University of Michigan.
2020 November: I gave an invited talk about neural encoding and decoding natural vision and language for Deep Learning meets Neuroscience Lecture Series at Trinity College Dublin.
2018 March: Received MERIT abstract awards from the OHBM for abstract entitled "High gamma electrocorticography in superior temporal gyrus represents words during natural speech".

Research Projects

Visually Grounded Language Learning

Most models for natural language processing learn words merely from texts. However, humans learn language by referring to real-world experience and knowledge. My research aims to ground language learning in visual perception, taking one step closer to making machines learn language like humans. To achieve this goal, I have designed a two-stream model with deep neural networks. One stream extracts image features. The other stream extracts language features. The two streams merge to connect image and language features in a joint representation space. By contrastive learning, I have first trained the model to align images with their captions, and then refined the model to retrieve visual objects with language queries and infer their visual relations. After training, the model’s language stream is a stand-alone system capable of embedding words in a visually grounded semantic space. This space manifests principal dimensions explainable with human intuition and neurobiological knowledge. The visually grounded language model also enables compositional language understanding based on visual knowledge and multimodal image search with queries based on image-text combination. This model can also explain human brain activity observed with functional magnetic resonance imaging during natural language comprehension. It sheds new light on how the brain stores concepts and organizes concepts by their semantic relations and attributes.

Connecting Concepts in the Brain: Mapping Cortical Representations of Semantic Relations through Word2vec Encoding Model

In the brain, the semantic system is thought to store concepts. However, little is known about how it connects different concepts and infers semantic relations. To address this question, we collected hours of functional magnetic resonance imaging (fMRI) data from human subjects listening to natural stories. We developed a predictive model of the voxel-wise response, and further applied it to thousands of new words. We found that both semantic categories and relations were represented by spatially overlapping cortical patterns, instead of anatomically segregated regions. Importantly, semantic relations that reflected conceptual progression from concreteness to abstractness were represented by similar cortical patterns of activation in the default mode network and deactivation in the frontoparietal attention network. Our results suggest that the human brain represents a continuous semantic space and uses distributed networks to encode not only concepts but also relationships between concepts. In particular, the default mode network plays a central role in semantic processing for abstraction of concepts.

Musical Imagery Involves Wernicke’s Area in Bilateral and Anti-correlated Network Interactions in Musicians

Musical imagery is the human experience of imagining music without actually hearing it. The neural basis of this mental ability is unclear, especially for musicians capable of engaging in accurate and vivid musical imagery. Here, we created a visualization of an 8-minute symphony as a silent movie and used it as real-time cue for musicians to continuously imagine the music for repeated and synchronized sessions during functional magnetic resonance imaging (fMRI). The activations and networks evoked by musical imagery were compared with those elicited by the subjects directly listening to the same music. Musical imagery and musical perception resulted in overlapping activations at the anterolateral belt and Wernicke’s area, where the responses were correlated with the auditory features of the music. Whereas Wernicke’s area interacted within the intrinsic auditory network during musical perception, it was involved in much more complex networks during musical imagery, showing positive correlations with the dorsal attention network and the motor-control network and negative correlations with the default-mode network. Our results highlight the important role of Wernicke’s area in forming vivid musical imagery through bilateral and anti-correlated network interactions, challenging the conventional view of segregated and lateralized processing of music versus language.

Page updated

Google Sites

Report abuse