Crown

---

Title: Simultaneously Active Brain Networks Cooperate and Compete to Determine Language Comprehension in Real-World Contexts

Idea: Jeremy I. Skipper, Sarah Kane, and Jason Zevin

Abstract:

Communication is accompanied by an abundance of contextual information relevant to speech perception and language comprehension, including both sensory information external to the listener (so-called “non-verbal” behavior, e.g., observed speech-associated mouth movements and gesture) and knowledge or expectations internal to the listener (e.g., those associated with the meaning of words). Most behavioral and neurobiological language research, however, discards context in favor of studying isolated speech sounds or words. We test an alternative perspective that the brain readily makes use of context and that there are multiple brain networks associated with different types of context (cooperation) whose relative weightings fluctuate as a function of their informativeness (competition). To test this model, participants will undergo dEEG/fMRI while watching an actress speaking sentences. The sentences vary on informativeness of mouth movements (more or less visible), gesture (none or more or less imagistic), and discourse content (high or low predictability of the sentence final word). We predict that mouth movements, gesture, and discourse content will each be associated with its own network and that increased informativeness of one of these sources of information will result in stronger weighting of the network associated with that source relative to the others. Thus, when mouth movements are informative with respect to phoneme identification, sensory-motor networks will be weighted more strongly than sensory-semantic networks. Increased informativeness of gesture or discourse content will increase weighting of associated sensory-semantic relative to sensory-motor networks. These results would suggest that a nearly identical interpretation of a sentence could be reached by different brain networks depending on available context and that, more generally, a full understanding of the organization of language and the brain requires that we study the brain in more real-world settings in which context is available.

Cups

---

Title: Dynamic Functional Reorganization of the Brain when Observed Individual Actions become Joint Actions

Idea and Pilot: Jeremy I. Skipper and Natalie Sebanz

Replication and Extension: Jamie Azdair and Jeremy I. Skipper

Abstract:

How does the brain keep track of the individual contributions to joint action so that we can understand social interactions? We have shown that when observing individual actions, e.g., watching someone pick up a cup to pass it, processing in action production areas is devoted to the person performing the action. During the observation of joint action, i.e., watching two people exchanging the cup, activity in these areas split into separable dorsal and ventral processing streams for each person. But why do these streams split in this manner? We hypothesize that the brain keeps track of the person whose actions are most easily mapped onto the observer’s own motor repertoire in the ventral processing stream (e.g., the actor over whose shoulder the joint action is being observed). In contrast, observed actions that can only be indirectly mapped onto motor programs are processed in the dorsal processing stream. This suggests that the actor represented in the ventral stream is more closely associated with the “self” and will, therefore, be liked more. We will test these hypotheses by replicating the aforementioned study with the addition that we manipulate camera position and passing direction to control the degree to which observed actions can be mapped onto the participants’ own motor repertoires. We will test liking of actors as a function of the degree of this match and amount of activity in the ventral processing stream. Confirmatory results would suggest that dynamic functional reorganization of the brain into spatially separable networks allows us to keep track of individual contributions during social interactions and, at least in part, arbitrarily dictates our feelings about the individuals involved in those interactions.

Facial

---

Title: Emotions Speak Louder, Then Words: Non-Verbal Emotional Cues Aid in Predicting Forthcoming Words

Idea and Pilot: Jeremy Safran (2011) and Jeremy I. Skipper

Replication:  Amanda O’Brien

Abstract:

The study of the organization of language and the brain has proceeded almost exclusively as if non-verbal expressions of emotion are independent of verbal communication. This is odd given that both forms of communication are often simultaneously used to express how we feel. We will test the theoretical perspective that no clear division between non-verbal and verbal communication can be drawn because these process are interactive. In particular, we test the hypothesis that non-verbal emotional displays are intrinsically associated with verbal semantic content that can be used to predict forthcoming verbal information. Participants will watch videos of two people having a brief conversation. One talker will display an emotion on the face (e.g., a smile) or maintain a neutral face. The other talker will then ask a question (e.g., “How did the interview go?”). The reply, with a neutral face, will be more or less predictable based on the prior facial display (e.g., “It went well”). We predict that brain activation for the sentence frame (i.e., “It went”) will demonstrate an early activation of the amygdala and auditory and motor cortices when initial facial displays have valence relative to neutral faces. This will be followed by a subsequent reduction in activity in those areas at the time of the final word (i.e., “well”). Taken together, these results would suggest that the brain pre-activates a neighborhood of words associated with the valence of emotional content that can be used to predict subsequent words. This prediction frees up brain resources that might be used for other purposes. This further suggest that a full understanding of the organization of language and the brain requires that we study the brain in more real-world settings in which non-verbal and verbal forms of communication naturally interact.

Mosey

---

Title: Yet to be assigned a fancy name

Idea: Jeremy I. Skipper, Matt Longo, Lily Jampol

Abstract:

It has been historically assumed that word meaning is abstractly encoded.  Recent "embodied cognition" research on action verbs suggests that the motor plan for the movement suggest by a verb may actually be encoded by the brain as part of the word representation (Pulvermuller, 1999).  Thus, the action verb ‘run’ may actually be encoded by simulating running.  Similarly, the noun "hammer" may be encoded by simulating the specific movements associated with it (e.g. moving the arm to strike a nail).  The hypothesis of the current study is that words associated with a particular effector will yield faster reaction time when the effector suggested by the words meaning is congruent with the response effector.  For example, the word ‘point’ should yield a faster reaction time with the hand relative to the foot.  This particular hypothesis will help to understand whether or not word meaning is abstractly encoded or whether word meaning is encoded in a more embodied manner.  This in turn may help us better understand speech perception. 

Rereading

---

Title: Cortical Re-Instatement of Reading Networks During Auditory Sentence Comprehension

Idea and Pilot: Jane Hannon (2011) and Jeremy I. Skipper

Replication: Alexandra Arenson & Charlotte Cosgrove

Abstract:

Most models of the organization of language and the brain maintain that auditory language comprehension relies on brain areas near primary auditory cortex (e.g., “Wernicke's Area”). An alternative is that language in the brain is dynamic and relies on many simultaneously active cooperating and competing networks distributed throughout the whole brain. The extent to which these networks contribute to comprehension is determined by the information in the environment and listeners prior experience with that information. This leads to the hypothesis that auditory language comprehension can rely on networks outside of auditory cortex if listeners’ primary experience with heard speech is in another sensory modality. We demonstrate that auditory language comprehension for previously unheard sentences does indeed rely on brain areas near auditory cortex. However, when sentences are heard and they had previously been encoded when participants could see the actor, brain areas near auditory and motor cortex support comprehension. In contrast, when these same sentences are heard (by a different group of participants) and they had previously been encoded through reading, auditory language comprehension involves mostly visual cortex (including the “Visual Word Form Area”). Results suggest that a re-visioning of existing models of language and the brain is required so that they correspond to a more dynamic and distributed model.

Resinging

---

Title: Resigning in the Brain: Cortical Re-Instatement of Words While Listening to Instrumental Music

Idea: Sam Briggs and Jeremy I. Skipper

Abstracts:

Instrumental music influences how we think and feel in a way that has eluded scientific description. One possible mechanism is that lyrics previously associated with music may be unconsciously reactivated at a later time by the same or similar sounding music in the absence of those lyrics. These activations would then influence emotions and thinking by biasing listeners toward the semantic associations of those words. To test this theory, participants will undergo dEEG/fMRI while listening to unfamiliar music paired with either sung lyrics or the same lyrics displayed visually on a screen (e.g., music with the lyrics “I was sailing on the sea”). Later, the same music will be played without the accompanying words while participants listen to spoken sentences that contain final words that are semantically related to but not contained in the lyrics that originally accompanied the music (e.g., “She thought about the water”). Alternately, these sentences will be paired with previously unheard music. It is predicted that, when lyrics accompanying the music had been previously heard, the auditory system will be more active for semantically related final words when compared to the same words accompanied by unheard music. In contrast, if the words accompanying the music had been read, the visual system is predicted to be more active. These results would demonstrate that instrumental music could cortically reinstate words that previously accompanied that music and that these words can then activate other related words. Thus, instrumental music, through a lifetime of lyrical associations, could come to have complex effects on how we think, feel, and even make decisions, something elevator companies and advertisers have intuitively known for some time.

Um

---

Title: Not yet assigned a fancy name.

Authors: Jeremy I. Skipper and Jason Zevin

Status:

Abstract:

|See more detailed information|