AMC23: Audio-Motor Coupling:
The case of speech and music

Topic leaders

Claire Pelofi | NYU | cp2830@nyu.edu
Vikash Gilja | UCSD | vgilja@eng.ucsd.edu

Vikash Gilja (UCSD)

Claire Pelofi (NYU)

Goals

We aim to establish the methodological and conceptual tools to test the hypothesis that speech and music productions rely on essentially the same cog-neuro processes. As a consequence, learning to speak and learning to play an instrument require similar neural mechanisms by which a mapping between motor and auditory areas must be established and reinforced through auditory-motor feedback. The novelty of this topic is to integrate music and speech, providing multiple strategies for evaluating sensory motor integration required for precise acoustic output generation.

Projects

The overall aim is to deploy paradigms to investigate the audio-motor coupling during speech and music production and perception.

Markers of expertise in speech/music production:

During speech and music production, we will introduce some feedback errors to both expert speakers and violin players. We will contrast the neural response locked to the stimulus error between experts and non-experts. We anticipate modulated neural responses captured by EEG, such that experts should elicit larger amplitude in response. This modulated response will be correlated with behavioral and eye-tracking markers of prediction error.

The hypothetical reason is that the predictive signals into the auditory cortex from than expert knowledge store and motor areas is more finely tuned and hence any manipulations in the motor output (that eventually makes it to the ear and auditory cortex) would be more substantial and thus sensitively detected in experts than in novices. The same logic applies to speech perturbations where we compare neural signals between native speakers and novice learners of a language.

Markers of expertise in speech/music perception:

Using linear decoders (CCA, TRF) while experts and non-experts musicians are passively watching excerpts of speech and music production in different languages (English and Arabic) and instruments (piano and violin), the strength of sensory response will be contrasted between experts and novices. We hypothesize that experts, due to a more efficient motor to audio mapping and thus prediction coding, will elicit enhanced encoding of critical temporal information (e.g. note and syllables onsets). A complimentary approach will consist in introducing audio-visual inconsistencies in video excerpts and associate the modulated response time-locked to the incongruencies to level of expertise in speech (arabic vs. english speakers) or music (piano vs. violin playing).

Decoding envelope information during active violin playing

This project aims to develop the tools to propel the processing of EEG data during active behavior. Using dry electrodes (somewhat more immune to movement artifacts) and a set of decoding techniques, we aim to decode, from active violin players, the information of bow movements (up-down-up) that determines the rhythm, hence critical information embedded in the envelope.

Materials, Equipment, and Tutorials:

Hardware: Brain Vision, 64 electrodes, Tobii eye tracker, Cognionics headsets, 3 laptops

Software: Matlab license, Python, SLS (TP)

Relevant Literature:

Poeppel, D., & Assaneo, M. F. (2020). Speech rhythms and their neural foundations. Nature reviews neuroscience, 21(6), 322-334.

Assaneo, M. F., Rimmele, J. M., Sanz Perl, Y., & Poeppel, D. (2021). Speaking rhythmically can shape hearing. Nature Human Behaviour, 5(1), 71-82.

Zatorre, R. J., Chen, J. L., & Penhune, V. B. (2007). When the brain plays music: auditory–motor interactions in music perception and production. Nature reviews neuroscience, 8(7), 547-558.