AUD25: Auditory Decoding

Focus and Goals

In 2025 the auditory group will endeavor to decode the brain, combining unsolved scientific questions and directing our knowledge towards novel brain-computer interfaces. We want to understand how the brain responds to external stimuli and build mathematical models to describe the results. We want to use the measurements to answer scientific questions about how the brain works, as well as investigate real-time brain-computer interfaces. The BCI work, in particular, is a new focus for the auditory work, and good for demos and public displays.

A central theme of the proposed effort this year is the topic of neural decoding: i.e. translating neural activity into meaningful information. We want to understand how sensory signals are represented; how they guide perception and behavior; and how we can close the loop from decoding these signals to control devices or inform actions.

Our Telluride work this year will emphasize musical signals, since the complexity of music can be tuned much easier than we can with language.  Thus we can test acoustic signals as simple as tone sequences, to instruments, to singing voices to full symphonies. There are a number of pilot experiments which we can perform in Telluride.  These include:

• Comparing music vs. speech decoding (most attention decoding work has been with speech).

• Comparing linear vs. non-linear decoding methods (DNN approaches have shown only slight improvement compared to linear methods in contrast to other problem areas where DNN have shown substantial improvements).

• Explainable decoding, which is useful for both scientific purposes and to find efficient solutions for real-time work.

• Decoding motor signals, both to understand where they come from in the brain, perhaps for imagined signals, as well as to remove them from other paradigms.

• Saliency and foundational models for low-SNR domains, such as brain science.

All of our work will be oriented towards real-time demos. We are especially interested in building a robust real-time toolkit for EEG analysis and processing.  Telluride’s BCI work will be our test case, across multiple types of hardware.

Keywords: Audio perception, brain decoding, speech and music perception, real-time, EEG, BCI, Machine Learning

Materials and Equipment:

We will bring several types of hardware to Telluride, including the normal 64 channel wet-electrode recording system, and mobile dry electrode systems. 

Tutorials:

In order to ensure successful engagement of Telluride participants in the activities of the group, we anticipate a number of tutorials that will benefit the workshop, including:

• Basic auditory processing, for cochlea to cortex

A tutorial on basic auditory processing from the cochlea to the cortex will provide a comprehensive overview of how sound is transformed into neural signals and processed by the brain. Beginning with the cochlea, the tutorial would explain its role as the auditory system's mechanical transducer, converting sound waves into electrical signals through the movement of hair cells. The tutorial would also cover relay stations including the brainstem, discussing models of signal timing and intensity coding. Finally, we will explore the auditory cortex and discuss modeling of temporal and spectral modulations.

• Basic decoding: linear to DNNs

Common methods used to explore brain representations in the auditory system are temporal transfer functions which provide a mathematical description of how auditory neurons respond to changes in sound intensity or frequency over time. These functions are crucial for understanding auditory perception because they can model the way different parts of an auditory signal, such as the onset, offset, and sustained phases of a sound. By examining these temporal dynamics, researchers can infer how sounds are represented in the brain, and how these representations contribute to our perception of sound patterns and specifically speech. A newer body of work exploring deep learning offers a different perspective, particularly in terms of adaptively adjusting parameters during training, allowing them to model intricate dependencies and variations in auditory signals that linear models might miss. With all their computational prowess, these methods are not showing great improvements. The tutorial will explore the general framework for decoding methods.

• Differentiable neural modeling

This tutorial explores the emerging field of bio-physically plausible, differentiable neural modeling, a hybrid approach combining precise mathematical modeling with adaptability of deep learning. Differentiable neural models integrate traditional differential equations into neural network architectures, allowing these networks to not only learn from data but also to incorporate established scientific principles. The tutorial will discuss foundations of differentiable modeling including automatic differentiation, vectorization, and just-in-time compilation; as well as tools commonly used for differentiable computing including JAX.