Responsible Speech Foundation Models II

Speech foundation models are emerging as a universal solution to various speech tasks. Indeed, their superior performance has extended beyond ASR. For instance, Whisper has proven to be a noise-robust audio event tagger [1], showcasing its potential beyond its original training objectives. However, the limitations and risks associated with foundation speech models have not been thoroughly studied. For example, foundation models have been found to exhibit biases in different paralinguistic features, emotions [2], accents [3, 4], as well as noise [5]. Besides, foundation models present challenges in terms of ethical concerns, including privacy, sustainability, fairness, safety [6], and social bias [7]. The responsible use of speech foundation models has attracted increasing attention not only in the speech community but also in the language community, including organizations like OpenAI [8].

Thus, it is necessary to investigate speech foundation models for de-biasing (e.g., consistent accuracy for different languages, different genders, and ages), enhancing factuality (not making mistakes in critical applications), and not used for malicious applications (e.g., using a TTS to attack speaker verification systems, not to use for surveillance).

In this special session, we aim to look beyond individual areas in regular sessions such as speech recognition, paralinguistics, or speech synthesis, which lack a particular focus on foundation models, especially their limitations and risks; To concern about the risks of foundation models and speech-LLMs recently emerging globally [9]; To catch up with NLP/ML community by addressing responsible foundation models within the speech community; And to draw attention from various speech areas to exchange ideas on this emerging topic.

*Speech foundation models are broadly defined, including emerging speech-LLMs.