Invited Talks

Abstract: Speech, teeming with rich and hierarchical information, offers different needs for diverse tasks. Certain tasks, like speech recognition, necessitate content extraction while discarding speaker-specific information. Conversely, tasks like speaker recognition demand the extraction of speaker information and dismissal of content. But is it possible to develop a universal speech model capable of addressing a myriad of speech tasks? This talk starts with the introduction of the Speech Processing Universal PERformance Benchmark (SUPERB), a benchmark and leaderboard designed to gauge the performance of Self-Supervised Learning (SSL) models across an extensive array of speech-processing tasks. Empirical results on SUPERB indicate that SSL representations exhibit generalizability across various speech-processing tasks. I will also shed light on the most recent advances and discoveries in the realm of SSL models for speech processing. Lastly, the talk will venture into the prospective directions for further advancements in this field.

Biography: Hung-yi Lee (李宏毅) is an associate professor of the Department of Electrical Engineering of National Taiwan University (NTU), with a joint appointment at the Department of Computer Science & Information Engineering of the university. His recent research focuses on developing technology that can reduce the requirement of annotated data for speech processing (including voice conversion and speech recognition) and natural language processing (including abstractive summarization and question answering). He won Salesforce Research Deep Learning Grant in 2019, AWS ML Research Award in 2020, Outstanding Young Engineer Award from The Chinese Institute of Electrical Engineering in 2018, Young Scholar Innovation Award from Foundation for the Advancement of Outstanding Scholarship in 2019, Ta-You Wu Memorial Award from Ministry of Science and Technology of Taiwan in 2019, and The 59th Ten Outstanding Young Person Award in Science and Technology Research & Development of Taiwan. He owns a YouTube channel teaching deep learning in Mandarin with about 100k Subscribers. 

Abstract: Despite the differences between the text and speech modalities, self-supervised speech models have seen remarkable improvements through cross-modal transfer learning from text, and the application of successful self-supervised learning techniques from NLP. This talk will delve into our explorations aimed at improving self-supervised speech representations by building joint models of speech and text, also going over some of the techniques and intuitions from NLP that carried over well to Speech, and others that haven’t. Next, I will explore some practical applications of these models, with a focus on Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) for very low-resource languages, learned metrics for speech synthesis evaluation, multimodal representation analysis, and their parallels to analogous work in the text setting. Lastly, I will attempt to draw connections between contemporary research trends in Speech SSL and NLP, and share my perspective on future research.

Biography: Ankur is a Software Engineer at Google Brain working on multilingual Natural Language Processing and Speech Understanding. Previously he was on the Google Translate  team, building 'zero-resource' translation models for under-represented languages. Before joining Google, Ankur completed his Masters in Electrical Engineering, with a focus on Machine Learning and Optimization from Stanford. His current research interests include NLP and Speech processing for under-represented languages and joint models for speech and text understanding and generation.

Abstract: In this talk, I will provide an overview of several speech technology applications for unwritten languages, i.e., languages that do not have a common writing system, that have been developed in recent years. These applications bypass the need for textual information, instead using other information sources, mainly images, to learn the mapping between speech and images. I will focus on two questions: 1) What are the possibilities and limitations of such technologies? 2) What information do the speech representations learned using visual grounding contain?

Biography: Odette Scharenborg is an Associate Professor and Delft Technology Fellow at Delft University of Technology. She has an interdisciplinary background in automatic speech recognition and psycholinguistics, and uses knowledge from how humans process speech to develop inclusive automatic speech recognition systems that are able to recognise speech from everyone, irrespective of how they speak or the language they speak. Since 2017, she is on the Board of the International Speech Communication Association (ISCA), where she currently serves as Vice-President. From 2018-2021, she was on the IEEE Speech and Language Processing Technical Committee. Since 2018, she is a Senior Associate Editor of IEEE Signal Processing Letters. In 2025, she will be the General Chair of Interspeech Rotterdam. Odette is an active proponent of diversity and inclusion and is involved in many national and international initiatives to promote diversity, gender equality, and inclusion.

Abstract: The advent of pretrained self-supervised models has impacted speaker and emotion recognition. In this talk, we will present some recently proposed methods for extracting speaker and emotion recognition from such pretrained models. We will also consider the use of multilayer neural nets (e.g. TDNN) as backend models for self-supervised models, and whether more shallow backends can perform equally well. Finally, we will discuss self-training approaches, where the self-supervised model is trained from scratch on the target dataset.

Biography: Dr. Themos Stafylakis received his PhD in Speaker Diarization for Broadcast News from National Technical University of Athens, Greece in 2011, his M.Sc. in Communication and Signal Processing from Imperial College London, UK in 2005 and his B.Eng. from National Technical University of Athens, Greece in 2004. In 2011 he joined CRIM (Canada) as a post-doc researcher on speaker recognition. In 2016 he joined the Computer Vision Laboratory at University of Nottingham (UK), as Marie Curie Research Fellow. He is currently head of Machine Learning and Voice Biometrics at Omilia - Conversational Intelligence (Greece) and a visiting researcher at Brno University of Technology (Czechia). His main research interests are audiovisual speech and speaker recognition, spoken language understanding and machine learning.

Abstract: All fields in science and technology occasionally go through periods of transformative change.  Speech and audio processing clearly appears to be in the middle of one such period today due to the use of "all neural" model architectures and innovative “learning” techniques for almost all signal transformation tasks: speech enhancement, transcription, paralinguistic labeling (e.g. speaker, language, emotion), audio event detection, and so on.  Such periods also offer an opportunity to reflect on what questions may need to be answered to get past the unavoidable hype, and start focussing on research to address challenges that still remain.  For instance, (i) Can we clearly articulate what we do and don’t understand about SSL representations?  (ii) What theoretical and empirical tools do we need to fully understand SSL representations?  (iii) Are the accepted paradigm of training-tuning-testing partitions (including the SUPERB benchmark) obsolete?  (iv) What important applications are not well served by existing SSL representations, and why?  This presentation does not aim to answer these questions, only to frame them and hear what audience members and the other panelists think!

Biography: Sanjeev Khudanpur is a founding member of the Johns Hopkins University Human Language Technology Center of Excellence. He has a secondary appointment in the Department of Computer Science. Since 2022, Sanjeev is the Center Director of AI2AI, the JHU + Amazon Initiative for Interactive AI. His research interests are in the application of information theoretic and statistical methods to human language technologies, including automatic speech recognition, machine translation, information retrieval and natural language processing. He organizes the annual Johns Hopkins Summer Workshops to advance the greater research agenda of this field. Sanjeev received a B.Tech in Electrical Engineering from the Indian Institute of Technology, Bombay, in 1988, and a Ph.D. in Electrical Engineering from the University of Maryland, College Park, in 1997. Since 1996, he has been on the faculty of Johns Hopkins University. Until June 2001, he was an Associate Research Scientist in the Center for Language and Speech Processing and, from July 2001 to June 2008, an Assistant Professor in the Department of Electrical and Computer Engineering and the Department of Computer Science; he became an Associate Professor in July 2008.