Invited Speakers

Abstract: We present SeaEval, a benchmark for multilingual foundation models. In addition to characterizing how these models understand and reason with natural language, we also investigate how well they comprehend cultural practices, nuances, and values. Alongside standard accuracy metrics, we examine the brittleness of foundation models in the dimensions of semantics and multilinguality. Our investigations encompasses both open-source and proprietary models, shedding light on their behavior in classic NLP tasks, reasoning, and cultural contexts. Notably, (1) Most models respond inconsistently to paraphrased instructions. (2) Exposure bias pervades, evident in both standard NLP tasks and cultural understanding. (3) For questions rooted in factual, scientific, or common sense knowledge, consistent responses are expected across multilingual queries that are semantically equivalent. Yet, many models intriguingly demonstrate inconsistent performance on such queries. (4) Models trained multilingually still lack ``balanced multilingual'' capabilities. Our endeavors underscore the need for more generalizable semantic representations and enhanced multilingual contextualization. SeaEval can serve as a launchpad for in-depth investigations for multilingual and multicultural evaluations.

Biography: Nancy F. Chen heads the generative AI group at A*STAR. She has served as the program chair of ICLR 2023, 2023 IEEE SPS Distinguished Lecturer, ISCA Board Member, in addition to being listed as 100 Women in Tech in Singapore 2021. She has won numerous professional awards, including the 2020 Procter & Gamble (P&G) Connect + Develop Open Innovation Award, the 2019 L’Oréal Singapore For Women in Science National Fellowship, A*STAR Fellow, and best paper awards from EMNLP, MICCAI, SIGDIAL, APSIPA, IEEE ICASSP. Speech evaluation technology developed by her team is deployed at the Ministry of Education in Singapore to support home-based learning during the COVID-19 pandemic. Nomopai is a spin-off company that uses technology from her lab to make customer agents more confident and empathetic. Prior to working at A*STAR, Dr. Chen worked at MIT Lincoln Laboratory on multilingual speech processing.  

Abstract: All fields in science and technology occasionally go through periods of transformative change.  Speech and audio processing clearly appears to be in the middle of one such period today due to the use of "all neural" model architectures and innovative “learning” techniques for almost all signal transformation tasks: speech enhancement, transcription, paralinguistic labeling (e.g. speaker, language, emotion), audio event detection, and so on.  Such periods also offer an opportunity to reflect on what questions may need to be answered to get past the unavoidable hype, and start focussing on research to address challenges that still remain.  For instance, (i) Can we clearly articulate what we do and don’t understand about SSL representations?  (ii) What theoretical and empirical tools do we need to fully understand SSL representations?  (iii) Are the accepted paradigm of training-tuning-testing partitions (including the SUPERB benchmark) obsolete?  (iv) What important applications are not well served by existing SSL representations, and why?  This presentation does not aim to answer these questions, only to frame them and hear what audience members think!

Biography: Sanjeev Khudanpur is a founding member of the Johns Hopkins University Human Language Technology Center of Excellence. He has a secondary appointment in the Department of Computer Science. Since 2022, Sanjeev is the Center Director of AI2AI, the JHU + Amazon Initiative for Interactive AI. His research interests are in the application of information theoretic and statistical methods to human language technologies, including automatic speech recognition, machine translation, information retrieval and natural language processing. He organizes the annual Johns Hopkins Summer Workshops to advance the greater research agenda of this field. Sanjeev received a B.Tech in Electrical Engineering from the Indian Institute of Technology, Bombay, in 1988, and a Ph.D. in Electrical Engineering from the University of Maryland, College Park, in 1997. Since 1996, he has been on the faculty of Johns Hopkins University. Until June 2001, he was an Associate Research Scientist in the Center for Language and Speech Processing and, from July 2001 to June 2008, an Assistant Professor in the Department of Electrical and Computer Engineering and the Department of Computer Science; he became an Associate Professor in July 2008.

Abstract: Unsupervised learning, in speech applications, has come to refer to the task of learning a mapping from modality X to modality Y using only unpaired examples.  If there is a one-to-one mapping between acoustic units and phonemes, and if the language model sufficiently distinguishes phonemes, then unsupervised speech recognition can learn the mapping using classical code-breaking techniques.  This talk will recap a key theoretical result showing that, with probability 1, natural languages have phonotactic language models that can be learned using unsupervised speech recognition.  The requirement that acoustic units should have a one-to-one mapping with phonemes has not yet been satisfied, but several interesting advances have been made.  First, it is possible to use unsupervised ASR to learn an unsupervised text-to-speech system.  Second, similar methods may be used to learn an unsupervised translation from speech to spoken-order sign language. Third, several experiments, with several types of data (multi-lingual data, child data, and data with a variety of speech disabilities), show that self-supervised pre-training can be improved by reducing the divergence between the data distributions observed during pre-training and test.

Biography: Mark A. Hasegawa-Johnson received his Ph.D. degree from MIT, was an NIH post-doctoral fellow at UCLA from 1996-1999, and joined the Illinois ECE faculty in 1999.  His research converts facts about speech production into low-resource transfer learning algorithms that can be used to make speech technology more fair, more inclusive, and more accessible.  His research has been featured recently in WSJ's The Future of Everything, in the Young Onset Parkinson's Network, and in a special report by the OECD.  Dr. Hasegawa-Johnson is a Fellow of the IEEE, of the Acoustical Society of America, and of the International Speech Communication Association, and he is currently Deputy Editor of the IEEE Transactions on Audio, Speech, and Language Processing.