Talks 

"Towards Expressive Artificial Musical Co-performers: The ACCompanion Story"

Abstract

With advances in AI and machine learning, there is a rising interest in developing interactive interfaces that allow people (both musicians and non-musicians) to explore music performance. The ACCompanion is an expressive automatic accompaniment system which enables collaborative music performance between humans and computers.  This system is capable of creating a human-like rendition of the accompaniment part of a piece of music in real-time, adapting to the soloist's tempo, dynamics, and articulation choices. This talk will discuss the components of the ACCompanion, including real-time score following, and expressive performance generation.  We will also explore the concept of togetherness in musical human-computer interaction, highlighting the challenges and potential of using artificial co-performers, which could serve as a platform in which to test hypotheses about the process of togetherness, as well as being a useful tool for music education and promoting the engagement of non-musicians with music.

Biography

Carlos Cancino-Chacón is an Assistant Professor at the Institute of Computational Perception at Johannes Kepler University Linz (JKU), Austria. Before this role, he conducted research at the Austrian Research Institute for Artificial Intelligence and was a Guest Researcher at RITMO Centre for Interdisciplinary Studies in Rhythm, Time, and Motion at the University of Oslo, Norway. His work revolves around developing machine learning models that understand the way humans perform and listen to music, with an emphasis on three topics: computational modeling of expressive performance, (real-time) human-computer interaction in musical contexts and cognitively plausible machine listening. He holds a PhD in Computer Science from JKU, an MSc in Electrical and Audio Engineering from the Graz University of Technology, a Bachelor's degree in Physics from the National Autonomous University of Mexico, and a Bachelor's degree in Piano Performance from the National Conservatory of Music of Mexico.

"The Journey of VirtuosoNet, a DNN-based AI pianist"

Abstract

Expressive performance modeling addresses multiple complex challenges, such as interpreting the music score, capturing shared traits among human performers, and accommodating diverse interpretive possibilities. This talk introduces how a deep-learning-based expressive performance modeling system, VirtuosoNet, was proposed and implemented, covering its feature encoding scheme and dataset composition, neural network architectures, and the training procedure. The presentation aims to equip attendees with insights into how expressive performance modeling can be conceived as a deep learning task and the considerations it involves.

Biography

Dasaem Jeong, Assistant Professor in the Department of Art & Technology at Sogang University, South Korea. He obtained his Ph.D. in culture technology from KAIST under the supervision of Juhan Nam. His research focuses on various music information retrieval tasks, including expressive performance modeling and symbolic music generation.

"Understanding and Generation of Audio-Visual Multimedia Content with Deep Learning"

Abstract

Current generative AI technology still faces significant technical hurdles in generating audiovisual content conditioned on themes, music, and storytelling. Thus, the current animation industry still relies heavily on manual labor and expensive techniques like human motion capture (MOCAP) in content production. In recent years, the rise of virtual musicians and VTubers in the Metaverse has increased the demand for automatic content generation in movies and character animation. Motivated from this, we would like to know whether current generative AI technology can solve a technical hurdle: generating animation of virtual musicians that fits the music reasonably. In this talk, we will start from one of our research paper, “A Human-Computer Duet System for Music Performance” (Best Paper Award Candidate in the ACM Multimedia Conference 2020), for a system view, and introduce facial expression generation, fingering generation, body movement generation, visual storytelling, and other related research topics on music performance and AI.

Biography

Li Su is a Ph.D. from the Graduated Institute of Communication Engineering at National Taiwan University, Taiwan. Since 2017, he has been serving as an Assistant Research Fellow at the Institute of Information Science, Academia Sinica, Taiwan. In 2021, he was promoted to Associate Research Fellow. His research interests span across artificial intelligence, multimedia technology, and musicology. He has been recognized with the Best Paper Award at the International Society for Music Information Retrieval (ISMIR) conference and nomination for the Best Paper Award at the ACM International Conference on Multimedia (ACM MM). He was invited as a panelist for the 29th Golden Melody Awards International Music Festival Forum. His performance collaborations include "Rising of the New Sound" (at the National Concert Hall, 2017) and "Whispers of the Night" (at the Kaohsiung Wei-Wu-Ying Concert Hall, 2019).

Yu-Fen Huang is a post-doctoral research fellow at Music and Culture Technology Laboratory, Institute of Information Science, Academia Sinica, Taiwan. Her research applies Music Information Retrieval (MIR) and AI techniques to explore the expressive elements in musical audio and body movement. Her research topics include: 1) the cross-modal mapping between musical sound and body movement, 2) audio analysis for piano and string performances using AI models, and 3) the expressive semantics in musical body movements. She endeavors to collaborate and integrate methodologies in diverse disciplines including systematic musicology, music technology, music psychology, biomechanics, and 3-D motion capture technology.

"Human-AI Music Ensembles on Stage and Lessons Learned"

Abstract

Human-AI music ensemble involves various analysis and generation capabilities for communication, synchronization, and expression in music performance. The KAIST AI Music Performance Team has been working on computational methods to enable the musical capabilities such as score following, cue detection, automatic music transcription, and expressive performance rendering, as well as developing systems to execute the algorithms in concert settings. In this talk, we will present several case studies of concert stage performances in which the KAIST AI music performance team has participated, and discuss the lessons learned.

Biography

Juhan Nam is an Associate Professor in the Graduate School of Culture Technology at the Korea Advanced Institute of Science and Technology (KAIST) in South Korea. He is leading the Music and Audio Computing Group at KAIST, working on various topics at the intersection of music, AI, digital signal processing, and human-computer interaction. In particular, his AI music performance team has developed an "AI pianist" system that can adapt to human performance and play musical pieces expressively. The AI Pianist system has been demonstrated in several concerts, collaborating with renowned artists such as Sumi Jo, Jasmin Choi, and Jonghwa Park.