AI Music Creation by Example

Music Tech, Human-Centered AI, User Research, UX, Accessibility. 

Publication, files and data. 

R (ggplot2), IBM Watson (Watson Beat, AI-powered music composition), Electron, User testing, Max/MSP, VSTs for audio generation (MIDI), Adobe XD. 

Short online videos have become the dominating media on social platforms, but finding suitable music to accompany videos can be a challenging task, due to copyright constraints, limitations in search engines, and requirements for audio-editing expertise. A possible solution to this problem is to use AI music generation. This project focused on an interface paradigm that allows users to input a song to an AI engine and then interactively regenerate and mix AI-generated music based on the example song. 

The aim of this work was to explore novel interface solutions that would make music generation more intuitive and accessible for non-expert AI users. The project was carried out at Adobe Creative Intelligence Lab, specifically targeting video creators. 

This project used an iterative design process informed by user studies to address the specific needs of video creators. Key methods included user testing, online questionnaires, thematic analysis, brainstorming workshops, parallel prototyping, wireframing, and semi-structured interviews, along with think-aloud protocols and expert evaluation. Initial pre-studies helped identifying common challenges and potential solutions, setting the foundation for users to create music for videos using an example song. Feedback from early prototypes was used to refine the system, which was then evaluated in depth. Data collected from 104 participants across multiple phases included user satisfaction ratings, time-on-task metrics, and qualitative feedback. Both quantitative and qualitative analyses were conducted to extract meaningful insights and optimize the interface.

This project addressed a critical gap in video creation tools by offering a simple, accessible solution for music generation. Using a user-centered approach, we designed and evaluated an AI-powered system that empowers video creators to curate music through an interactive interface. User studies yielded essential insights, informing prototype improvements and resulting in an overall satisfaction rating of 8.1/10. Participants found the system intuitive and enjoyable, with minimal learning time required to achieve satisfying musical results. This work underscores the importance of balancing automation and human agency in the design of effective creative tools.

This work was published at the prestigious ACM CHI Conference on Human Factors in Computing Systems (24.3% acceptance rate). Despite being a four-month project, the research has garnered substantial attention, achieving 61 scholarly citations and over 1,000 views on YouTube.