Paper: https://arxiv.org/abs/2310.12404

 Code: https://github.com/ldzhangyx/loop-copilot/

(We are working on the final refinement to this code repo. It will be set to public very soon.)

Abstract

Creating music is iterative, requiring varied methods at each stage. However, existing AI music systems fall short in orchestrating multiple subsystems for diverse needs. To address this gap, we introduce Loop Copilot, a novel system that enables users to generate and iteratively refine music through an interactive, multi-round dialogue interface. The system uses a large language model to interpret user intentions and select appropriate AI models for task execution. Each backend model is specialized for a specific task, and their outputs are aggregated to meet the user's requirements. To ensure musical coherence, essential attributes are maintained in a centralized table.  We evaluate the effectiveness of the proposed system through semi-structured interviews and questionnaires, highlighting its utility not only in facilitating music creation but also its potential for broader applications. 

Highlights

1. We introduce Loop Copilot, a novel system that integrates LLMs with specialized AI music models. This enables a conversational interface for collaborative human-AI creation of music loops;

2. We develop the Global Attribute Table that serves as a dynamic state recorder for the music loop under construction, thereby ensuring that the musical attributes remain consistent in the iterative editing process.

3. We conduct an interview-based comprehensive evaluation, which not only measures the performance of our system but also sheds light on the advantages and limitations of using an LLM-driven iterative editing interface in music co-creation.