Embedded Files
Abstract
Abstract
UniComposer
A novel music generation pipeline that composes at the band level, utilizing a hierarchical multi-track music representation complemented by four cascaded diffusion models which progressively generate rhythm features, and unified features extracted from both symbolic and audio music by autoencoders.
Band-level Music Generation
Capable of allocating instruments based on musical features, their expressive potential and performance characteristics differences.
Unification of Symbolic and Audio
Architecture of joining the advantages of both format together, harnessing the richness of audio data and expressiveness of symbolic music.
Architecture
Architecture
Generation Pipeline
Generation Pipeline
Given input audio/symbolic input, musicology feature (e.g., time signature) and melody feature are extracted.
Four cascaded DMs to gradually generate features for monophonic (e.g., piano), polyphonic (e.g., flute) and percussion (e.g., drum).
Symbolic output are decoded, and can be converted to audio (optional).
Hierarchical Separation Intuition
Hierarchical Separation Intuition
Band-level music representation.
Full MIDI
57d2cdba98bb3730ae8a3ac76e96cb71.mp3Main Melody
57d2cdba98bb3730ae8a3ac76e96cb71_melody.mp3Reduced Mono.
57d2cdba98bb3730ae8a3ac76e96cb71_mono_reduced.mp3Reduced Poly.
57d2cdba98bb3730ae8a3ac76e96cb71_poly_reduced.mp3Reduced Perc.
57d2cdba98bb3730ae8a3ac76e96cb71_drum.mp3Detailed Mono.
57d2cdba98bb3730ae8a3ac76e96cb71_mono.mp3Detailed Poly.
57d2cdba98bb3730ae8a3ac76e96cb71_poly.mp3Detailed Perc.
57d2cdba98bb3730ae8a3ac76e96cb71_drum.mp3Showcases
Showcases
Multi-Instrument Music Generation
Multi-Instrument Music Generation
From SINGLE-TRACK input to BAND-LEVEL output.
Input Melody
1_1.mp3Input Melody
2_1.mp3Input Melody
3_1.mp3Output Band-level Music
1.mp3Output Band-level Music
2.mp3Output Band-level Music
3.mp3Unification of Audio and Symbolic Music
Unification of Audio and Symbolic Music
Dealing with MP3, WAV and MIDI music in SINGLE framework.
Input: Violin Melody (AUDIO)
violin_piece.mp3Input: Piano Melody (AUDIO)
flower_dance.mp3Input: Human Cappella (AUDIO)
en_songs.mp3Output: Band-level Music
violin_piece_acc_guitar_trumpet.mp3Output: Band-level Music
flower_dance_acc_guitar_violin.mp3Output: Band Accompaniment
en_songs_acc_guitar_violin.mp32. Ability of translating MP3, WAV into MIDI.
Input: Single-track AUDIO
flower_dance.mp3Input: Multi-track AUDIO
eg2.mp3Input: Special Instrument AUDIO
violin_piece.mp3Converted MIDI
flower_dance_byDM1.mp3Converted MIDI
eg2_trans.mp3Converted MIDI
violin_piece_byDM1.mp3Google Sites
Report abuse