Flat Latent Manifolds for Human-machine Co-creation of Music

Latent space exploration by a professional drummer

The video shows the interaction of the drummer and the FM-Music-VAE model. The drummer and the model take turns playing drums and output two bars each time. The input of the model is the previous two bars by model and two bars by the drummer. The output of the model is generated by the extrapolation from the latent space of inputs with a Gaussian noise. The right side is a three-dimensional space compressed from the latent space using a U-MAP. The green and yellow dots represent the data points of the drummer and the model, respectively. The melody starting at 6:16 is another soundtrack that is neither from the drummer nor from the model.

Music interpolation

Given the starting and ending two bars, seven points are linearly interpolated in the latent space.

Drum interpolation sample 1

Drum interpolation sample 2

Drum interpolation sample 3

Piano interpolation sample 1

Cello interpolation sample 1

Each data point is four bars. The output sound is adapted to the piano.

Cello interpolation sample 2

Attribute vector

The middle data point (two bars) of each sub-figure is from the datasets, and from the left to the right directions are the attribute vector with the scale factor from -1 to 1.

Drum density sample 1

Piano density sample 1

Drum velocity sample 1

The color represents the velocity

Further Drum samples

FM-Music-VAE - Varying note density

FM-Music-VAE - Varying note velocity

FM-Music-VAE- Interpolating between sequences