How we identify the predominant melody of a music piece from its complicated sonorities? Up to date, the exact answer to this question is still mysterious to us. Automatic melody recognition, the problem that attempts to simulate human's ability in recognizing and resolving melody information from computation, is thus an attractive yet challenging problem in music AI.

In the melody recognition project, we systematically investigate this problem from multiple perspectives, including novel techniques in signal processing and machine learning, beneficial findings from musicology and psychology, and experimental data in both audio and symbolic format, in order to capture various semantic levels from musical expressions in a local scale to musical structures in a global scale.

We particularly focus on the following three approaches for melody recognition:

• Signal processing such as combined frequency and periodicity (CFP) and multi-layer cepstrum (MLC)
• Deep learning methods such as Patch-CNN, LSTM RNN, and semantic segmentation
• Transfer learning from symbolic to audio domains

Result