t-SNE: going non-linear

The above plot is a little cramped if you viewed the html version on Github. So, what happens if we use a non-linear clustering algorithm? It has been widely shown that t-distributed stochastic neighbor embedding (t-SNE) can reveal structure where other clustering algorithms cannot. It effectively maintains a much higher variance in the data set when you reduce high dimensional data sets down to 2. Repeating the steps above we get the plot below (do play with the interactive versions!), which gives a much more effective separation of the artists in 2 dimensions, as compared to SVD.

The Flashbulb: single artist study

It has become easier to test t-SNEs on an artist I am very familiar with, because, at the end of the day I need to know if the computer has clustered songs sensibly or not. So, let's take a second to delve into The Flashbulb's music. I've taken all 43 (!) of his albums (560 songs!), and tried adjusting parameters in the t-SNE to get the best results (having one artist make this much music is an awesome data set). Specifically I am changing the learning rate, controlling how quickly it groups similar songs, and another hyper-parameter called 'Perplexity', which controls how to cluster small groups with large groups (or how many nearest neighbours each point is expected to have). This is a great example of how to understand t-SNE a bit more.

As the docs state, t-SNE has a cost function that is not convex, i.e. with different initialisations we can get different results every time we run it (shown by the two plots below). I'm trying to get the t-SNE to group distinctly different types of song, I want to avoid low learning rates which would cluster all songs too close to each other, and high learning rates which create too many distinct islands and eventually fail to cluster anything. Getting the learning rate (and perplexity) just right will enable it to distinguish between broad genres changes, but does not create isolated islands, such that we see a more gradual transition from his solo piano compositions to his glitchy drum compositions. An understanding of his music will probably help you interpret it more, but I will do this for a bunch of artists later on (some are on my Github page already).

In the plot below, the bottom right is his space music, piano albums, and ambient works. Then moving from there along to the left and upwards contains most of the songs on his major album releases such as "Nothing Is Real", "Opus At The End Of Everything". From there we quickly transitions back to the right where a large clump forms containing glitchier drum albums such as 'Hardscrabble", which still contain a lot of melody. Finally, moving up to the left we go towards his most glitchy and abstract releases under "Acidwolf", "Dr. Lefty" and "Human Action Network", and the bulk of tracks from albums such as "Red Extensions Of Me" and "Resent And The April Sunshine Shed". It is very interesting to note albums such as "Soundtrack To A Vacant Life", which contains songs covering much of this entire space, perhaps contributing to the large appeal that album had nearer the start of The Flashbulb's career (who knows?). Also note the spread in the style of Albums like "Arboreal", and his most recent release "Piety Of Ashes". This is probably starting to make less sense if you don't know the Flashbulb, but if you do, I recommend you click on the plot and explore the interactive version, toggling albums in the legend is the best way to explore, it's fascinating! Note that I varied the Perplexity parameter in the t-SNE to achieve the two different shaped plots - the one on the right is basically the same but without the bends! Note - I had trouble selecting 43 distinguishable colours... so they repeat... toggling them on/off in the legend is the best way to explore where individual albums sit.

You can check out similar plots for Avril Lavigne (run1, run2), where I did two runs showing the non-convex cost function giving different looking plots. Still, you get the same result (if you're familiar with Avril's songs), just displayed differently. Obviously she has less music, and generally sticks to a specific genre, so the diversity in the plot is less so than for The Flashbulb.

Supervised Learning

Here I split the data up into two chunks, one to train a computer with (60% in all the following plots), and one to test it on afterwards (40%). I'll use a random forest, and prune away some of the less useful features before using the remaining features for the classification. A lot of artists don't have much data (songs) to train on, so we'll see them having poorer scores.

A simple example is only comparing Classical music with Taylor Swift. Can a computer tell the different? Yes, very easily. Even when only training on half the data we get an almost perfect prediction. (Support is how many songs we used to test on in the prediction, so support/0.4=total songs, since we're testing on 40% of the data).

Let's make it harder, can it tell the difference between Taylor Swift and Avril Lavigne? Yes, but not so perfectly. In fact our precision and recall is very good (much better than random) considering how similar their music is (female singers, pop music, etc). The feature pruning made the accuracy a little worse, but not by anything significant.

OK so now I'll throw a whole bunch of artists into the picture. Overall the precision/recall is very good considering the opportunities to confuse artists with each other.

There are many important points to make here. Classical music and The Flashbulb perform the best, as they both have very large training sets compared to the other artists. Classical music is very musically distinct to most other artists in this run (at least our features are very sensitive to this), and we have a large training set, both making for high precision and recall scores. The Flashbulb, however, writes music over numerous genres, but because we have such a large training set we get high precision and recall scores - in fact, The Flashbulb's precision score suffers because we are accidentally classifying other artists music as The Flashbulb, which makes sense given the wide range of genres in his music. If a song really is by the Flashbulb, we have a 95% recall rate (we don't confuse his music with others).

In general, my features are not sensitive enough to achieve scores above 90% for pop music. So next up I'm exploring alternative ways to extract features, and more effectively train a computer on artists who haven't written so much music. For example, I'm looking at splitting songs up into mini songs, to bulk up the training set, and also exploring more complex feature extraction. Ultimately, I'm working on a simple vocal recognition algorithm, which should massively improve the scores for artists who have a main singer.

Now let's throw in Boards of Canada

We all love Boards of Canada. They've got a fair bit of music to train on (144 songs - more than most popular music artists, but not as much as The Flashbulb or Classical). However, it generally does not perform so well in my set up. Booooo. If you know Boards you can probably already guess why it doesn't perform well in this comparison - it's confusing their music a lot with Classical and The Flashbulb (hence their recall score is particularly bad). It has also now dragged down the precision of Clasical and The Flashbulb, given that BoC songs are now sometimes misclassified as The Flashbulb.

Xmas Song Classification!

It's almost Xmas, so why not explore some Xmas music. Turns out it's fairly difficult, but performs better than expected given the diversity of Xmas songs out there. Despite a library of 300 Christmas songs which I have amassed, having such a good training set doesn't help massively. We are confusing Xmas music with Classical, and confusing Pink Floyd and REM with Xmas music a lot of the time.... :/ Still, it works fairly well, and much better than artists with small training sets!