Genre Evolution and Innovation in Popular Music

The study of genre classification and stylistic evolution seeks to better understand the defining characteristics of different genres. It looks at how different genres and styles emerge and evolve by analyzing changes in musical and cultural trends. With advances in Music Information Retrieval (MIR) software and the increasing availability of digital music data, work in this area is able to utilize extracted sonic feature data to better understand the role that human perception plays in genre categorization and in evaluating stylistic changes. In this paper, we seek to define a quantitative method for measuring the amount of stylistic change within a genre and the amount of novelty introduced into the genre by divergent songs. We will then create a model of this relationship in order to predict the degree of genre change caused by varying levels of novelty. Our findings allow us to quantitatively measure the impact of novel cultural products, which can lead to a better understanding of the dynamics of creativity, and the conditions that help foster innovation.

Introduction

Musical Genre Analysis

A key aspect to understanding and analyzing music is the concept of genre, which is typically defined as a group of songs or music styles that share defining characteristics. Studying how we organize songs, albums, and artists into broader groups that share similar musical characteristics can lead to a better understanding of the human perception of music. Research in this area also aims to understand how humans determine whether different audio features cause songs to sound more or less alike, leading to differences in how we classify and evaluate them. This includes looking at changes in popularity levels of different genres over time, determining what makes different cultural products more or less successful than others, and identifying the key drivers of stylistic changes within and across musical genres.

The ubiquity of digital music consumption has lead to an increase in the availability of data that can be extracted through Music Information Retrieval (MIR). Platforms such as The Echo Nest utilize MIR to extract quantifiable data about key sonic features that provide information about a song’s musical attributes and defining aural characteristics. This feature level data plays a large role in being able to classify and organize digital music, as well as offering the opportunity for new analysis of musical characteristics.

Impact of Novelty on Genre Evolution and Stylistic Changes

In this paper, we seek to model the relationship between the amount of novelty present in a genre at a point in time, and the magnitude of the stylistic shifts that subsequently occur within that genre. Here, we define novelty as the degree to which the audio feature values of a song deviate from what is considered typical for the genre.

Using audio features that were extracted via The Echo Nest’s Music Information Retrieval system, we will define a method for measuring the degree to which a genre has changed over time, define a method for measuring the amount of novelty represented by all the songs within a genre for a specific period of time, and develop a quantitative model of the relationship between these measures of novelty and genre evolution.

Previous studies tracking style changes within genres tend to focus more on general compositional structures, looking at data such as the chord progressions and intervals being used, and how overall tonal complexity changes over time. The research in this paper differs from these previous approaches in that it looks at measuring genre change as a direct result of how much novelty is present, as well as focusing on audio features that are most impactful to the human perception of the song as a whole.

While the relationship we’re looking at here is fairly intuitive, being able to quantifiably measure the degree of change that novel cultural products cause can then enable us to better identify examples of truly impactful innovation in popular music, and empirically measure the amount of influence they exerted. This also offers an alternative way of measuring the success of cultural products by looking at the degree of influence they exert over the direction of a genre’s evolution, rather than by popularity or commercial success.

Data

The data used for this analysis consisted of sonic feature data from The Echo Nest, derived by the platform's Music Information Retrieval (MIR) for the 24,300 songs that were included on the Hot100 Billboard charts from 1958-2013, along with genre and style data from Discogs.com. It encompassed 19 unique genres, 5,930 unique artists, and 1,099 unique record labels.

This data set allowed us to limit our analysis to only popular music that was most representative of the prevailing cultural space at each point in time. It is a subset of a data set originally created by Noah Askin and Micheal Mauskapf for their 2017 study on optimal differentiation in popular music, and was made available courtesy of Ágnes Horvát.

Methods

Data Preparation

For the analysis, we excluded key, mode, or time signature, as we did not find significant changes to the distribution of their feature values over time. Additionally, we did not include time in seconds, as overall changes to the average length of songs are more reflective of how the Billboard Hot100 songs are calculated and which music platforms are included, rather than intrinsic changes to a genre.

As all of the continuous features were measured on a 0-1 scale, our first step was to scale the loudness and tempo feature values to match. We created a scaled field for loudness, with 0 representing the decibel level of the quietest song in the dataset, and 1 representing the decibel level of the loudest song. The same was done for tempo, with 0 representing the beats per minute of the slowest song in the dataset, and 1 representing the beats per minute of the fastest.

We also gathered summary level information for each year by genre, which included the total number of unique artists, total number of songs, total number of unique sub-genres associated with the primary genre, and the total number of unique styles associated with the primary genre. These gave us additional metrics for assessing and measuring the level of diversity within a genre for each year.

Genre Shift

With the extracted sonic feature data, it is possible to represent each song as a point in N-dimensional space based on their feature values. To compare the distribution of songs within the feature space by genre, we performed primary component analysis on all songs within the data set, decomposing their array of feature values into two components, which allowed us to represent the song level distribution for each genre as a scatterplot. Although there was a high degree of overlap across genres, song distributions within each genre were fairly clustered around a central point, which demonstrated that we could use this average position as a way to represent the genre as a whole within feature space.

We can then measure the magnitude of change for a genre over time by measuring the amount that its position within the feature space changes. For each year, took the average feature values of all the songs within the genre, and collapsed them into a feature vector. For a genre G in year Y, the vector, G_y then represented the position of that genre within feature space for that particular year.

We then took the feature vector for the previous year, G_y-1, and calculated the distance between G_y and G_y-1. For each vector pair, we calculated the Euclidean distance, cosine distance, and correlation distance.

We then performed a one component primary component analysis on the year over year feature vectors for each genre. This high level view of the genre’s position shift over time confirmed that there is long term directionality to the genre shift that occurs, as opposed to the genre's average position experiencing year to year variations, but remaining in the same general location.

Additionally, we measured the autocorrelation of genre shift over time. As we can see, the autocorrelation measure never crosses the threshold of the 95% confidence interval. This tells us that the times series model of genre shift is random, meaning that the genre shift for each year cannot be predicted on the basis of past history, and is instead mainly influenced by exogenous variables.

Novelty Metric

To measure the amount of novelty in a genre, we first calculated a novelty score for each song. For this, we wanted a quantitative measure that represented the degree to which the feature values of the song diverged from what could be considered typical for the genre.

For each song, we collapsed its feature values into a vector S_GY , where G is the song’s primary genre, and Y is the year in which is appears on the Hot100 chart. We then measured the distance between S_GY and G_Y-1, thereby comparing the song to the average genre vector for the year prior. As with the genre shift metric, we measure Euclidean, cosine, and correlation distance. We then took the average novelty score of all the songs within a genre for each year, to calculate the amount of novelty present within the genre for that year.

The key concept here is that by comparing the song to the previous year’s averages, rather than the average for the year it charted in, we are not measuring how much variation is in the genre at that point in time, but rather looking at novelty as the degree of divergence from where the genre was positioned previously. In cases where the genre did not have songs present on the chart for the year prior, we excluded the data for that year, as there was no valid point of comparison for the genre shift and novelty calculations.

Model Training

Using this data, we ran linear regressions and random forest regressions using the scikit-learn Python package, to approximate the relationship between average song novelty in a genre for a given year, and how much the genre position shifted in the following year.

For each of the three distance measures we collected, Euclidean, cosine, and correlation, we used an 80/20 test-train split to train a linear regression, a Lasso regression with alpha = 0.0001, and a random forest regression with a maximum depth of 3.

Results

We found that all models improved significantly when adding in data fields that contextualized the novelty metric, so in addition to the average song novelty for each year, our models also included the standard deviation of song novelty within the genre for that year, as well as the number of unique artists, songs, sub-genres, and styles that were present in the genre for that same year.

Additionally, we found that although the correlation distance had the highest correlation coefficient, using the euclidean distance yielded the most accurate results for each of the regression models.

Figures 8-10 show the performance of our models when trained to predict the magnitude of genre shift based on the average song novelty of the prior year. Each point is a coordinate pair that represents the amount of novelty seen in a genre for an individual year, and the amount of genre shift measured in the subsequent year. The actual values of the testing data are shown in blue, with the values predicted by the model shown in orange.

We found that the Lasso regressions were the least accurate, with the linear regressions performing only slightly better. The random forest regression provided the most accurate model when using the Euclidean distance metric. It is also worth noting that the random forest regression model had an error distribution that was centered around zero, whereas both the Lasso regression and linear regression models tended to generate predictions that were lower than the actual values.

Discussion

Novelty Impact

Although there is still further refinement that can be done, this method of measuring the relationship between novelty and genre shift allows us to quantitatively understand the degree of influence that feature value divergence has on genre evolution. Understanding this relationship can then be used to create more accurate models of genre and style evolution than simply forecasting based on previous trends.

Additionally, it allows us to create a ‘baseline’ of how much shift we expect to see, and gives us a way to roughly classify instances of novelty as high or low impact. This could allow us to assess the value of specific instances of novelty and novel cultural products by looking at the impact they have within their cultural space, and whether the divergent properties they exhibit are absorbed into the status quo.

Further isolating cultural products or trends and assessing whether they have high or low impact novelty would enable further study into the differentiating factors between the two groups. Ideally, this would lead to identifying and understanding the key drivers and influences behind trends and changes in popular music.

Music Recommendation Software

Understanding the degree to which the defining characteristics of musical genres change over time also has applications for music recommendation software. Current recommendation software typically looks at tagged attributes of songs in an individual's library and tries to recommend similar music. However, if we can determine the degree to which more novel outliers shape the evolution of a subset of music with key commonalities, we can take that into account when tracking an individual's music preferences, and better predict what they might like as their tastes evolve.

Another consideration is that measuring the correlation between novelty in specific audio features and subsequent genre shift in this feature might provide insight into which features are more or less important to human perception of music and genre. Features that seem to carry the most weight in driving stylistic changes within genres might be the ones that play the largest role in how we perceive similarities between songs and categorize them into distinct genres.

Future Work

As the Hot100 data contains only a very targeted view of popular music, future work would involve gathering a larger data set that would encompass a broader representation of modern music. With the acquisition of a sufficiently sized data set, our goal would also be to model the novelty-genre shift relationship for each genre individually, to see if there are significant differences in this relationship across genres. Additionally, we would want to try and identify other metrics that could be included to further improve the accuracy of our model. One particular area we would like to explore further would be the influence that social networks have in introducing novelty, both for musicians and listeners, and how existing social connections influence the degree to which trends become widespread.

In order to look at isolating cultural products based on whether they are high or low impact, the next step would be to build a model that can forecast changes to the average audio feature values within an individual genre. This would allow us to link genre shifts back to specific songs or artists by determining whether the genre position shift mirrors the feature level divergence that they introduced. Being able to identify this causal link would also help in developing a better understanding of how much lag to expect between the time when a novel cultural product is introduced, and the time when we can see it’s impact on the genre position shift.

Acknowledgements

Special thanks to Ágnes Horvát for providing the Hot100 dataset used in this study.

References