Machine Learning and Art

[Posted on 10/2018]

With the advancements in computing technology and artificial intelligence, computers have started to perform human activities from factory operations to driving. Following this trend, an intriguing question arises: can computers create art?

To answer this question, perhaps, we should first answer: can computers understand art?

As a conceptual art enthusiast, I have played with the idea of testing the ability of machine learning in understanding different illustration styles from realism to abstract.

For the following tests, a freely available image recognition tool, Google Cloud’s Vision API, is used to find labels describing each picture. 

We start with a painting in Realism style that is fairly easy for a well-trained ML model to recognize. Feeding the following picture to the API tool, the top assigned labels are as follows:

Pretty accurate! However, if the model is to select a single object that best describes this picture, it returns:

Interesting!

Ascension: pastel drawing by DD (2016)

Let's take it up a notch for the second test. Feeding the following painting in Expressionism style to the API returns the following labels:

Still accurate, but very generic! It seems that the model is able to recognize the picture as art but does not seem to understand it, unlike the previous example. Let's see what the output of the API is as the object best describing the content of this picture:

Very interesting!

Void cafe: oil painting by DD (2008)

For the ultimate challenge, we next feed the following drawing in Abstract style to the API. The top characterized labels are as follows:

Similarly to the previous test, the model is able to recognize but not quite understand the picture (understanding here can be interpreted as recognizing some of the many specific figures that a human can recognize within a few seconds looking at it; give it a try!).

Interestingly, the API detects the following (low percentage) labels as recognized figures:

Can you find any figure in the picture with any similarity to these two found by the API? 

To Mercury: ink drawing by DD (2018)

After this small experiment, what can we say about the first question: Can computers understand art?

It is important to note that the used API tool is a pre-trained machine learning model. The outcome of the model heavily depends on the training set, of course. While a comprehensive training set can improve some aspects of art recognition (for instance for realism and impressionism styles,) there are some other aspects that seem to remain hard to be fully recognized by machines (for instance for expressionism and abstract styles.)

The silver lining is that, despite all its limitations, the API is able to detect some figures, as seen in the last test, that a human may not be able to recognize (at least at first glance).

So, maybe the answer to the above questions is not a YES-NO type of answer, but something in the middle. Maybe computers can complement human's understanding of art...