Artistic Influence

Research Highlight: Automated Artistic Influence Discovery

In the last decade there have been impressive advances in developing computer vision algorithms for different object recognition-related problems including: instance recognition, categorization, scene recognition, pose estimation, etc. When we look into an image we not only recognize object categories and scene category, we can also infer various aesthetic, cultural and historical aspects. For example, when we look at a fine-art paining, an expert, or even an average person can infer information about the style of that paining (e.g. Baroque vs. Impressionism), the genre of the painting (e.g. a portrait or a landscape), or even can guess the artist who painted it.  People can look at two painting and find similarities between them in different aspects (composition, color, texture, subject matter, etc.) This is an impressive ability of human perception for learning and judging complex aesthetic-related visual concepts, which for long have been thought not to be a logical process. In contrast, we tackle this problem using a computational methodology, to show that machines can in fact learn such aesthetic-related concepts.

Finding influences and connections between artists is an important task for art. By doing so, the conversation of art continues and new intuitions about art can be made. An artist might be inspired by one painting, a body of work, or even an entire style of art. Art historians study which artists might influence others by examining the descriptive attributes of art. However, finding influences is a sophisticated process that involves studying the historical, social, and personal context related to the artist and to the art. After all, we will not know if an artist was ever truly inspired by a work unless he or she has said so. However, for the sake of finding connections and progressing through movements of art, a general consensus is agreed upon if the argument is convincing enough. For example, Figure.1 illustrates a commonly cited comparison for studying influence, in the work of Francis Bacon's Study After Velazquez's Portrait of Pope Innocent X (1953), where similarity is clear in composition, pose, and subject matter.

Figure 1: An example of an often-cited comparison in the context of influence. Left: Diego Velazquez's Portrait of Pope Innocent X  (1650), and, Right: Francis Bacon's Study After Velazquez's Portrait of Pope Innocent X (1953). Similar compositions, pose, and subject matter but a different view of the work.

Is artistic influence a task that an intelligent agent can measure? Although there has been some research on automated classification of paintings, there is almost no research done on computer-based measuring and determining of influence between artists. However, there is limited research on the classification of painting style. Measuring influence is a very difficult task because of the broad criteria for what influence between artists can mean. Besides the scientific merit of the problem from the computational point of view, there are various application-oriented motivations. With the increasing volumes of digitized art databases on the Internet comes the daunting task of organization and retrieval of paintings. There are millions of paintings present on the Internet. To manage properly the databases of these paintings, it becomes very essential to classify paintings into different categories and sub-categories. This classification structure can be utilized as an index and thus can improve the speed of retrieval process. Also it will be of great significance if we can infer new information about an unknown painting using already existing databases of paintings.

Figure 2: Frederic Bazille's  Studio 9 Rue de la Condamine (left) and Norman Rockwell's Shuffleton's Barber Shop (right). The composition of both paintings is divided in a similar way. Yellow circles indicate similar objects, red lines indicate composition, and the blue square represents similar structural element. The objects seen -- a fire stove, three men clustered, chairs, and window are seen in both paintings along with a similar position in the paintings. After browsing through many publications and websites, we conclude that this comparison has not been made by an art historian before.

Although the meaning of a painting is unique to each artist and is completely subjective, it can somewhat be measured by the symbols and objects in the painting. Symbols are visual words that often express something about the meaning of a work as well. For example, the works of Renaissance artists such as Giovanni Bellini and Jan Van-Eyck use religious symbols such as a cross, wings, and animals to tell stories in the Bible. This shows the need for an object-based representation of images. We should be able to describe the painting from a list of many different object classes. By having an object-based representation, the image is described in a high-level semantic as opposed to low-level features such as color and texture, which facilitates suggesting influences based on subject matter.  Paintings do not necessarily have to look alike, but if they do, or have reoccurring objects (high-level semantics), then they might be considered similar. If influence is found by looking at similar characteristics of paintings, the importance of finding a good similarity measure becomes prominent. Time is also an essential factor in determining influence. An artist cannot influence another artist in the past. Therefore the linearity of paintings cuts down the possibilities of influence. 

The contribution of our work [1,2,3,4] is in exploring the problem of computer-automated suggestion of influences between artists, a problem that was not addressed before in a general setting. From a machine-learning point of view, we approach the problem as an unsupervised knowledge discovery problem.  Our methodology is based on three components: 1) studying different representations of painting to determine which is more useful for the task of influence detection; 2) measuring similarity between paintings; 3) studying different measures of similarity between artists. We collected a comprehensive painting dataset for conducting our study. The data set contains 1710 high-resolution images of paintings by 66 artist spanning the time period of 1412-1996 and containing 13 painting styles. We also collect a ground-truth data set for the task of artistic influences, which mainly contains positive influences claimed by art historian. This ground-truth is only used for the overall evaluation of our discovered/suggested influences, and is not used in the learning process.

We hypothesized that a high-level semantic representation of painting would be more useful for the task of influence detection. We tried to find this useful representation by considering the task of classifying painting style. The hypothesis is that features and representations that are good for style classification (which is a supervised learning problem) would also be good for determining influences (which is an unsupervised problem).  Therefore, we performed a comprehensive comparative study of different features and classification models for the task of classifying painting style among seven different styles [1,3]. The study spanned different level of state-of-the-art features from low-level to semantic-level and discriminative vs. generative learning methodologies (See Figure 3).  Recently we also studied the use of metric learning to combine various features for the task of style classification [4]. The conclusion of this study confirms our hypothesis that high-level semantic features would be more useful for the task of style classification, and hence useful for determining influences. 

Using the right features to represent the painting paves the way to judge similarity between paintings in a quantifiable way. Figure 2 illustrates an example of similar paintings detected by our automated methodology; Frederic Bazille's “Studio 9 Rue de la Condamine” (1870) and Norman Rockwell's “ Shuffleton's Barber Shop” (1950).   After browsing through many publications and websites, we concluded, to the best of our knowledge that this comparison has not been made by an art historian before. The painting might not look similar at the first glance; however, a closer look reveals striking similarity in composition and subject matter, which is detected by our automated methodology. 

Measuring similarity between paintings is fundamental to discover influences, however, it is not clear how painting similarity might be used to suggest influences between artists. The paintings of a given artist can span extended period of time and can be influenced by several other contemporary and prior artists. Therefore, we investigated several artist distance measures to judge similarity in their work and suggest influences. As a result of this distance measures, we can achieve visualizations of how artists are similar to each other, which we denote by a map of artists. An example of these maps is shown in Figure 4. 

Figure 4: Map of Artists

After all, we are not asserting truths, but instead we are just scratching the surface and suggesting a possible path towards a difficult task of automated measuring of influence. 

References: (See the publication page for details)

[1] R. Aroara and A. Elgammal “Towards Automated Classification of Fine-Art Painting Style: A Comparative Study”, 21st International Conference on Pattern Recognition (ICPR’12), Tsukuba Science City, Japan, November 2012.

[2] Kanako Abe, Babak Saleh and Ahmed Elgammal “An Early Framework for Determining Artistic Influence” The 2nd International Workshop on Multimedia for Cultural Heritage, MM4CH Naples, Italy 2013

[3] Babak Saleh, Kanako Abe, Ravneet Arora, Ahmed Elgammal “Toward Automated Discovery of Artistic Influence” Multimedia Tools and Applications Journal – Springer – 2014

[4] Babak Saleh, Kanako Abe and Ahmed Elgammal “Knowledge Discovery of Artistic Influences: A Metric Learning Approach” The Fifth International Conference on Computational Creativity, ICCC 2014.