Rembrandt Portraits

Over a long career marked by renown, personal tragedy, and the familiar reversals of celebrity, the iconic Dutch master Rembrandt Harmenszoon van Rijn (1606-1669) created some of the most famous (and valuable) works of visual art across a wide range of styles and subjects.  Following a few years of traditional apprenticeship, he opened his own studio before his 20th birthday and soon began admitting students.  Rembrandt’s star rose quickly as he earned important commissions from the court of The Hague and found himself in great demand as a portraitist.  By 1634 he was a burgess of Amsterdam and married to Saskia van Uylenburgh, with whom he had four children — only one of whom, his son Titus, survived into adulthood.  Saskia died shortly after Titus’s birth.

Rembrandt’s subsequent controversial relationships, spendthrift acquisitions of art and property, and his inevitable descent from the heights of fame led to insolvency in 1656.  His portraits, widely regarded for their realism and psychological depth, did not always please their subjects.[1]  Despite numerous important commissions in his later years, he died a poor man and was buried anonymously.

Rembrandt’s students learned by copying his works and serving as his assistants.  Many, such Govaert Flinck, Ferdinand Bol, and Carel Fabritius, became recognized artists in their own right.  Their sheer number — maybe 100 or so — and Rembrandt’s working habits have complicated scholarship of his works for hundreds of years.  One authority notes that “it seems most likely that Rembrandt, like Rubens in Antwerp and Van Dyck in England, used studio assistants to help him produce paintings for the market, especially during the 1630s when his work was in great demand.”[2]  Indeed, experts have long suspected that Rembrandt included drawings by his students along with his own in albums sold as his work.[3]  Rembrandt also signed his students’ work on occasion, enhancing the profit he made on its sale.[4]

In 1968 the Rembrandt Research Project (RRP) was formed to answer the many persistent attribution questions with some finality.  They did so energetically, rejecting dozens of works.  When the Dutch art historians of the RRP disagreed over stylistic criteria, which was frequently, the very existence of such disagreement often resulted in de-attribution.  Rejected works included a signed 1637 self-portrait and the Frick Collection’s beloved Polish Rider.[5]  A century ago Rembrandt’s total output was estimated at 711 works, but by 1989, only 250 works had survived the RRP juggernaut.

Although cooler heads later prevailed and the committee restored 90 or so works to the canon before disbanding in 2011, many Rembrandt paintings remain mired in controversy.  While The Polish Rider (which the Frick dates ca. 1655) is now recognized as a Rembrandt, another painting, The Man with the Golden Helmet (Gemäldegalerie, Berlin), is among those that have bounced from attribution to de-attribution, with the current scholarly consensus being that it isn’t a Rembrandt.[6]

The A-EyeTM for Rembrandt

Given the uncertainties surrounding the authenticity of many purported Rembrandt works — uncertainties that cannot always be resolved by provenance or even scientific analysis — assembling a reliable dataset and interpreting results pose particular challenges.  Rembrandt painted in different styles and genres, ranging from theatrical religious and historical subjects to quiet landscapes and formal portraits.  We focused on portraits.  The dataset we employed to train The A-Eye for Rembrandt included high-resolution images of portraits he painted from the early 1630s nearly until his death in 1669.  The other 50% of our training set consisted of portraits selected for varying degrees of pictorial similarity to the Rembrandts.  

The objective in selecting the comparative non-Rembrandt images was to train the A-Eye to make fine as well as coarse distinctions and to generalize beyond the training images.  The paintings in the training set include works that had once been attributed to the master himself, but which are now qualified as "school of," "workshop of" or "circle of" Rembrandt; portraits attributed to students of Rembrandt (Govaert Flinck, Carel Fabritius, and others); and works by Dutch contemporaries of Rembrandt, including several (like Frans Hals) whose style is easily distinguishable from Rembrandt's, even to a non-specialist.

Our Rembrandt dataset was not large — about half the size of the one we used for our van Gogh study — which allowed us to steer clear of controversial attributions and also to assess whether, trained on a small number of paintings, The A-Eye could nonetheless produce reasonably accurate classifications.  The extant work of many Old Master artists is quite limited, yet this does not deter forgers: only 36 authentic paintings by Johannes Vermeer have been identified, for example, but his work was famously forged by Han van Meegeren during the Nazi era.  Good performance on a limited training set would demonstrate the versatility of The A-Eye as a tool for studying very scarce artwork.

What we Found

Our “Salient Slices” technique, described in detail here, first divides a source image into overlapping tiles of a given size.  These tiles are then sifted in accordance with a discriminator that identifies the tiles likely to contribute meaningfully to classification.  Our discriminator computes image entropy, which corresponds to the degree of visual diversity exhibited by a tile, and retains only those tiles whose image entropy equals or exceeds that of the entire source image.

The A-Eye analyzes each qualifying tile of a test image and assigns it a probability between zero and one:  values equal to or exceeding 0.5 correspond to classification as Rembrandt, while values below 0.5 correspond to a non-Rembrandt classification.  We classify the image based on both the average probability across all tiles and “majority vote,” i.e., the majority tile classification determines the image classification.  The closer a classification probability is to one, the greater is the likelihood, according to the trained CNN model, that the tile represents Rembrandt’s work; conversely, classification probabilities approaching zero strongly suggest a different artist’s work.

We trained and tested our CNN model at tile sizes ranging from 100 x 100 to 650 x 650 pixels.  For this Rembrandt study, we obtained maximum classification accuracy using 450 x 450-pixel tiles — roughly the scale of a face in a portrait.  With tiles confined to brushstroke-level detail, by contrast, The A-Eye performs little better than guessing.  Maybe Rembrandt taught the mechanics of his painting style especially well, or perhaps the system finds his stroke simply is not all that unique.  What others have not been able to duplicate — at least insofar as The A-Eye can discern — is his larger compositional vision and rendition.  Rembrandt’s “signature” style emerges at this broader scale.

We find that classification error can be reduced further by combining the probabilities associated with different tile sets, since the different CNN models “see” differently scaled features; the combination can mitigate errors associated with each tile set considered separately.  Rather than taking a straight average, we computed weights for the 450 x 450 and 550 x 550 probabilities that minimize the overall classification error.[7]  We found that the optimized weightings modestly reduce the already low error.

We can explore the regions important for classification using “probability maps,” which color-code the probabilities assigned to the examined regions of an image at a granular level: red corresponds to high-likelihood (≥ 0.65) classification as Rembrandt, gold to moderate-likelihood (0.5 ≤ p < 0.65) classification as Rembrandt, green to moderate-likelihood (0.5 > p > 0.35) classification as not Rembrandt, and blue to high-likelihood (< 0.35) classification as not Rembrandt.  Gray image regions correspond to tiles that did not pass the image-entropy selection criterion and were not examined.

The probability map derived from Rembrandt’s 1642 portrait of Saskia, owned by the Altemeister Museum in Kassel, Germany, illustrates the synergies that can be achieved by combining CNN analysis with traditional scholarship.  As shown in our demo, the “Kassel Saskia” is attributed to Rembrandt, but the flat, rather featureless face seems anomalous (and maybe a bit disappointing) for Rembrandt.

And indeed, Rembrandt scholar Ernst van de Wetering notes that the face was given a later “porcelain-like overpainting” that obscures the “livelier peinture” beneath.[8]  The Altemeister Museum’s catalog text, moreover, notes that the background in this region also was not painted by Rembrandt.  The A-Eye properly classifies the overall painting as a Rembrandt but correctly identifies these areas as the work of other artists.  Without the supporting scholarship, those dark blue regions might be dismissed as simple misclassification, since The A-Eye’s accuracy is by no means perfect.

This leaves us wondering about the probability map for Rembrandt’s Man in Oriental Costume (The Noble Slav) (1632; Metropolitan Museum of Art, New York):

Although The A-Eye correctly classifies the painting as Rembrandt’s work, it attributes a sizable region of the clothing and background to a different hand.  In the absence of corroborating scholarship this is most safely considered a simple error, due perhaps to the stylistic ambiguity of a dark, unfeatured area.  Still, the cohesive unity of the blue-mapped region is provocative; if it is an error, it is not a random one.  The same model also appears in works by Jacob Backer, who worked in Rembrandt's studio between 1632 and 1634.[9]

To further test performance, we evaluated six works that have been subject to attribution controversy to compare the A-Eye’s classification to the current scholarly consensus, with the following results:

Title                                                                           Scholarly Consensus           A-Eye Classification        Probability

Man with the Golden Helmet                                School of Rembrandt            Rembrandt                                   0.85

Portrait of a Young Gentleman                             Rembrandt                              Rembrandt                                   0.81

Portrait of Elisabeth Bas                                        Ferdinand Bol                         Not Rembrandt                           0.39 

The Polish Rider                                                      Rembrandt                              Rembrandt                                   0.57 

Portrait of a Man ("The Auctioneer")                    Follower of Rembrandt        Not Rembrandt                            0.29 

Portrait of a Young Woman                                   Rembrandt                              Rembrandt                                   0.82

As with Man in Oriental Costume, the probability map of Portrait of Elisabeth Bas (Rijksmuseum, Amsterdam) includes regions that the A-Eye misclassifies – this time as by Rembrandt. The portrait is now attributed to Rembrandt's student Ferdinand Bol and dated ca. 1640 - ca. 1645.

Again, one can speculate as to the reason. Though small, our Rembrandt training set did include both Rembrandt and non-Rembrandt portraits of subjects with crossed hands, so the red regions probably do not arise as a result of simple training bias. Indeed, it may be the very commonality of the pose — combined, perhaps, with less artistic attention paid to the hands than to the face and visually exciting details of attire — that undermines classification accuracy.

The Auctioneer, owned by the Metropolitan Museum of art, was attributed to Rembrandt until 1982, when, according to the museum, "scholars 'demoted' it ... arguing that the rendering of space, texture, and anatomy suggests a skillful imitation by an unknown artist."  Portrait of a Young Woman, part of the Kress Collection and exhibited at the Allentown Art Museum, had long been attributed to Rembrandt's workshop; but following "routine" conservation that began in 2018, the conservators discovered strong evidence it was painted by Rembrandt.  The A-Eye concurs in both cases.

[1] The text accompanying the Portrait of Gerard de Lairesse (1665-67), owned by the Metropolitan Museum of Art, notes that Rembrandt “directed his unsparing scrutiny to the sitter’s features, disfigured by what was likely congenital syphilis. This condition eventually caused De Lairesse to go blind, and he devoted the rest of his life to writings on art in which he disparagingly likened Rembrandt’s thick application of paint to ‘liquid mud on the canvas.’”

[2] Arthur K. Wheelock Jr., "Issues of Attribution in the Rembrandt Workshop," in Dutch Paintings of the Seventeenth Century, NGA Online Editions (Washington, 2014).

[3] Jori Finkel, “A Rembrandt Identity Crisis,” New York Times, Dec. 4, 2009, p. AR37.

[4] Catherine B. Scallen, Rembrandt, Reputation, and the Practice of Connoisseurship, Amsterdam Univ. Press (2003), p. 257.

[5] RRP member and leading Rembrandt scholar Ernst van de Wetering more recently characterizes the RRP’s initial verdict as “doubt” rather than de-attribution. Ernst van de Wetering, Rembrandt’s Paintings Revisited, Springer (2017) (hereafter “van de Wetering”), p. 627.

[6] Michael Brenson, “Scholars Re-Examining Rembrandt Attributions,” New York Times, Nov. 25, 1985, p. C13.

[7] We use the L1 measure for classification error, defined as the sum of the absolute difference between 0.5 and the probability associated with an incorrect classification (so that, for example, a van Gogh incorrectly classified with a probability of 0.4 would have an L1 error of 0.5 - 0.4 = 0.1).

[8] van de Wetering, p. 655.

[9] Metropolitan Museum of Art, online catalog text available at https://www.metmuseum.org/art/collection/search/437385. 

[10] van de Wetering, p. 627.

[11] K. Kleinert and C. Laurenze-Landsberg, “Der Mann mit dem Goldhelm,” Staatliche Museen zu Berlin, available at http://www.smb-digital.de/eMuseumPlus?service=ExternalInterface&module=collection&objectId=865524&viewType=detailView