Update #1: Search, Website, Awesome Pittsburgh and more!

posted on September 6, 2018 @ 4:00pm

Welcome back to ReMasterpieces. Today, I want to give you an update on my project.


If you are new to ReMasterpieces, this project aims to use deep learning to recreate paintings stolen and lost by the Nazis prior to and during WWII. I hope to give the recreated paintings to the victims of Nazi looting or their heirs. (If you are completely new to ReMasterpieces, watch the presentation I gave to the US-UK Fulbright commission on the homepage: www.ReMasterpieces.org)

First, I want to thank Awesome Pittsburgh, which is a chapter of the Awesome Foundation, for awarding my project a grant in March. I’ve used the grant to dive deeper into deep learning and computer vision, which has yielded my first ReMasterpieces results. Shout outs to Awesome Pittsburgh!

Now, to the project.

The first step in recreating the lost paintings was to figure out what paintings were stolen and are currently lost. I developed a web-scraper to scrape over 1,000,000 webpages to find missing paintings that were photographed before their theft. I have identified over 15,000 such paintings. I now have a database of the missing paintings which includes provenance information and black and white images.

This database is very valuable. As a by-product of this work, I have created a search engine that allows mobile device users to take photographs of paintings in museums, homes, or other non-museum spaces to see if they are included in the database of missing paintings: effectively establishing a crowd-sourced search party for missing art. This is currently a python program I run on my computer, but I will develop an API for art and museum apps to incorporate into their own apps. If a match is found, we can alert the proper authorities such as INTERPOL, the FBI, or other law enforcement agencies. (Submit images via the "Search (beta)" tab)


I also created a modified logo for the search!

My Deep Learning Model:

Without going into too much detail, I’ve trained a deep learning model using a standard deep learning architecture where I used a second database I’ve created, to train the model. This database contains color and black and white photographs of paintings that are not lost. I analyzed each pixel in the paintings and labeled them appropriately for my model.


Once trained, I applied the model to a missing painting, rebuilding the color image pixel by pixel. While the process was a success, my model was only 2% accurate. So, I still have a lot of work to do before I can accurately recreate a missing painting.

I hope this update has shed some light on where ReMasterpieces is today. I will update you again once I have made some more progress.


Thanks for stopping by!