Fine Grained Image Inpainting

Evaluation

For the evaluation section, let's revisit the primary motivations for the project:

Determining how good the results are, using the proposed approach for image inpainting. This meant comparing the generated image with the ground truth and determining if the generated content looks similar to the ground truth.
Determining whether Image inpainting can be used for preserving the privacy of the visual data uploaded to the social media websites by the users. Essentially, determining if image inpainting can produce images which appear similar to the ground truth images, from the perception of a human but appear to be different when perceived by a Machine Learning model used in one of the facial recognition tools

Interpretation of the results

As depicted in the images below, there are two different kinds of results, the first kind is what we call "The Good" kind where the generated image appears similar to the ground truth image and the two images are pretty much indistinguishable from one another. The second kind is what we call "The Bad" kind where the generated image looks aesthetically pleasing but is not similar to the ground truth image, even to the human eye. Examples of both these kinds are presented below.

The Good

Ground Truth

Inpainted Image

Ground Truth

Inpainted Image

Ground Truth

Inpainted Image

Ground Truth

Inpainted Image

The Bad

Ground Truth

Inpainted Image

Ground Truth

Inpainted Image

Ground Truth

Inpainted Image

Ground Truth

Inpainted Image

Similarity Percentage

To determine whether the image inpainting solution will force an AI model based facial recognition tool to consider the generated image as a different person from the one in the ground truth; we did a similarity test. The first approach was to use the Facebook's image face recognition tool which would've automatically determined if the newly generated inpainted image is of the same person. However, as one can see in the image attached to the left; that system has been shut down and is no longer publicly available.

After this initial setback, the alternative approach was to use an online AI model based Facial Recognition tool (Face Comparison on ToolPie) to determine how similar are the two images (ground truth and inpainted image). In the pie chart below, the results are summarized. It is to be noted that the tool that was used for this evaluation considered two images to be the same person if the two images had a similarity score of >80%. Further below, we can look at some samples from each label and compare ourselves (through a human's perception) how accurately the facial recognition tool is able to determine if the inpainted image is for the same person.