Research
Image inpainting
Image inpainting is the task of filling in a hole (or "occlusion") in an image. The goal is generally to remove some unwanted region of the image which has been degraded or contains some unwanted object (a rubbish bin, for example). It can be used for personal or professional use (in restoring films, for example). Here is a classic example of image inpainting (this image is taken from the work of Pablo Arias et al.).
Input image
Inpainted Image
Matlab image inpainting code
Here is a Matlab code which implements a multi-resolution patch-based variational image inpainting approach :
If you wish to use this code, please cite me (Alasdair Newson) as the author, and cite this webpage :
https://sites.google.com/site/alasdairnewson/research/image-inpainting
If you have any questions or discover any bugs, please email me at the following address :
alasdairnewson 'dot' work 'at' gmail 'dot' com
Inpainting demo
Try out our image inpainting demo online :
Tips and Tricks !!
Patch comparisons:
When comparing two patches, we often use the sum of squared differences as a patch metric. For textured patches, this can pose a problem. Often, the metric will have a tendency to associate smooth patches with more stochastic (textured, in a certain sense) ones. Let's take a look at a simple illustration of this.
Let X, Y and Z be three patches. X (the reference patch) and Y (a comparison patch) are random patches, whose pixels are iid., following a Gaussian distribution. Z (another, 'smoother', comparison patch) is constant, and equal to the average value of X and Y:
In this case, on average, the distance between X and Y is twice as great as that between X and Z, so the metric prefers to associate the patch X with the smooth patch Z, even though visually we prefer to associate the patch Z. This effect is often made worse when approximate nearest neighbour search algorithms are used.
How to solve the problem :
We need to be careful when comparing textured patches ! There are two common ways of resolving this problem :
- Searching for patches in a local neighbourhood around each source patch. In this case, the textures are most likely to be coherent, and so the problem of finding patches with different textures does not appear
- Changing the patch metric to reflect differences in textures.
Multiscale image inpainting methods :
One of the most powerful and popular inpainting approaches is the patch-based variational approach. This usually implies the minimisation of some kind of energy based on patches. For a few important examples of this approach, take a look at the work of Arias et al., and Wexler et al., in the case of videos. In these algorithms, a multiscale implementation is often used to produce better results by avoiding local energy minima. For this, we create an image (or video) pyramid.
However, we need to set the number of levels in the pyramid. How do we choose the maximum level L ?
In many methods, it is advised to set a minimum resolution, and choose L so that this resolution is reached. In fact, this is not a good way to set L. In my thesis, I explored this and showed that a more reasonable way is to set L such that the occlusion (hole to fill) is roughly twice the size of the patches which you want to use, at the coarsest level.
