Progress Report
Given an image Y with some missing features, we want to find a reconstructed image X. However, this problem is very challenging because the missing patch of an image could be any arbitrary value unless we have some prior knowledge of both the image and the patch.
The most basic assumption is that we know the locations of the pixels that are missing. However, in real-life situations, for example, when raw astronomical images are collected by the Hubble Space Telescope, it is likely that the observed image has missing information that we do not know the exact location of. Therefore, before we implement the image inpainting methods, we first need to determine the pixel locations of the missing patch. I propose to find the missing patch using the Canny edge detector under the OpenCV library from Python.
I first created a square patch onto the image and assuming that we do not know the pixel locations of this patch, I used the Canny edge detector to obtain the pixel locations of the wide range of edges on the image. Then to extract the desired patch, I must calculate the contour areas of the wide range of edges and find the maximum contour area. This will return an array containing the pixel locations of the perimeter of the missing patch (Refer to the code below and the output images). With the perimeter of the patch known, the patch is filled with zero values.
Once we have the pixel locations of the missing patch, which we denote as the sampling mask M, we hope that:
M ⊙ Y ≈ M ⊙ X
Where ⊙ denotes element-wise multiplication of matrices.
As a result, by the maximum a posteriori (MAP) estimate, I aim to solve the proposed cost function:
Where the first term of the cost function is the log-likelihood and the second term is the log-prior of the image. For the log-likelihood, we use the Frobenius norm, a matrix norm. It is essentially the image we start with, the raw image with missing information. The log-prior is basically all the different possible outcomes of the reconstructed image we can get based on the dictionary/transformation we apply. It consists of beta, a parameter that balances the importance of the terms in the cost function. For instance, if beta increases, this implies that we are more confident about our prior assumption of the image. R(X) can be represented in two different ways depending on whether we use a dictionary or a transformation matrix.
For a known dictionary matrix D:
For a known transformation matrix T:
Where Z refers to the coefficients of DCT/ODWT and in both cases, we assume that Z is sparse, meaning that it consists of mostly zero entries.
The cost function proposed earlier is not trivial to solve. Hence, a common approach is to apply majorize-minimize (MM) algorithm, leading to the following function:
Where k denotes the iteration number and M ̃ denotes the complement of M.
In this project, I plan to implement three different algorithms that use pre-designed dictionaries: low-rank matrix constraint, sparse dictionary coefficients, and sparse transformations.
On the other hand, instead of pre-defining the dictionary D, the question is how can we learn D from training the data? Since dictionary learning is a complex process that requires extensive knowledge of machine learning, I decide to learn the dictionary by running a pre-existing algorithm.
In conclusion, here are the three tasks that I plan to complete until the project is due:
Find a working, pre-existing dictionary learning algorithm and test it with various types of astronomical images. If we cannot find a working algorithm, choose an alternative astronomical image that is easier to work with, meaning the patterns are easily distinguishable by inspection so that we can create our own dictionary manually using a so-called dictionary of experts method.
Implement the above three algorithms in Python
Compare the results of the three algorithms to the dictionary learning algorithm by using the peak signal-to-noise ratio (PSNR) or structural similarity index (SSIM).
One thing that I learned so far:
After numerous literature reviews and studying optimization methods on my own, I learned how to define a cost function to optimize a routine. It was very interesting to learn how the probabilistic methods I learned from EECS 301 are applied to solving the cost functions. This is an essential technique that I must know in order to calculate the final inpainted image that maximizes the probability of obtaining a reconstructed image that is close to the ground truth. Also, it was interesting to see how the DSP tools we learned in class such as DCT and matrix operations can be applied to a problem that is more complex and real-life based.