Methods

The progress report page shows 3 different algorithms for image inpainting, namely "low-rank matrix completion", "sparse dictionary coefficients", and "sparse transform" (for a brief refresh, see the below screenshots of my presentation slides). This page will mainly show 2 machine learning methods for image inpainting, namely generative adversarial network (GAN) with contextual attention and denoising diffusion probabilistic models (DDPM). 

Machine Learning Methods

GAN: generative adversarial networks (GAN) for image inpainting can be treated as a conditional image generator where high-level recognition and low-level pixel synthesis are formulated into a convolutional encoder-decoder network, jointly trained with adversarial networks to encourage the coherency between generated and existing pixels. In this project, we aim to test the algorithm proposed by paper "Generative Image Inpainting with Contextual Attention (Yu et al., 2018)" on astronomical images (The model we used for testing was pre-trained on ImageNet). This paper proposes to contextual attention, of which the core idea is to use the features of known patches as convolutional filters to process the generated patches. It is designed and implemented with convolution for matching generated patches with known contextual patches, channel-wise softmax to weigh relevant patches and deconvolution to reconstruct the generated patches with contextual patches. The contextual attention module also has spatial propagation layer to encourage spatial coherency of attention. The following two figures show the whole image inpainting pipeline using GAN and the "contextual attention".

DDPM: Recently, score-based diffusion models received much attention. In this project, we aim to test the algorithm proposed by the paper "RePaint: Inpainting using Denoising Diffusion Probabilistic Models (Lugmayr et al., 2022)" on astronomical images (The model we used for testing was pre-trained on the place256 dataset). The DDPM is a variant of it that tries to sample a random noise vector and gradually denoising it until it reaches a high-quality output image. During training, DDPM methods define a diffusion process that transforms an image to white Gaussian noise in T time steps. As the goal of inpainting is to predict missing pixels of an image using a mask region as a condition, so we consider a trained unconditional denoising diffusion probabilistic model. Since the forward process of the diffusion is a Markov Chain of added Gaussian noise, we can sample the intermediate image xt at any point in time t. This allows us to sample the know regions at any time step t. And the unknown regions will be sampled from the trained neural network. The following figure shows the framework of image inpainting using DDPM. 

Algorithm 1 illustrates each step of using DDPM for the image inpainting task. The DDPM and GAN are both generative models, but the key difference is the DDPM requires iterative sampling during the inference time, which could be more time consuming than GANs. One remark of the "RePaint" paper is that it can sample varying outputs since the diffusion process is stochastic. We will show such "diversity" when tested on astronomical images in the results page.

You can also experiment with my code using your own images! Feel free to visit the GitHub pages below:

My GitHub page

GAN by Yu et al.

DDPM by Lugmayr et al.

References

Fessler, J. (2019, December). Matrix Methods in Signal Processing. Chapter 9: Matrix Completion

Fessler, J. (2020, April). Optimization Methods for SIPML. Chapter 1: Application Overview

Lugmayr, A., Danelljan, M., et al. (2022). RePaint: Inpainting using Denoising Diffusion Probabilistic Models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR52688.2022.01117

Origins: Hubble: Picture this: Raw data. Exploratorium. (2001). Retrieved April 10, 2023, from https://www.exploratorium.edu/origins/hubble/ideas/picture/picture2.html

Pesenson, M., Pesenson, I., Carey, S., McCollum, B., & Roby, W. (2009) High-Dimensional Data Reduction, Image Inpainting and their Astronomical Applications. In: Astronomical Data Analysis Software and Systems XVIII. ASP Conference Series. No.411. Astronomical Society of the Pacific , San Francisco, CA, pp. 81-85. ISBN 978-1-58381-702-5. https://resolver.caltech.edu/CaltechAUTHORS:20100913-150157129

Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., & Huang, T. S. (2018). Generative image inpainting with contextual attention. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/cvpr.2018.00577