• I got to learn a lot about different image inpainting methods. Prior to this project, I had absolutely no idea about the concept. Not only I learnt about the different techniques for image inpainting; I also got some insights into how the photo editing applications/software work behind the scenes. This project acted as a medium to demystify the "magic" of many photo editing tools.

• Working on this project enabled me to expand on the current understanding that I had; about the artificial neural networks and their applications. Before this project, the concept of deep neural networks was somewhat of a "black box" to me. This project provided me some hands-on experience with the neural networks (and how painful it could be to train one)!

• Witnessed Murphy's law in action. The details for the same are shared under the next section "Challenges"

The Murphy's law says:

Anything that can go wrong, will go wrong!

This was true for most part, during the course of this project. Collating the list of challenges that were faced here:

• No access to any kind of GPU hardware for training the model. This was the very first challenge that I faced. Over the course of a few weeks, I ended up trying different set of hardware including CloudLab GPU instances (which was made available to me for a particular assignment in another course), Google Colab before I got access to the GPU machines in the Chameleon Cloud cluster.

• Working on different platforms/machines come with one of its kind challenges. For example, any progress that I made in the Google Colab was lost once I closed the session. With such (frustrating) experiences, one is bound to become wiser nonetheless and have a set of scripts (Jupyter Notebook) which contains all the instructions needed to install/download the required packages and dependencies.

• Getting access to the instance(s) in the Chameleon Cloud cluster didn't really vanish the problems that I was facing. It just offset them from being a daily occurrence to being a weekly occurrence. As most of the leases are a week long, and the cluster is being shared across different universities; it became hard to extend any given particular lease. In short, I had one particular GPU machine for at most a week.

• Apart from the hardware-related issues; getting the codebase mentioned in the paper (that I was following) was no menial task either. The GitHub repo isn't actively maintained and is quite outdated (last commit being 2 years ago). I tried reaching out to the (only) individual contributor/admin of the repo but received no response. I ended up fixing a lot of configuration and python (code) files to get to a stage where the code for training the model was in a running state. (Similar experience happened for the testing state as well, but that's a story for another day/website)

• Without going deep into the details of the troubles that I faced while getting the CelebA-HQ dataset for training/testing; one advice I want to give to the people who plan to work with the same dataset (in the future) is to keep a copy of the dataset on the local machine. It'd be better than going through the multiple blogs/Stack Overflow answers that get you nowhere!

• Other functional issues included having conflicting versions of different packages (or dependencies) which would eat up a lot of time while debugging what the issues are. Things got better with time and the set-up on the new machine (every week) got smoother as I learnt from my experiences. However, it was a bit demoralising to begin with!

• After making everything else work, the biggest pain was not being able to visualize the results for the PyTorch profiler using Tensorboard. After trying multiple solutions available online, the one that actually worked was using Google Chrome (instead of my default browser Safari). The screenshot attached below will always have a special place in my heart!

• Blog: Pushing the Limits of Deep Image Inpainting Using Partial Convolutions

• Zheng Hui, Jie Li, Xiumei Wang, and Xinbo Gao. 2020. Image fine-grained inpainting. arXiv preprint arXiv:2002.02609 (2020). [link]

• Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3 (2009), 24. [link]

• G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” in European Conference on Computer Vision (ECCV), 2018, pp. 85–100. [link]

• C. Yang, X. Lu, Z. Lin, E. Shechtman, O. Wang, and H. Li, “High-resolution image inpainting using multi-scale neural patch synthesis,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6721–6729. [link]

• Fang Cao, Jiayi Sun, Xiangyang Luo, Chuan Qin, and Ching-Chun Chang. 2021. Privacy-preserving inpainting for outsourced image. International Journal of Distributed Sensor Networks 17, 11 (2021), 15501477211059092. [link]

• David Khachaturov, Ilia Shumailov, Yiren Zhao, Nicolas Papernot, and Ross Anderson. 2021. Markpainting: Adversarial machine learning meets inpainting. In International Conference on Machine Learning. PMLR, 5409–5419. [link]