Super Resolution: A Deep Dive

Abstract

With the advent of high fidelity monitors and television, consumers today want the best visual graphics, be it for gaming or videos. Going from a low-resolution image to a high-resolution image is a challenging task as a lot of information gets lost due to upscaling and interpolation. As a result, we get a blurred-out image. To circumvent this problem and to produce a clear and sharp up-scaled image, we deploy the concept of super-resolution that uses machine learning techniques to upscale a low-resolution image to a high-resolution image, without extensive loss to the information contained in it.

Introduction

In this project we will develop two different machine learning techniques commonly used to perform super-resolution. The two techniques are as follows:

Auto Encoder
Generative adversarial network (GAN)

We will look in detail how each of these techniques work in terms of time taken for execution, quality of the output image in terms of Peak Signal to Noise Ratio (SNR) and computation complexity. The main idea behind the project is it examine the dependence of the inverse reconstruction function (used to transform a low-resolution image to a high-resolution image) on the dataset that each model is trained.. Super resolution has varying application ranging from satellite image processing to microscopy and multimedia industry.

Teaser Figures

Left - Low Resolution Image

Middle - High Resolution Autoencoder

Right- High Resolution GAN

Approach

We will implement each of the two aforementioned techniques from scratch. Like in any research project, we will first go over the literature survey of recent and state-of-the-art techniques in GAN, and Auto-encoders for super-resolution e.g. [1,2]. Using the research as a baseline, we will then develop our own versions of these techniques by tweaking the hyper-parameters to push the performance. We will implement each of the techniques using the Pytorch library. We will be using the DIV2K, CelebA and Stanford-Cars dataset for our experiments.

Experiments

The DIV2K[3] dataset has the following bifurcations:

Training data: Consists of 800 high definition high resolution images and corresponding low resolution images with downscaling factors of 2, 3 and 4
Validation data: Consists of 100 high definition high resolution images and corresponding low resolution images
Test data: 100 diverse images are given to check the performance of the implementation

Some of the example of the images in the dataset are given below. We will be looking at the Signal to Noise ratio as our performance metric to evaluate the quality of the reconstructed image. The SNR is inversely proportional to the logarithm of the Mean Squared Error (MSE) between the ground truth image and the generated image. In our project we will try to achieve the performance of the state-of-the art. The set of experiments that we will try on the models can be given as follows:

Till date, we have implemented the neural networks for the GAN approach, the network design for the generator and the discriminator can be seen in the image shown below. The code, that we have implemented from scratch, can be found by clicking the button below.

Code on Co-labs

Following are some of the initial outputs from the generator network. The low resolution images are obtained by bicubic interpolation of the high-resolution images form the DIV2K dataset. The "high resolution fake image" is the image generated by the generator and the "high resolution real/target image" is the original image from the dataset that we are trying to recreate. Fine tuning of the network is required to exactly match the target image.

References

[1] K. Zeng, J. Yu, R. Wang, C. Li and D. Tao, "Coupled Deep Autoencoder for Single Image Super-Resolution," in IEEE Transactions on Cybernetics, vol. 47, no. 1, pp. 27-37, Jan. 2017, doi: 10.1109/TCYB.2015.2501373.

[2] C. Ledig et al., "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

[5] E. Agustsson and R. Timofte, "NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study," 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, 2017, pp. 1122-1131, doi: 10.1109/CVPRW.2017.150.

Page updated

Report abuse