Image Upscaling

Upscaling with Local Self-Examples and Sparisty in Transform Domain

Upscaling is also intimately related to a variety of other problems such as image inpainting, deblurring, denoising, and compression. Based on local self-similarity assumption on natural images [3], extract patches from extremely localized regions in the input image can reduce considerably the nearest-patch search time without compromising quality in most images, especially for small scaling factors where there are more example patches of greater relevance.

Fig. 1, Local Example-based Upscaling

Shown in Fig. 2 [3], a patch of lower frequency band from the upsampled image is matched (green arrow) with its nearest patch within a small window in the low-passed input image (purple). The upper frequency band of the matched patch in the input is used (red arrow) to fill in the missing upper band in the output upsampled image.

Fig. 2, Upsampling Scheme

Given an input image, defined on a coarse grid of pixels, start off by interpolating it to a finer grid using linear interpolation operator. This initial upsampled image lacks a fraction of its upper frequency band, proportional to the scaling factor. This missing band is then predicted using a non-parametric patch-based model that does not rely on external example databases but rather exploits the local self-similarity assumption [3]. Example patches are extracted from a smoothed version of the input image, which is upsampling of downsampling the original image. The high frequency prediction is done by first matching every patch in the upsampled image with its most-similar patch in the smoothed input. This search is not performed against every patch but rather against restricted small windows (purple window in the Figure 2) centered around the same relative coordinates as the center coordinates of the query patch.The complement high-frequency content in the input image at the matched patch,is used to fill-in the missing higher frequency band in the upsampled image by simply pasting it. Different accounts for the same pixel, due to overlaps between nearby patches are averaged together.

Fig. 3, Flowchart of BM3D-based Upscaling

Fig. 4, Results of BM3D-based Upscaling

References

1. Freeman, W. T., Jones, T. R., Pasztor, E. C., Example-based Super-Resolution, IEEE Comput. Graph. Appl. 22, 2 (March), 56–65., 2002.

2. Glasner, D., Bagon, S., Irani, M. Super-resolution From A Single Image, ICCV09. 349–356, 2009.

3. Freeman G. and Fattal R., Image and Video Upscaling from Local Self-Examples, ACM SIGGRAPH 2010.

4. M. Protter, M. Elad, H. Takeda, and P. Milanfar, Generalizing the Non-Local-Means to Super-Resolution Reconstruction, IEEE T-IP, Vol. 18, No. 1, Jan. 2009.

5. Danielyan A., Foi A., Katkovnik V., and Egiazarian K., Image Upsampling Via Spatially Adaptive Block-matching Filtering, EURSIPCO, 2008.