Super-Resolution

The Problem

There are many tradeoffs to be considered when designing an imaging system for image acquisition.  The desire for high spatial resolution and a wide field of view generally leads to camera systems employing small f-number optics.  This produces an image with very high spatial-frequency bandwidth at the focal plane. A focal plane array (FPA) with sufficiently small detector spacing is required to sample the scene at the Nyquist rate.  However, several factors may make it impractical in most cases to do so.  Consequently, undersampled imaging systems are very common and produce images that suffer from aliasing artifacts.  A very active area of research, referred to as super-resolution (SR), involves post-processing techniques to overcome such limitations.  

Multi-frame SR algorithms generally process a set of low-resolution (LR) aliased frames from a video sequence to produce a high-resolution (HR) frame.  When sub-pixel relative motion is present between the objects in the scene and the detector array, a unique set of scene samples is acquired with each frame. This provides the mechanism for effectively increasing the spatial sampling rate of the imaging system.   Video SR output is possible by using a sliding temporal window of input frames.

Dr. Hardie is a leader in this research area.  Along with several colleagues, he received the Rudolf Kingslake Medal and Prize from SPIE in 1998 for work on SR.  Dr. Hardie is currently involved with developing real-time algorithms and implementations for SR.

SR images from: R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong and E. A. Watson, “High Resolution Image Reconstruction From a Sequence of Rotated and Translated Frames and its Application to an Infrared Imaging System,” Optical Engineering, Vol. 37, No. 1, Jan. 1998, pp. 247-260. Winner of the 1998 Rudolf Kingslake Medal and Prize.

Video link

Video results from: R. C. Hardie, D. R. Droege, A. J. Dapore, and M. E. Greiner, “Impact of detector-element active-area shape and fill factor on super-resolution,” Frontiers in Physics: Optics and Photonics (Special Issue on Super-Resolution), Accepted, April 2015.

R. C. Hardie, “A Fast Super-Resolution Algorithm Using an Adaptive Wiener Filter,” IEEE Transactions on Image Processing, Vol. 16, No. 12, Dec. 2007 pp. 2953-2964.

R. C. Hardie and K. Barnard, "Fast super-resolution using an adaptive Wiener filter with robustness to local motion," OSA Optics Express, Vol.  20, 21053-21073 (2012).

Selected References

1. Hamed Elwarfalli, Russell C. Hardie, "FIFNET: A convolutional neural network for motion-based multiframe super-resolution using fusion of interpolated frames," Computer Vision and Image Understanding, Volume 202, 2021, 103097, ISSN 1077-3142, https://doi.org/10.1016/j.cviu.2020.103097. NEW PAPER!

2. R. C. Hardie, M. Rucci, B. K. Karch, A. J. Dapore, D. R. Droege, and J. C. French, "Fusion of interpolated frames superresolution in the presence of atmospheric optical turbulence," Opt. Eng. 58(8) 083103 (22 August 2019) .

3. R. C. Hardie, D. R. Droege, A. J. Dapore, and M. E. Greiner, “Impact of detector-element active-area shape and fill factor on super-resolution,” Frontiers in Physics: Optics and Photonics (Special Issue on Super-Resolution), Accepted, April 2015.

4. K. M. Mohamed and R. C. Hardie, “A Collaborative Adaptive Wiener Filter for Multi-Frame Super-Resolution”, Frontiers in Physics: Optics and Photonics (Special Issue on Super-Resolution), Accepted, April 2015.

5. B. K. Karch and R. C. Hardie, “Robust super-resolution by fusion of interpolated frames for color and grayscale images," Frontiers in Physics: Optics and Photonics (Special Issue on Super-Resolution), Accepted, April 2015.

6. B. N. Narayanan, R. C. Hardie, and E. J. Balster, “Multiframe adaptive Wiener filter super-resolution with JPEG2000-compressed images,” EURASIP Journal on Advances in Signal Processing, April 2014 (Highly Accessed Status).

7. B. K. Karch and R. C. Hardie, "Adaptive Wiener filter super-resolution of color filter array images," OSA Optics Express, Vol. 21, Issue 16, pp. 18820-18841 (2013).

8. R. C. Hardie and K. Barnard, "Fast super-resolution using an adaptive Wiener filter with robustness to local motion," OSA Optics Express, Vol.  20, 21053-21073 (2012).

9. R. C. Hardie, K. Barnard, and R. Ordonez, “Fast super-resolution with affine motion using an adaptive Wiener filter and its application to airborne imaging,” OSA Optics Express, Vol. 19, Issue 27, 26208-26231 (2011).

10. R. C. Hardie, D. A. Lemaster, and B. M. Ratliff, “Super-Resolution for Imagery from Integrated Microgrid Polarimeters,” OSA Optics Express, Vol. 19, No. 14, pp. 12937-12960, June 2011.

11. R. C. Hardie, “A Fast Super-Resolution Algorithm Using an Adaptive Wiener Filter,” IEEE Transactions on Image Processing, Vol. 16, No. 12, Dec. 2007 pp. 2953-2964.

    • MATLAB Software Available

12.  R. C. Hardie, R. R. Schultz, and K. E. Barner, “Super-resolution Enhancement of Digital Video,” EURASIP JASP, Vol. 2007,  Article ID 20984, (3 pages).

13.  R. C. Hardie  and D. Droege, “A MAP Estimator for Simultaneous Super-Resolution and Detector Nonuniformity Correction,” EURASIP JASP, Vol. 2007,  Article ID 89354, (11 pages).

14. B. Narayanan, R. C. Hardie, K. E. Barner, and M. Shao, “A Computationally Efficient Super-Resolution Algorithm for Video Processing Using Partition Filters,” IEEE Transactions on Circuits and Systems for Video Technology, Vol. 17, No. 5, May 2007.

15. M. Shao, K. E. Barner, and R. C. Hardie, “Partition-Based Filters for Image Demosaicking and Super-Resolution Reconstruction,” Optical Engineering, 44, pp. 107003-1-107003-14, 2005.

16. M. S. Alam, J. G. Bognar, R. C. Hardie and B. J. Yasuda, “Infrared Image Registration and High Resolution Reconstruction Using Multiple Translationally Shifted Aliased Video Frames,” IEEE Transactions on Instrumentation and Measurement, Vol. 49, 2000, pp. 915-923.

17. T. R. Tuinstra and R. C. Hardie, “High Resolution Image Reconstruction From Digital Video by Exploitation on Non-Global Motion,” Optical Engineering, special issue on sampling in imaging systems, Vol. 38, No. 5, May 1999.

18. F. Baxley and R. C. Hardie, “Application of Multi-frame High Resolution Image Reconstruction to Digital Microscopy,” Applied Optics, Vol. 38, No. 11, April 1999.

19. R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong and E. A. Watson, “High Resolution Image Reconstruction From a Sequence ofRotated and Translated Frames and its Application to an Infrared Imaging System,” Optical Engineering, Vol. 37, No. 1, Jan. 1998, pp. 247-260.  Winner of the 1998 Rudolf Kingslake Medal and Prize.

20. R. C. Hardie, K. J. Barnard, and E. E. Armstrong, “Joint MAP Registration and High Resolution Image Estimation Using a Sequence of Undersampled Images,” IEEE Transactions on Image Processing, Vol. 6, No. 12, Dec. 1997, pp. 1621-1633.

21. J. C. Gillette, T. M. Stadtmiller, and R. C. Hardie, “Reduction of Aliasing in Staring Infrared Imagers Utilizing Subpixel Techniques,” Optical Engineering, Vol.34, No. 11, November 1995, pp. 3130-3137.

Image results from: F. Baxley and R. C. Hardie, “Application of Multi-frame High Resolution Image Reconstruction to Digital Microscopy,” Applied Optics, Vol. 38, No. 11, April 1999.

Multi-frame resolution enhancement applied to digital microscopy.  A single raw frame of a medical slide is shown on the left.  By moving the translation stage 24 frames were collected.   The multi-frame estimate using the 24 frame is shown on the right.  Note the significant improvement in image resolution using the multi-frame processing.