Video Superresolution

Super-Resolution with Gradient Profile Prior and Jointed Bilateral Filtering

The gradient profile is a 1-D profile along the gradient direction of the zero-crossing pixel in the image. The gradient profile prior is a parametric distribution describing the shape and the sharpness of the gradient profiles in natural image. It is shown [2] that the shape statistics of the gradient profiles in natural image is quite stable and invariant to the image resolution. With this stable statistics, the statistical relationship of the sharpness of the gradient profile between the HR image and the LR image can be learned. Using the learned gradient profile prior and relationship, it is possible to provide a constraint on the gradient field of the HR image.

Fig. 1, Gradient Profile Prior

Basically the joint or cross bilateral filter is a way to filter an image with respect to the geometry of objects. When you filter an image (e.g blurring it) you're mixing a pixel value with its neighbourhood so it can remove aliasing, noise, and undesirable features. But it happens that you're averaging values that possibly don't belong to the same object, and can lead to incoherent values. That's what cross bilateral filtering tries to solve. Instead of having only one source image, you add others that help you to identify objects (basically z-buffer and normal buffer in the case of 3d application). So when filtering, you can weight neighbourhood pixel values by their screen space distance (that's usually what a basic filter does), but also weight them with an expression taking into account the depth difference (and normals difference) to try to check if both pixels belong to the same object. This way you're averaging colors of objects respecting their edges, and not mixing with other objects. It can be applied for upsampling if LR and HR are joined [1] to avoid aliasing.

Fig. 2, Cross or joint bilateral filter

Guided interpolation with gradient information is useful for image editing [3]. Shown in Fig. 3, unknown function f interpolates in image definition domain the destination function f*, under guidance of vector field v, which might be or not the gradient field of a source function g. This is formulated to be a Poisson equation with Dirichlet boundary conditions.

Fig. 3, Gradient Interpolation with Poisson Equation

A gradient field transformation approach [2] can approximate the HR gradient field by transforming the LR gradient field using the gradient profile prior. Shown in Fig. 4, (a) left and middle subfigures illustrate a gradient profile passing through x and x0 in the up-sampled image and the gradient of x is transformed to its HR version (right) by multiplying a ratio. (b) and (c) are an up-sampled image and its gradient field. (d) and (e) are transformed gradient field and reconstructed image by solving Poisson equation, seen in Fig. 3.

Fig. 4, Gradient transform for backprojection

Here a framework is built to extend work of SR by gradient profile with joint bilateral filtering. A LR image is upsampled first by cubic interpolation, and its gradient filed is transformed by using gradient profile. Based on enhanced gradient, an approximated HR image is reconstructed by solving Possion equation, in which a joint bilateral filter is applied in gradient-illumination domain.

Fig. 5, Flowchart

Fig. 6, Demo.

References

1. J. Kopf, M. F. Cohen, D Lischinski, M Uyttendaele, Joint bilateral upsampling, ACM SIGGRAPH 2007.

2. J. Sun, J. Sun, Z. Xu, and H. Shum. Image superresolution using gradient profile prior, CVPR, 2008.

3. P Pérez, M Gangnet, A Blake, Poisson image editing, ACM SIGGRAPH 2003.

4. Y Tai, S. Liu, M. S. Brown and S. Lin, Super Resolution using Edge Prior and Single Image Detail Synthesis, CVPR, 2010.