pwnlk
Video Deblurring via Semantic Segmentation and Pixel-Wise Non-Linear Kernel
Wenqi Ren1,2, Jinshan Pan3, Xiaochun Cao1, and Ming-Hsuan Yang3
1IIE, CAS, 2Tencent AI Lab 3Nanjing University of Science and Technology, 4University of California, Merced
Abstract
Video deblurring is a challenging problem as the blur is complex and usually caused by the combination of camera shakes, object motions, and depth variations. Optical flow can be used for kernel estimation since it predicts motion trajectories. However, the estimates are often inaccurate in complex scenes at object boundaries, which are crucial in kernel estimation. In this paper, we exploit semantic segmentation in each blurry frame to understand the scene contents and use different motion models for image regions to guide optical flow estimation. While existing pixel-wise blur models assume that the blur kernel is the same as optical flow during the exposure time, this assumption does not hold when the motion blur trajectory at a pixel is different from the estimated linear optical flow. We analyze the relationship between motion blur trajectory and optical flow, and present a novel pixel-wise non-linear kernel model to account for motion blur. The proposed blur model is based on the non-linear optical flow, which describes complex motion blur more effectively. Extensive experiments on challenging blurry videos demonstrate the proposed algorithm performs favorably against the state-of-the-art methods.
Model
Figure 1. Video motion blur. The green line represents the true motion blur trajectory of the highlighted pixels. The blur line denotes the estimated optical flow. The ground truth motion blur trajectory is smooth and different from optical flow. Based on this observation, we approximate the true motion blur trajectory using the PWNLK model (red line) obtained from a quadratic function of optical flow.
@inproceedings{Ren-ICCV-2017,
author = {Ren, Wenqi and Pan, Jinshan and Cao, Xiaochun and Yang, Ming-Hsuan},
title = {Video Deblurring via Semantic Segmentation and Pixel-Wise Non-Linear Kernel},
booktitle = {International Conference on Computer Vision},
year = {2017}
}