Frequency-aware Event-based Video Deblurring for Real-World Motion Blur

Abstracts: 

Video deblurring aims to restore sharp frames from blurred video clips. Despite notable progress in video deblurring works, it is still a challenging problem because of the loss of motion information during the duration of the exposure time. Since event cameras can capture clear motion information asynchronously with high temporal resolution, several works exploit the event camera for deblurring as they can provide abundant motion information. However, despite these approaches, there were few cases of actively exploiting the long-range temporal dependency of videos. To tackle these deficiencies, we present an event-based video deblurring framework by actively utilizing temporal information from videos. To be specific, we first introduce a frequency-based cross-modal feature enhancement module. Second, we propose event-guided video alignment modules by considering the valuable characteristics of the event and videos. In addition, we designed a hybrid camera system to collect the first real-world event-based video deblurring dataset. For the first time, we build a dataset containing synchronized high-resolution real-world blurred videos and corresponding sharp videos and event streams. Experimental results validate that our frameworks significantly outperform the state-of-the-art frame-based and event-based deblurring works in the various datasets. 

Train: Download link

Test: Download link

FEVD dataset includes:

• Blurry Images (png) in 'blur_down' directory

• Sharp (GT) Images (png) in 'gt_down_corrected' directory

• Corresponding Raw Events (npz) in 'warped_events' directory

The directory of dataset is as follows:

• Images:

  • {root}/{split}/{sequence}/{class}/{name}.png

• Events:

  • {root}/{split}/{sequence}/{class}/{name}.npz

The meaning of the individual directory levels is:

• root : the root directory where the dataset is stored.

• split : train, test.

• sequence : the sequence name.

• class : blur image, gt image, events.

• name : index of the data.

The single event npz file contains the raw events (x,y,p,t) in the time range of the exposure time of the blurry image. You can obtain the particular event data as follows:

 ex) x_data = np.load('event_path')['x']

      y_data = np.load('event_path')['y']