Joint Filtering of Intensity Images and Neuromorphic Events for High-Resolution Noise-Robust Imaging


Our RGB-DAVIS imaging system. We collocate an event camera and an RGB camera with a beam splitter.

Abstract

We present a novel computational imaging system with high resolution and low noise. Our system consists of a traditional video camera which captures high-resolution intensity images, and an event camera which encodes high-speed motion as a stream of asynchronous binary events. To process the hybrid input, we propose a unifying framework that first bridges the two sensing modalities via a noise-robust motion compensation model, and then performs joint image filtering. The filtered output represents the temporal gradient of the captured space-time volume, which can be viewed as motion-compensated event frames with high resolution and low noise. Therefore, the output can be widely applied to many existing event-based algorithms that are highly dependent on spatial resolution and noise robustness. In experimental results performed on both publicly available datasets as well as our contributing RGB-DAVIS dataset, we show systematic performance improvement in applications such as high frame-rate video synthesis, feature/corner detection and tracking, as well as high dynamic range image reconstruction.

(a) Joint plot of image & events
(b)Motion-compensated events
(c) Captured image
(d) Filtered events

Compared to traditional frame cameras, event cameras (e.g., DAVIS240) can capture high-speed motion (a), but bear low resolution and severe noise (b). Our system jointly filters between a high-resolution image (c) and high-speed events to produce a high-resolution low noise event frame (d),which can interface with downstream event-based algorithms with improved performance.

Paper

  • Zihao Wang, Peiqi Duan, Oliver Cossairt, Aggelos Katsaggelos, Tiejun Huang, and Boxin Shi, "Joint Filtering of Intensity Images and Neuromorphic Events for High-Resolution Noise-Robust Imaging", In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle , WA, USA, Jun. 2020.

Bibtex

@inproceedings{GEF,

author = {Wang, Zihao and Duan, Peiqi and Cossairt, Oliver and Katsaggelos, Aggelos and Huang, Tiejun and Shi, Boxin},

title = {Joint Filtering of Intensity Images and Neuromorphic Events for High-Resolution Noise-Robust Imaging},

booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},

year = {2020}

}

paper


supplementary


Dataset



Indoor:

#1
#6
#2
#7
#3
#8
#4
#9
#5
#10



Outdoor:

#11
#16
#12
#17
#13
#18
#14
#19
#15
#20

Applications

  • Future frame prediction

With GEF, the reconstruction performance improves +1.53 (PSNR) and +0.0377 (SSIM) than DMR.

  • Motion deblur

Compared to EDI, GEF is more effective in eliminating the event noise.

  • HDR

The reconstructed HDR image w/ GEF has higher contrast and less artifacts than w/o GEF.

  • Corner detection and tracking

The corner points that are upsampled by the GEF can be tracked more accurately than the original frames.