Joint Filtering of Intensity Images and Neuromorphic Events for High-Resolution Noise-Robust Imaging
Our RGB-DAVIS imaging system. We collocate an event camera and an RGB camera with a beam splitter.
Abstract
We present a novel computational imaging system with high resolution and low noise. Our system consists of a traditional video camera which captures high-resolution intensity images, and an event camera which encodes high-speed motion as a stream of asynchronous binary events. To process the hybrid input, we propose a unifying framework that first bridges the two sensing modalities via a noise-robust motion compensation model, and then performs joint image filtering. The filtered output represents the temporal gradient of the captured space-time volume, which can be viewed as motion-compensated event frames with high resolution and low noise. Therefore, the output can be widely applied to many existing event-based algorithms that are highly dependent on spatial resolution and noise robustness. In experimental results performed on both publicly available datasets as well as our contributing RGB-DAVIS dataset, we show systematic performance improvement in applications such as high frame-rate video synthesis, feature/corner detection and tracking, as well as high dynamic range image reconstruction.
Compared to traditional frame cameras, event cameras (e.g., DAVIS240) can capture high-speed motion (a), but bear low resolution and severe noise (b). Our system jointly filters between a high-resolution image (c) and high-speed events to produce a high-resolution low noise event frame (d),which can interface with downstream event-based algorithms with improved performance.
Paper
Zihao Wang, Peiqi Duan, Oliver Cossairt, Aggelos Katsaggelos, Tiejun Huang, and Boxin Shi, "Joint Filtering of Intensity Images and Neuromorphic Events for High-Resolution Noise-Robust Imaging", In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle , WA, USA, Jun. 2020.
Bibtex
@inproceedings{GEF,
author = {Wang, Zihao and Duan, Peiqi and Cossairt, Oliver and Katsaggelos, Aggelos and Huang, Tiejun and Shi, Boxin},
title = {Joint Filtering of Intensity Images and Neuromorphic Events for High-Resolution Noise-Robust Imaging},
booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
year = {2020}
}
paper
supplementary
Indoor:
Outdoor:
Applications
- Future frame prediction
With GEF, the reconstruction performance improves +1.53 (PSNR) and +0.0377 (SSIM) than DMR.
- Motion deblur
Compared to EDI, GEF is more effective in eliminating the event noise.
- HDR
The reconstructed HDR image w/ GEF has higher contrast and less artifacts than w/o GEF.
- Corner detection and tracking
The corner points that are upsampled by the GEF can be tracked more accurately than the original frames.