Joint Filtering of Intensity Images and Neuromorphic Events for High-Resolution Noise-Robust Imaging

We present a novel computational imaging system with high resolution and low noise.  Our system consists of a traditional video camera which captures high-resolution intensity images, and an event camera which encodes high-speed motion as a stream of asynchronous binary events. To process the hybrid input, we propose a unifying framework that first bridges the two sensing modalities via a noise-robust motion compensation model, and then performs joint image filtering. The filtered output represents the temporal gradient of the captured space-time volume, which can be viewed as motion-compensated event frames with high resolution and low noise. Therefore, the output can be widely applied to many existing event-based algorithms that are highly dependent on spatial resolution and noise robustness. In experimental results performed on both publicly available datasets as well as our contributing RGB-DAVIS dataset, we show systematic performance improvement in applications such as high frame-rate video synthesis, feature/corner detection and tracking, as well as high dynamic range image reconstruction.

Compared to traditional frame cameras, event cameras (e.g., DAVIS240) can capture high-speed motion (a), but bear low resolution and severe noise (b). Our system jointly filters between a high-resolution image (c) and high-speed events to produce a high-resolution low noise event frame (d),which can interface with downstream event-based algorithms with improved performance.

• Future frame prediction

With GEF, the reconstruction performance improves +1.53 (PSNR) and +0.0377 (SSIM) than DMR.

• Motion deblur

Compared to EDI, GEF is more effective in eliminating the event noise. 

• HDR

The reconstructed HDR image w/ GEF has higher contrast and less artifacts than w/o GEF.

• Corner detection and tracking

The corner points that are upsampled by the GEF can be tracked more accurately than the original frames.