Abstract: Visual reconstruction of fast non-rigid object deformations over time is a challenge for conventional frame-based cameras. In this paper, we propose a novel approach for reconstructing such deformations using measurements from event-based cameras. Under the assumption of a static background, where all events are generated by the motion, our approach estimates the deformation of objects from events generated at the object contour in a probabilistic optimization framework. It associates events to mesh faces on the contour and maximizes the alignment of the line of sight through the event pixel with the associated face. In experiments on synthetic and real data, we demonstrate the advantages of our method over state-of-the-art optimization and learning-based approaches for reconstructing the motion of human hands.

Our algorithm takes event streams as input and outputs the reconstructed object pose parameters, assuming a low-dimensional parameterized shape template of a deforming object (i.e. hand and body model). We propose a novel optimization-based method based on expectation maximization (EM). Our method models event measurements at contours in a probabilistic way to estimate the association likelihood of events to mesh faces and maximize the measurement likelihood. In the E-step, we estimate the association probability of each event w.r.t. all mesh faces; In the M-step, we first estimate the measurement likelihood and compute the expectation of the logarithmic likelihood. We use the expectation of the log-likelihood as our objective function and maximize it to update the pose parameter. We use the updated pose parameter to redo E-step. After several alternating between the E-step and the M-step, we arrive the desired pose parameter, which generates the motion described by observed events.

Inspired by state-of-the-art event simulators, we develop an event stream simulator which is multi-modal, more efficient, and supports more parametric body models.

Our simulator is / supports

• multi-modal: we simulate intensity frame, depth map, normal map, 2D motion field, and events

• adaptive sampling: based on motion field, it adaptively samples frame to generate events

• more efficient: due to parallel processing of all pixels on image frame

• more parametric model: supports SMPL body, MANO hand, and SMPL-X model

Video (Results)

Citation

@misc{https://doi.org/10.48550/arxiv.2210.06270,

doi = {10.48550/ARXIV.2210.06270},

url = {https://arxiv.org/abs/2210.06270},

author = {Xue, Yuxuan and Li, Haolong and Leutenegger, Stefan and Stückler, Jörg},

keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},

title = {Event-based Non-Rigid Reconstruction from Contours},

publisher = {arXiv},

year = {2022},

copyright = {arXiv.org perpetual, non-exclusive license}

}