SMASH-G: A System for Modeling, Analyzing, and Synthesizing Hand Gestures

Raw Data

Captured & Rendered into MANO Model

We note that the gestures call me and let's drink are not captured very well. For call me, the inherent occlusion involved in rotating the hand in the roll axis may be acting against the 3D hand pose estimator's performance. In let's drink, the inverse kinematic confuses the pinky with the ring finger.

Synthesized & Rendered into MANO Model

The gestures call me and let's drink are yet again failure cases in synthesized trajectories. However, looking at the synthesized results reveals something interesting. The gesture call me seems to show a cyclic structure as the fingers open and close towards the end of the animation. This might be largely due to the roll-axis rotation, which throws off the joint estimation until occlusion is cleared. Let's drink, on the other hand, captures the extension of the pinky joints, albeit all other joints are involved. In all cases, the learned dynamics model is encapsulating some information about the motions of the gestures, showing that each model is a feasible representation of gestures, provided enough data is given for accurate training.

Synthesized with Varying Speeds