Reconstructing Robot Motion from Video

In robotics, the concept of learning often relates to direct instruction or methods that rely on reinforcement and punishment. However, in nature, a great deal of learning happens more subtly and depends on observing others and imitating their behaviors. The natural way of learning behavior can also be applied to robotics, where the robot videos on social media can be utilized as a learning resource. The main challenge is how to model the robot motion from the videos, where the environments are unpredictable, and calibration is infeasible. In this work, we present a framework for reconstructing the robot motion from videos; this is realized by tracking the keypoints pertaining to the robot body parts and reconstructing the robot trajectories with differentiable state estimation. We demonstrate that the method works on two common classes of articulated robots: quadruped and manipulator robots. We take raw videos found online from YouTube of the Boston Dynamics Spot quadruped, and then solve for and replay the motions in a high-fidelity robot simulator model. We also show that the learning robot need not learn from video of an identical robot, and the demonstration can translate across robots of the same class, i.e., from Rethink Baxter manipulator to KUKA LBR iiwa.