Human-in-the-loop Pose Estimation via Shared Autonomy

Research Summary

Reliable, efficient shared autonomy requires balancing human operation and robot automation on complex tasks, such as dexterous manipulation. Adding to the difficulty of shared autonomy is a robot's limited ability to perceive the 6 degree-of-freedom pose of objects, which is essential to perform manipulations those objects afforded. Inspired by Monte Carlo Localization, we propose a generative human-in-the-loop approach to estimating object pose. We characterize the performance of our mixed-initiative 3D registration approach using 2D pointing devices via a user study. Seeking an analog for Fitts's Law for 3D registration, we introduce a new evaluation framework that takes the entire registration process into account instead of only the outcome. When combined with estimates of registration confidence, we posit that mixed-initiative registration will reduce the human workload while maintaining or even improving final pose estimation accuracy.

This video shows the entire registration process of our system, followed by the robot executing the task.

Related Publications:

  1. Ye, Zhefan, Jean Y. Song, Zhiqiang Sui, Stephen Hart, Jorge Vilchis, Walter S. Lasecki, Odest Chadwicke Jenkins. “Human-in-the-loop Pose Estimation via Shared Autonomy.” In ACM Conference on Intelligent User Interfaces (IUI), 2021 [paper] Best Paper Honorable Mention

  2. Ye, Zhefan, Odest Chadwicke Jenkins, Zhiqiang Sui, and Stephen Hart. "Human-in-the-loop Affordance Registration via Pose Estimation." In Workshop of Human-Centered Robotics: Interaction, Physiological Integration and Autonomy, Robotics Science and Systems, 2017