Robotic Tool Manipulation by Using Eye Tracking Virtual Reality and Robotic Multi-modality Feedback

Collaboration with Miguel Cienfuegos.

Other team member: Tobias Schumacher, Bassel Weiss, Christian Stollen, Jonas Blattgerste, Noureldin Badr, Sayandev Paul

Virtual reality (VR) is a computer technology which makes users wearing virtual reality equipment be able to "look around" the artificial world. It has been widely used in game development, movie and entertainment domain. Recent year, it was also used in robotic grasping tasks.

In this project, we propose to use eye tracking and virtual reality technology to help robot using a tool in a semi-autonomous way. To this end, (1) Human intention will be detected and tracked by the eye tracking system. (2) Rendered virtual environment will serve as feedback to guide user to move a tool. (3) Because of the partly occluded scenario in the virtual world, Vision and force perception in robotic system will quickly adapt robot behaviour while the tool is physically interacting with environment in an online fashion.

Following topics will be focused on:

(1) Reconstructing the scenario by data fusion method. e.g. combining data from HTC vive and RGB-D camera.

(2) Goal detector by eye tracking system in virtual really environment.

(3) Controling simulated/real Kuka LWR to manipulate a rigid or an articulated tool.

Experimental setup (VR)

Robot setup

Video from VR