The first part of this project is to construct the hardware system, which includes a micro-manipulator, a microgripper with force sensing capability, a remote controller with haptic feedback, and a vision system that comprises two digital microscopes to provide visual feedback. Most of the commercial haptic devices provide 3 DoFs force feedback to humans, while limited systems have incorporated force, and tactile information. To this end, I will integrate new materials and novel actuators with commercial haptic devices to provide additional tactile feedback to operators. Therefore, the operators can handle the target micro-objects without damaging them by excessive force.

The second part of this project is to develop machine learning-based methods for user intention recognition. I will develop eye-tracking techniques in conjunction with other sensing modalities to recognize the operator’s intention, which paves a way for seamless human-robot cooperative control. The eye gaze information can be used to track the target objects and identify the regions of interest of the operator. With human intention recognition, teleoperation efficiency can be improved, since the micromanipulators can approach the target area with the desired pose.

The third part of this project is to embed intelligence into the micromanipulation system, so the robot can fulfill some repetitive tasks automatically and reduce the human operator’s workload. Imitation learning is a promising method to automate some of the micromanipulation tasks. However, traditional deep imitation learning has inherent black-box effects, which means that the decision-making process is not transparent to users.

Due to the high cost of failure caused by wrong decisions, robots are expected to have the ability of reasoning for failures in the case of executing wrong actions in the physical environment. Therefore, I aim to combine deep learning techniques with probability reasoning to ensure safety during micromanipulation.

The fourth part of this project is to incorporate adaptive mechanisms into the human-robot shared control framework. Machine learning-based context-awareness should be incorporated into the shared control framework. It can be used to determine the weight parameter for combining human operator commands obtained via remote controllers and robot commands determined by the regressed trajectory obtained via robot learning techniques. To this end, role adaptation between humans and robots for shared control can be achieved after combining user intention recognition and robot intelligence in an adjustable manner.