Cooperative Manipulation

To verify the usability of the proposed IoHIRT framework, we adopt human and multi-robot cooperative control in a complex, long-horizon manipulation task.  

The following figure provides an overview of the case study called cooperative manipulation. It visually depicts five transitional stages: i) the Mycobot320 robot arm (Elephant Robotics) propels the brown box towards the Mycobot280 robot arm (Elephant Robotics); ii) the Mycobot320  robot arm nudges the cube (resting atop the box) towards the Mycobot280's end-effector; iii) the Mycobot320 robot arm adjusts the position of its end-effector to optimize the in-hand camera's view; iv) the Mycobot280 robot arm extends towards the cube in preparation for grasping based on the view of the in-hand camera of the Mycobot320; v) the Mycobot280 robot arm grasps the cube and relocates it to a different position.  

The motivation behind designing this task lies in the fact that neither the Mycobot320 nor the Mycobot280 can independently complete the proposed task. The robots employed in this case study possess differing capabilities, speeds, and constraints. For instance, while the Mycobot320 lacks a suitable gripper, the Mycobot280 is devoid of an in-hand camera necessary for object observation. Furthermore, the workspace of the Mycobot280 is too restricted, thereby preventing it from reaching the target object (the cube) without repositioning the holder (the box). This necessitates the assistance of the Mycobot320 in adjusting the positions of the box and the cube for the purpose of grasping, as well as adjusting the camera view to provide better observation information to the Mycobot280. 

The collaborative control of multiple robots engaged in complex, long-horizon manipulation tasks presents a variety of challenges. These tasks frequently involve a sequence of interdependent actions, and managing these in a multi-robot scenario can be complex. Synchronization and coordination among multiple robots are paramount, as any deviation can result in task failure. Reliable and efficient communication between humans and robots is key. The success of the task completion demonstrated the usability of our proposed framework. 

Overview of the case study which involves humans and multi-robot cooperative control for a complex long-horizon manipulation task. 

Illustration of the five transitions, including
i) the Mycobot320 pushed the box to approach the Mycobot280;
ii) the Mycobot320 pushed the cube (on top of the box) to approach the end-effector of the Mycobot280;
iii) the Mycobot320 adjusted the position of its end-effector to generate a better view for the in-hand camera;
iv)   the Mycobot280 reached the cube for grasping;
v) the Mycobot280 grasped the cube and transferred the cube to another place.