Bi-Manual Block Assembly via Sim-to-Real Reinforcement Learning

Abstract

Most successes in robotic manipulation have been restricted to single-arm gripper robots, whose low dexterity limits the range of solvable tasks to pick-and-place, insertion, and object rearrangement. More complex tasks such as assembly require dual and multi-arm platforms, but entail a suite of unique challenges such as bi-arm coordination and collision avoidance, robust grasping, and long-horizon planning. In this work we investigate the feasibility of training deep reinforcement learning (RL) policies in simulation and transferring them to the real world (Sim2Real) as a generic  methodology for obtaining performant controllers for real-world bi-manual robotic manipulation tasks. As a testbed for bi-manual manipulation, we develop the “U-Shape Magnetic Block Assembly Task”, wherein two robots with parallel grippers must connect 3 magnetic blocks to form a “U”-shape. Without a manually-designed controller nor human demonstrations, we demonstrate that with careful Sim2Real considerations, our policies trained with RL in simulation enable two xArm6 robots to solve the U-shape assembly task with a success rate of above 90% in simulation, and 50% on real hardware without any additional real-world fine-tuning. Through careful ablations, we highlight how each component of the system is critical for such simple and successful policy learning and transfer, including task specification, learning algorithm, direct joint-space control, behavior constraints, perception and actuation noises, action delays and action interpolation. Our results present a significant step forward for bi-arm capability on real hardware, and we hope our system can inspire future research on deep RL and Sim2Real transfer of bi-manual policies, drastically scaling up the capability of real-world robot manipulators.

U-Shape Block Assembly

To study the efficacy of simulated deep RL and Sim2Real transfer, we design the “U-Shape Magnetic Block Assembly Task” as a testbed for studying key challenges of bi-manual manipulation in the real world. In this task, two robot arms equipped with parallel grippers must connect 3 magnetic blocks in order to form a ”U” shape.

U-Shape Block Assembly in Real

U-Shape Block Assembly emergent behavior

Future Directions

Concurrent to this work, we have been studying bimanual assembly via blueprint assembly environments with simulated direct actuations. To briefly introduce that research, it demonstrated training of a single agent that can simultaneously solve all seen and unseen assembly tasks via a combination of large-scale RL, structured policies, and multi-task training.  Our current efforts in this directions can be viewed in the link below, with a video of our results presented below.