Reinforcement Learning Enables Real-Time Planning and Control of Agile Maneuvers for Soft Robot Arms

Rianna Jitosho*, Tyler Ga Wei Lum*, Allison Okamura, C. Karen Liu

Conference on Robot Learning (CoRL) 2023

*equal contribution

Key Takeaway

This is the first work that demonstrates real-time planning and control of agile maneuvers by soft robot arms, which is achieved by using reinforcement learning and key insights with simulation and actuator modeling to overcome sim-to-real challenges for zero-shot sim-to-real transfer.

While soft robots typically require complex models to be simulated precisely, we find that a simple physics-based dynamic model is sufficient to train a control policy to perform highly dynamic behaviors in the real world.

Abstract

Control policies for soft robot arms typically assume quasi-static motion or require a hand-designed motion plan. To achieve real-time planning and control for tasks requiring highly dynamic maneuvers, we apply deep reinforcement learning to train a policy entirely in simulation, and we identify strategies and insights that bridge the gap between simulation and reality. In particular, we strengthen the policy’s tolerance for inaccuracies with domain randomization and implement crucial simulator modifications that improve actuation and sensor modeling, enabling zero-shot sim-to-real transfer without requiring high-fidelity soft robot dynamics. We demonstrate the effectiveness of this approach with experiments on physical hardware and show that our soft robot can reach target positions that require dynamic swinging motions. This is the first work to achieve such agile maneuvers on a physical soft robot, advancing the field of soft robot arm planning and control.

Our Framework

In this work, we introduce a framework that combines deep reinforcement learning with dynamic models to achieve real-time planning and control of agile maneuvers for soft robot arms. In our proposed framework, the first step is to fit parameters for a physics-based dynamic model using about 30 seconds of real experimental data. In the second step, we train a control policy using deep reinforcement learning and a simulator that uses the dynamic model from the previous step. In the third step, we deploy the learned control policy directly on real hardware.

Zero-shot Sim2Real Transfer

While control policies learned in simulation often do not transfer to the real world, we overcome this challenge with crucial improvements to our actuation and sensing models in simulation, enabling zero-shot sim-to-real transfer of the learned control policy.

Video

We provide a detailed project video that summarizes our proposed framework and presents our experimental results.

Hardware

Our inflated-beam robot uses a pneumatic bending actuator and a 1D mobile base to perform agile maneuvers such as swinging onto a target surface. We use a motion capture system to measure the robot's state and compute control actions with a policy trained using reinforcement learning.