Low Level Control of a Quadrotor with
Deep Model-Based Reinforcement Learning
Nathan Lambert, Daniel Drew, Joseph Yaconelli, Roberto Calandra, Sergey Levine, and Kristofer Pister
Contact: nol@berkeley,edu
ieee: https://ieeexplore.ieee.org/document/8769882 , arxiv: https://arxiv.org/abs/1901.03737
Abstract:
Designing effective low-level robot controllers often entail platform-specific implementations that require manual heuristic parameter tuning, significant system knowledge, or long design times. With the rising number of robotic and mechatronic systems deployed across areas ranging from industrial automation to intelligent toys, the need for a general approach to generating low-level controllers is increasing. To address the challenge of rapidly generating low-level controllers, we argue for using model-based reinforcement learning (MBRL) trained on relatively small amounts of automatically generated (i.e., without system simulation) data. In this letter, we explore the capabilities of MBRL on a Crazyflie centimeter-scale quadrotor with rapid dynamics to predict and control at < 50 Hz. To our knowledge, this is the first use of MBRL for controlled hover of a quadrotor using only on-board sensors, direct motor input signals, and no initial dynamics knowledge. Our controller leverages rapid simulation of a neural network forward dynamics model on a graphic processing unit enabled base station, which then transmits the best current action to the quadrotor firmware via radio. In our experiments, the quadrotor achieved hovering capability of up to 6 s with 3 min of experimental training data.
Experiments:
Rollout 0
These are some example flights of the Crazyflie on random policy, which was our initial data for training a forward dynamics model.
Rollout 1
First on policy data. There are noticeable different control strategies, but the average flight length is still short.
The rapid learned response. The top left is the pitch of the quadrotor for random rollouts, and then the others are the first few model iterations. After 30 short flights, the quadrotor can fly for over 2.5s.
Performance Summary
The learning curves verses rollout at 25 and 50 Hz. The two curves show the data dependency of learning a dynamics model, but both show the predictive power of the model-based model predictive controller.
Flight Performance
This video is mostly at 2x slow down to accentuate the control response, but the real time flight was about 6 seconds. The controller was trained on less than 5,000 datapoints for this experiment.
Future Work
The initial results in MBRL on the Crazyflie open many doors for applications of our system. One such application, to the left, is the Ionocraft, a flying micro-robot with no moving parts that uses the same onboard IMU as the Crazyflie (MPU-9250). Additionally, much of the future work will go to ironing out system limitations in flight and pursuing improvements in controller design. Three potential controller changes could be to use a low frequency, long time horizon MPC, a generated policy on a more substantial learned dynamics model than can be deployed at high frequency, or generating standard controllers, such as a PID, to create onboard controllers.