Safe Autonomous Racing

We present results related to our paper, 'Safe Autonomous Racing via Approximate Reachability on Ego-vision'. We provide a brief overview of our method, then present qualitative results, such as videos and gifs, that cannot be effectively displayed in the paper.

Overview

Racing demands each vehicle to drive at its physical limits, when any safety infraction could lead to catastrophic failure. In this work, we study the problem of safe reinforcement learning (RL) for autonomous racing, using the vehicle's ego-camera view and speed as input.

Given the nature of the task, autonomous agents need to be able to 1) identify and avoid unsafe scenarios under the complex vehicle dynamics, and 2) make sub-second decision in a fast-changing environment. To satisfy these criteria, we propose to incorporate Hamilton-Jacobi (HJ) reachability theory, a safety verification method for general non-linear systems, into the constrained Markov decision process (CMDP) framework. HJ reachability not only provides a control-theoretic approach to learn about safety, but also enables low-latency safety verification. Though HJ reachability is traditionally not scalable to high-dimensional systems, we demonstrate that with neural approximation, the HJ safety value can be learned directly on vision context — the highest-dimensional problem studied via the method, to-date.

Classical Control Benchmarks: What learning rule is most effective for safety critic?

We compare HJ Bellman update to alternative learning rules for safety critic on two classical control benchmarks (Double Integrator and Dubins' Car), where the ground truth of safe vs. unsafe states are known, to evaluate the performance of different learning rules for safety analysis.

In our experiment, HJ Bellman update consistently outperforms SQRL and CQL, and has AUROC close to 1 on both tasks and very small variance over different runs.

The key intuition here is that the HJ safety value characterize the worst case outcome given best possible actions over an infinite horizon, while SQRL and CSC define the safety value function as the expected cumulative cost for safety violations. For safety-critical applications, such as autonomous driving and assistive robotics, we are concerned with whether failures can occur at all, not the probability of failure. Thus, HJ Bellman update is more appropriate for safety analysis.

(We refer interested readers to our paper for description of the methods and implementation details.)

Experiment: Safety Gym Benchmark

Qualitative Behavior Comparisons

While CPO and PPO-Lagrangian take into account that a certain number of violations are permissible, there is no such mechanism in SAGE, as HJ Reachability theory defines safety in an absolute sense. The inability to allow for some level of safety infractions, unfortunately, compromises performance.

Violations that do occur in SAGE result from neural approximation error, and the number of violations decreases over time as the safety actor-critic gains experience, despite the randomized and constantly-changing episodic layouts.

Experiment: Learn-to-Race Benchmark

We evaluate SAGE on Learn-to-Race (L2R), a recently-released, high-fidelity autonomous racing environment. This environment provides simulated racing tracks that are modeled after real-world counterparts, such as the famed Thruxton Circuit in the UK. Here, learning-based agents can be trained and evaluated according to challenging metrics and realistic vehicle and environmental dynamics, making L2R a compelling target for safe reinforcement learning. Each track features challenging components for autonomous agents, such as sharp turns (shown in (b)), where SAGE only uses ego-camera views (in (c)) and speed.

Qualitative Behavior Comparisons

We compare the learnable safety actor-critic in SAGE to a static safety actor-critic pre-computed from a nominal vehicle model. The videos here are from early stage of training to highlight the behavior of the safety actor-critic, because the safety actor intervenes less frequently as the performance agent gains experience.

Performance plots

SAGE learns safety directly from vision context and can recover from marginally safe states more smoothly. Having a safety actor-critic that is dedicated to learning about safety significantly boosted the initial safety performance of SAGE, in comparison to the SAC agent, even though both the performance and the safety actor-critics are randomly initialized. In practice, we envision the safety actor-critic to be warm-started with the nominal model or observational data, and fine-tuned by interactions with the environment