Schedule

Abstract: Skydio aspires to build intelligent autonomous systems that operate in the real world and do useful work to keep people safer and help them do their jobs better. As robots are deployed at scale, the field faces a new set of ethical questions, especially in public safety and defense where the stakes are high: the potential impact is positive, but the potential for abuse is also very real.

In this talk, I will share our experiences building our technology stack that has shipped in 40,000 drones that serve over 1,900 organizations across 20 different industries from energy to transportation to construction. I will also share our learnings and perspective on serving public safety and defense customers - how we approach the important ethical and practical questions involved in deploying robots in the highest stakes scenarios, and what we’ve learned along the way.

Bio: Adam Bry is co-founder and CEO at Skydio, the leading U.S. drone company, and the world leader in autonomous flight. He has over two decades of experience with drones, starting at 16 when he was a national champion R/C airplane aerobatics pilot. As a grad student at MIT, he helped lead an award-winning research program that pioneered autonomous flight for drones, transferring much of what he learned as an R/C pilot into software that enables drones to fly themselves. After graduate school, Adam co-founded Google[x]’s Project Wing. He has co-authored numerous technical papers and patents, and was recognized on MIT’s TR35 list for young innovators. In 2021, Adam was appointed to the FAA’s Advanced Aviation Advisory Committee.

Adam holds a Bachelor of Science in Mechanical Engineering from Olin College. He received his Master of Science degree in Aerospace Engineering from MIT.

Abstract: Safe autonomous driving critically depends on how well the ego-vehicle can predict the trajectories of neighbouring vehicles. To this end, several trajectory prediction algorithms have been presented in the existing literature which outputs a multi-modal distribution of obstacle trajectories instead of a single deterministic prediction to account for the underlying uncertainty. However, existing planners cannot handle the multi-modality based on just sample-level information of the predictions. 

With this motivation, this paper proposes a trajectory optimizer that can leverage the distributional aspects of an arbitrarily complex distribution, in particular, an output distribution represented as a deep neural network, in a computationally tractable and sample-efficient manner. The core of our approach is built on embedding distribution in Reproducing Kernel Hilbert Space (RKHS) , which we leverage in two ways. First, we propose an RKHS embedding approach to select probable samples from the obstacle trajectory distribution. Second, we rephrase chance-constrained optimization as distribution matching in RKHS and propose a novel sampling-based optimizer for its solution.

Abstract: This paper proposes an approach for solving a reach-avoid motion planning problem given a mobile robot with black-box dynamics. We consider the specific case where obstacles are near the robot’s goal, causing a near danger (i.e., narrow gap) scenario. We apply neural networks to learn a space of motion plans, which the robot tracks with a feedback controller. This results in a lumped type of uncertainty that we call tracking error to represent the fact that the robot cannot perfectly track any plan. We show how our recent work on Piecewise Affine Reachability Computation (PARC) combines elegantly with feedforward, rectified linear unit (ReLU)-activated neural network planners to discover safe, goal-reaching trajectories near danger. For this short paper, we demonstrate the approach on a 2-D double integrator as a preliminary example.

Abstract: In this paper, we seek to learn a robot policy guaranteed to satisfy state constraints. Typical RL algorithms only encourage constraint satisfaction through reward shaping. However, such soft constraints cannot offer safety guarantees. To address this gap, we propose POLICEd RL, a novel RL algorithm explicitly designed to enforce affine hard constraints in closed-loop with a black-box environment. Our key insight is to make the learned policy be affine around the unsafe set and to use this affine region as a repulsive buffer to prevent trajectories from violating the constraint. We prove that such policies guarantee constraint satisfaction. Our results demonstrate the capacity of POLICEd RL to enforce hard constraints in robotic tasks while significantly outperforming existing methods.

Abstract: An outstanding challenge for robotic systems like autonomous vehicles is ensuring safe interaction with humans without sacrificing performance. Existing safety methods often neglect the robot’s ability to learn and adapt at runtime, leading to overly conservative behavior. This paper proposes a new closed-loop paradigm for synthesizing safe control policies that explicitly account for the robot’s evolving uncertainty and its ability to quickly respond to future scenarios as they arise, by jointly considering the physical dynamics and the robot’s learning algorithm. We leverage adversarial reinforcement learning for tractable safety analysis under high-dimensional learning dynamics and demonstrate our framework’s ability to work with both Bayesian belief propagation and implicit learning through large pre-trained neural trajectory predictors.

Abstract: Quadrotors are among the most agile flying robots. Despite recent advances in learning-based control and computer vision, autonomous drones still rely on explicit state estimation. On the other hand, human pilots only rely on a first-person-view video stream from the drone onboard camera to push the platform to its limits and fly robustly in unseen environments. To the best of our knowledge, we present the first vision-based quadrotor system that autonomously navigates through a sequence of gates at high speeds while directly mapping pixels to control commands. Like professional drone-racing pilots, our system does not use explicit state estimation and leverages the same control commands humans use (collective thrust and body rates). We demonstrate agile flight at speeds up to 40km/h with accelerations up to 2g. This is achieved by training vision-based policies with reinforcement learning (RL). The training is facilitated using an asymmetric actor-critic with access to privileged information. To overcome the computational complexity during image-based RL training, we use the inner edges of the gates as a sensor abstraction. This simple yet robust, task-relevant representation can be simulated during training without rendering images. During deployment, a Swin-transformer-based gate detector is used. Our approach enables autonomous agile flight with standard, off-the-shelf hardware. Although our demonstration focuses on drone racing, we believe that our method has an impact beyond drone racing and can serve as a foundation for future research into real-world applications in structured environments.

Abstract: We address the problem of large-scale traffic management for autonomous vehicles. We derive a safety risk prediction framework for autonomous vehicle interactions at road intersections using Graph Neural Networks (GNN) as well as a unique “traffic signal” control plan for each intersection. The plan controls the status and duration of the traffic lights at road intersections while addressing the safety implications of the chosen plan. 

Initially, due to existing limitations on the availability of Urban Air Mobility (UAM) vehicle trajectory datasets, we develop safe traffic control algorithms for autonomous vehicle interactions based on both simulation and real-world data obtained from ground vehicles, applicable to traffic management of autonomous ground vehicles. Next, we present an approach that adapts the obtained results to scenarios involving electrical vertical takeoff and landing vehicles (eVTOLs) for aerial transportation in urban areas by extending the concept of 2 dimensional highways into a 2.5-dimensional air-highway network on which vehicles travel.

To the best of our knowledge, we propose the first framework that integrates both safety constraints as well as adaptive traffic signal control reinforcement learning based protocols for safe and efficient movement coordination on air highways.

Abstract: Rapid advances in perception have enabled large pre-trained models to be used out of the box for transforming high-dimensional, noisy, and partial observations of the world into rich occupancy representations. However, the reliability of these models and consequently their safe integration onto robots remains unknown when deployed in environments unseen during training. In this work, we address this challenge by rigorously quantifying the uncertainty of pre-trained perception systems for object detection via a novel calibration technique based on conformal prediction. As a result, the calibrated perception system can be used in combination with any safe planner to provide an end-to-end statistical assurance on safety in unseen environments. We evaluate the resulting approach, Perceive with Confidence (PwC), with experiments in simulation and on hardware where a quadruped robot navigates through previously unseen indoor, static environments. These experiments validate the safety assurances for obstacle avoidance provided by PwC and demonstrate up to 40% improvements in empirical safety compared to baselines. Videos and code can be found at https://perceive-with-confidence.github.io.

Abstract: Modern neural trajectory predictors in autonomous driving are developed using imitation learning (IL) from driving logs. Although IL benefits from its ability to glean nuanced and multi-modal human driving behaviors from large datasets, the resulting predictors often struggle with out-of-distribution (OOD) scenarios and with traffic rule compliance. On the other hand, classical rule-based predictors, by design, can predict traffic rule satisfying behaviors while being robust to OOD scenarios, but these predictors fail to capture nuances in agent-to-agent interactions and human driver's intent. In this paper, we present RuleFuser, a posterior-net inspired evidential framework that combines neural predictors with classical rule-based predictors to draw on the complementary benefits of both, thereby striking a balance between performance and traffic rule compliance. The efficacy of our approach is demonstrated on the real-world nuPlan dataset where RuleFuser leverages the higher performance of the neural predictor in in-distribution (ID) scenarios and the higher safety offered by the rule-based predictor in OOD scenarios.

Abstract: Legged robots navigating cluttered environments must be jointly agile for efficient task execution and safe to avoid collisions with obstacles or humans. Existing studies either develop conservative controllers (< 1.0 m/s) to ensure safety, or focus on agility without considering potentially fatal collisions. This paper introduces Agile But Safe (ABS), a learning-based control framework that enables agile and collision-free locomotion for quadrupedal robots. ABS involves an agile policy to execute agile motor skills amidst obstacles and a recovery policy to prevent failures, collaboratively achieving high-speed and collision-free navigation. The policy switch in ABS is governed by a learned control-theoretic reach-avoid value network, which also guides the recovery policy as an objective function, thereby safeguarding the robot in a closed loop. The training process involves the learning of the agile policy, the reach-avoid value network, the recovery policy, and an exteroception representation network, all in simulation. These trained modules can be directly deployed in the real world with onboard sensing and computation, leading to high-speed and collision-free navigation in confined indoor and outdoor spaces with both static and dynamic obstacles.

Abstract: Robot behavior policies trained via imitation learning are prone to failure under conditions that deviate from their training data. Thus, algorithms that monitor learned policies at test time and provide early warnings of failure are necessary to facilitate scalable deployment. We propose a runtime monitoring framework that splits the detection of failures into two complementary categories: 1) Erratic failures, which we propose to detect using a statistical measure of temporal action consistency, and 2) task progression failures, where we use VLMs to detect when the policy confidently and consistently takes actions that do not solve the task. Our approach has two key strengths. First, because learned policies exhibit diverse failure modes, combining complementary detectors leads to significantly higher accuracy at failure detection. Second, using a statistical temporal action consistency measure ensures that we quickly detect when multi- modal, generative policies exhibit erratic behavior at negligible computation cost. In contrast, we only use VLMs to detect failure modes that are less time-sensitive. We demonstrate our approach in the context of diffusion policies trained on a multi-modal domain and simulated bi-manual robotic manipulation domains. The resultant system, unifying temporal consistency detection with VLM runtime monitoring, detects 15% more failures than using either of the two detectors, thus highlighting the importance of assigning specialized detectors to complementary categories of failure.

Abstract: Learning-based neural network (NN) control policies have shown impressive empirical performance in a wide range of tasks in robotics and control. However, formal (Lyapunov) stability guarantees over the region-of-attraction (ROA) for NN controllers with nonlinear dynamical systems are challenging to obtain, and most existing approaches rely on expensive solvers such as sums-of-squares (SOS), mixed-integer programming (MIP), or satisfiability modulo theories (SMT). In this paper, we demonstrate a new framework for learning NN controllers together with Lyapunov certificates using fast empirical falsification and strategic regularizations. We propose a novel formulation that defines a larger verifiable region-of-attraction (ROA) than shown in the literature, and refines the conventional restrictive constraints on Lyapunov derivatives to focus only on certifiable ROAs. The Lyapunov condition is rigorously verified post-hoc using branch-and-bound with scalable linear bound propagation-based NN verification techniques. The approach is efficient and flexible, and the full training and verification procedure is accelerated on GPUs without relying on expensive solvers for SOS, MIP, nor SMT. The flexibility and efficiency of our framework allow us to demonstrate Lyapunov-stable output feedback control with synthesized NN-based controllers and NN-based observers with formal stability guarantees, for the first time in literature.

Abstract: We provide a concise overview of our progress in identifying and certifying globally optimal solutions of state estimation problems in robotics. We give a summary of the theoretical background, providing pointers for novices in the field to learn about this topic. We then given an overview of the problems we have treated, putting them in a unified framework to make it easier to find, compare, and build upon them. We discuss our methods for simplifying the generation and analysis of these solutions, and finally, present advances to allow to embed them in real-world robotics pipelines. We conclude with a discussion of important open research problems.

Abstract: Ensuring safety for learning-based techniques, especially reinforcement learning, is a critical step toward deploying robot learning in the real world. This problem poses several challenges, such as deriving safety policies from given constraints, tackling challenges in dynamic environments, designing constraints that consider long-term safety, and addressing uncertainty. ATACOM attempts to construct a Constraint Manifold and a safe action space tangent to this manifold, exploiting the prior knowledge of robot dynamics and differentiable constraints. This paper provides an overview of the ATACOM method of handling various types of constraints, including learnable ones. We show that ATACOM effectively incorporates prior knowledge into learning-based methods to solve high-dimensional tasks with complex constraints.

Abstract: This paper presents an overview of the Burer-Monteiro method (BM), a technique that has been applied to solve robot perception problems to certifiable optimality in real-time. BM is often used to solve semidefinite programming relaxations, which can be used to perform global optimization for non-convex perception problems. Specifically, BM leverages the low-rank structure of typical semidefinite programs to dramatically reduce the computational cost of performing optimization. This paper discusses BM in certifiable perception, with three main objectives: (i) to consolidate information from the literature into a unified presentation, (ii) to elucidate the role of the linear independence constraint qualification (LICQ), a concept not yet well-covered in certifiable perception literature, and (iii) to share practical considerations that are discussed among practitioners but not thoroughly covered in the literature. Our general aim is to offer a practical primer for applying BM towards certifiable perception.

Abstract: Event cameras are bio-inspired vision sensors with much lower latency and bandwidth than standard cameras. This talk will present current trends and opportunities with event cameras to increase safety and robustness of autonomous systems.