Risk Aware Decision Making:
From Optimal Control to Reinforcement Learning

Abstract:

We present an historical overview about the connections between the analysis of risk and the control of autonomous systems. We propose three overlapping paradigms to classify the vast body of literature: the worst-case, risk-neutral, and risk-averse paradigms. We consider an appropriate assessment for the risk of an autonomous system to depend on the application at hand. In contrast, it is typical to assess risk using an expectation, variance, or probability alone. In addition, we unify the concepts of risk and autonomous systems. We achieve this by connecting approaches for quantifying and optimizing the risk that arises from a system’s behaviour across academic fields. The talk is highly multidisciplinary. We include research from the communities of reinforcement learning, stochastic and robust control theory, operations research, and formal verification. We describe both model-based and model-free methods, with emphasis on the former. Lastly, we highlight fruitful areas for further research. A key direction is to blend risk-averse model-based and model-free methods to enhance the real-time adaptive capabilities of systems to improve human and environmental welfare. This talk is based on a recent paper that has been accepted by the Journal of Artificial Intelligence as part of the Special Issue on Risk-aware Autonomous Systems: Theory and Practice. This is joint work with Yuheng Wang (Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto).

Abstract:

Over the last two decades, a host of studies in computational movement neuroscience have investigated human motor control as a continuous decision-making process in which uncertainty plays a key role. Leading theories of motor control, such as optimal feedback control, assume that motor behaviors can be explained as the optimization of a given expected payoff or cost, where uncertainty arises as risk based on the curvature of the corresponding utility functions. Here we discuss evidence that humans exhibit deviations from purely utility-based risk models. In particular, we study evidence for risk- and ambiguity-sensitivity in human motor behaviors demonstrating susceptibility with respect to the variability of motor costs or payoffs and sensitivity to model misspecification. We discuss in how far these sensitivities can be considered as a special case of a general decision-making framework that considers limited information-processing capabilities.

Abstract:

The unprecedented prediction accuracy of modern machine learning beckons for its application in a wide range of real-world applications, including autonomous robots and many others. A key challenge in such real-world applications is that the test cases are not well represented by the pre-collected training data. To properly leverage learning in such domains, especially safety-critical ones, we must go beyond the conventional learning paradigm of maximizing average prediction accuracy with generalization guarantees that rely on strong distributional relationships between training and test examples.

In this talk, I will describe a distributionally robust learning framework under data distribution shift. This framework yields appropriately conservative yet still accurate predictions to guide real-world decision-making and is easily integrated with modern deep learning. I will showcase the practicality of this framework in applications on agile robotic control. I will also introduce a survey of other real-world applications that would benefit from this framework for the future work.

• Shangzhe Yang: Feature-Based Risk-Averse Control in High-dimensional Markov Decision Processes

• James Zhu: Convergent iLQR for Underactuated Hybrid Systems

• Erfaun Noorani: Risk-Sensitive Reinforcement Learning via Exponential Criteria

Q&A

Abstract:

The ability of machine learning techniques to process rich sensory inputs such as vision makes them highly appealing for use in robotic systems (e.g., micro aerial vehicles and robotic manipulators). However, the increasing adoption of learning-based components in the robotics perception and control pipeline poses an important challenge: how can we guarantee the safety and performance of such systems? As an example, consider a micro aerial vehicle that learns to navigate using a thousand different obstacle environments or a robotic manipulator that learns to grasp using a million objects in a dataset. How likely are these systems to remain safe and perform well on a novel (i.e., previously unseen) environment or object? How can we learn control policies for robotic systems that provably generalize to environments that our robot has not previously encountered? Unfortunately, existing approaches either do not provide such guarantees or do so only under very restrictive assumptions.

In this talk, I will present our group’s work on developing a principled theoretical and algorithmic framework for learning control policies for robotic systems with formal guarantees on generalization and risk in novel environments. The key technical insight is to leverage and extend powerful techniques from generalization theory in theoretical machine learning. We apply our techniques on problems including vision-based navigation and grasping in order to demonstrate the ability to provide strong generalization guarantees and distributionally robust performance on robotic systems with complicated (e.g., nonlinear/hybrid) dynamics, rich sensory inputs (e.g., RGB-D), and neural network-based control policies.

• Kyra Rudy: Task-Agnostic Safety Constrained Shared Control for Human-Robot Systems

• Karan Muvvala: Near-optimal regret-minimizing strategies for strategic decision making

• Yiwei Lyu: Responsibility-associated Multi-agent Collision Avoidance with Social Preferences

• Rebecca Martin: Risk-Aware Collision Avoidance for Multi-drone Cinematography

Q&A

• Armand Jordana: Stagewise Newton Method for Dynamic Game Control with Imperfect State Observation

• Mingsong Ye: Risk-averse sequential decision problems with time-consistent stochastic dominance constraints

• Peter Amorese: Optimal Risk Avoidant Strategy Synthesis for Robotic Systems in Dynamic Environments

Q&A

Abstract:

AI-enabled autonomous systems promise to enable many future technologies such as autonomous driving, intelligent transportation, and robotics. Accelerated by the computational advances in machine learning and AI, there has been tremendous success in the development of autonomous systems over the past years. At the same time, however, new fundamental questions were raised regarding the safety of these increasingly complex systems that often operate in uncertain environments. In fact, such systems have been observed to take excessive risks in certain situations, often due to the use of neural networks which are known for their fragility. In this seminar, I will provide new insights in how to conceptualize risk for AI-enabled autonomous systems, and how to verify these systems in terms of their risk.

The main idea that I would like to convey in this talk is to use notions of spatial and temporal robustness to systematically define risk for autonomous systems. We are here particularly motivated by the fact that the safe deployment of autonomous systems critically relies on their robustness, e.g., against modeling or perception errors. In the first part of the talk, we will consider spatial robustness which can be understood in terms of safe tubes around nominal system trajectories. I will then show how risk measures, classically used in finance, can be used to quantify the risk of lacking robustness against failure, and how we can reliably estimate this robustness risk from finite data with high confidence. We will compare and verify four different neural network controllers in terms of their risk for a self-driving car in the autonomous driving simulator CARLA. In the second part of the talk, we will take a closer look at temporal robustness which has been much less studied than spatial robustness despite its importance, e.g., timing uncertainties in autonomous driving. I will introduce the notions of synchronous and asynchronous temporal robustness to quantify the robustness of system trajectories against various forms of timing uncertainties, and consecutively use risk measures to quantify the risk of lacking temporal robustness against failure. Finally, I am going to show that both notions of spatial and temporal robustness risk can be used for general forms of safety specifications including temporal logic specifications.

• Meng Song: Learning to Rearrange with Physics-Inspired Risk Awareness

• Haruki Nishimura & Jean Mercat: Risk-Biased Trajectory Forecasting for Safe Human-Robot Interaction

• Qi Heng Ho: Gaussian Belief Trees for Probabilistic Signal Temporal Logic Planning

• Aray Almen: Systemic Risk Measures for Multi-agent Systems

Q&A

Abstract:

Modern, critical applications require that stochastic decisions for estimation and control are made not only on the basis of minimizing average losses, but also safeguarding against less frequent, though possibly catastrophic events. Examples appear naturally in many areas, such as energy, finance, robotics, radar/lidar, networking and communications, autonomy, safety, and the Internet-of-Things. In such applications, the ultimate goal is to obtain risk-aware decision policies that optimally compensate against extreme events, even at the cost of slightly sacrificing performance under nominal conditions.

In the first part of the talk, we discuss a new risk-aware formulation of the classical and ubiquitous nonlinear MMSE estimation problem, trading between mean performance and risk by explicitly constraining the expected predictive variance of the squared error. We show that the optimal risk-aware solution can be evaluated stably and in closed-form regardless of the underlying generative model, as an appropriately biased, interpolated novel nonlinear MMSE estimator with a rational structure. We further illustrate the effectiveness of our approach via numerical examples, showcasing the advantages of risk-aware MMSE estimation against risk-neutral MMSE estimation, especially in models involving skewed and/or heavy-tailed distributions.

We then turn our attention to the stochastic LQR control paradigm. Driven by the ineffectiveness of risk-neutral LQR controllers at the presence of risky events, we present a new risk-constrained LQR formulation, which restricts the total expected predictive variance of the state penalty by a user-prescribed level. Again, the optimal controller can be evaluated in closed form. In fact, it is affine relative to the state, internally stable regardless of parameter tuning, and strictly optimal under minimal assumptions on the process noise (i.e., finite fourth-order moments), effectively resolving significant shortcomings of Linear-Exponential-Gaussian (LEG) control put forward by David Jacobson and Peter Whittle in the 1970-80's. The advertised advantages of the new risk-aware LQR framework are further illustrated via indicative numerical examples.