Schedule

Abstract: Reinforcement learning has achieved significant breakthroughs recently for outperforming humans in many challenging tasks. Behind the scene lies the integration of three key elements: nonlinear function approximation, off-policy learning, and efficient exploration, among various other techniques such as bootstrapping, target networks, experience replay, entropy regularization, etc. On the other side, the stability of RL algorithms is often haunted by the so-called deadly triad. This lecture aims to unveil the mysteries behind these RL techniques and reconcile them from fresh optimization perspectives.

08:30 - 10:20: Tabular RL – old and new (Niao He)

10:20 - 10:40: Break

10:40 - 12:30: RL with nonlinear function approximation (Niao He)

14:00 - 15:50: Offline RL (Bo Dai)

15:50 - 16:10: Break

16:10 - 18:00: RL with exploration (Bo Dai)

References:

• [1] Semih Cayci, Siddhartha Satpathi, Niao He and R. Srikant. "Sample Complexity and Overparameterization Bounds for Projection-Free Neural TD Learning". arXiv preprint arXiv:2103.01391, 2021

• [2] Donghwan Lee and Niao He. "A Unified Switching System Perspective and Convergence Analysis of Q-Learning Algorithms". Neural Information Processing Systems (NeurIPS), 2020.

• [3] Wentao Weng, Harsh Gupta, Niao He, Lei Ying and R. Srikant. "The Mean-Square Error of Double Q-Learning". Neural Information Processing Systems (NeurIPS), 2020.

• [4] Donghwan Lee and Niao He. "Target-Based Temporal-Difference Learning." International Conference on Machine Learning (ICML), 2019.

• [5] Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári and Dale Schuurmans. "CoinDICE: Off-Policy Confidence Interval Estimation". Neural Information Processing Systems (NeurIPS), 2020.

• [6] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li and Dale Schuurmans. "AlgaeDICE: Policy Gradient from Arbitrary Experience". arXiv preprint arXiv:1912.02074, 2019.

• [7] Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen and Le Song. "SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation". In International Conference on Machine Learning (ICML), 2018.

• [8] Bo Dai, Niao He, Yunpeng Pan, Byron Boots and Le Song. "Learning from Conditional Distributions via Dual Embeddings". In Artificial Intelligence and Statistics (AISTATS), 2017.

Abstract: In these lectures, we will present an overview of recent connections that have been made between nonlinear stability theory, online learning, and statistical learning theory. In particular, we will focus on three settings: (i) learning stability certificates for nonlinear systems from data, (ii) regret bounds for nonlinear adaptive control, and (iii) safe imitation learning. We will highlight how learning algorithms applied to systems satisfying appropriate notions of incremental stability can be shown to enjoy favorable generalization guarantees. The lectures will be as self-contained as possible, with no background in learning theory expected of the audience.

The schedule for this day is different from the other days.

14:00 - 15:50: Lecture

15:50 - 16:10: Break

16:10 - 18:00: Lecture

19:00 - 20:50: Lecture

20:50 - 21:10: Break

21:10 - 23:00: Lecture

Relevant work:

• [BTMSS20] Nicholas M. Boffi, Stephen Tu, Nikolai Matni, Jean-Jacques E. Slotine and Vikas Sindhwani, "Learning Stability Certificates From Data", 2020. arXiv: https://arxiv.org/abs/2008.05952

• [TRM21] Stephen Tu, Alexander Robey and Nikolai Matni, "Closing the Closed-Loop Distribution Shift in Safe Imitation Learning", 2021. arXiv: http://arxiv.org/abs/2102.09161

• [BTS20] Nicholas M. Boffi, Stephen Tu, Jean-Jacques E. Slotine, "Regret Bounds for Adaptive Nonlinear Control", 2021. arXiv: https://arxiv.org/abs/2011.13101

Abstract: The lectures focus on a recently introduced approach to design control policies for unknown systems starting from low-complexity input-output data collected during off-line experiments. The approach reduces the design to the solution of data-dependent semidefinite programs, which provide a computationally effective way to deal with the problem of learning control from data. We will see how problems that are central to data-driven control, such as stabilization, optimal regulation and robust controlled set invariance, can be studied with this approach. Some extensions of these results to nonlinear control systems will be also presented.

08:30 - 10:20: Lecture

10:20 - 10:40: Break

10:40 - 12:30: Lecture

14:00 - 15:50: Lecture

15:50 - 16:10: Break

16:10 - 18:00: Lecture

Relevant work:

• [BDPT20] Andrea Bisoffi, Claudio De Persis and Pietro Tesi, "Data-based stabilization of unknown bilinear systems with guaranteed basin of attraction", Systems & Control Letters 2020.

• [DPT20] Claudio De Persis and Pietro Tesi, "Formulas for Data-Driven Control: Stabilization, Optimality, and Robustness", IEEE Transactions on Automatic Control 2020.

• [GDPT20] Meichen Guo, Claudio De Persis and Pietro Tesi, "Data-Driven Stabilization of Nonlinear Polynomial Systems with Noisy Data", 2020. arXiv: https://arxiv.org/abs/2011.07833

• [BDPT20] Andrea Bisoffi, Claudio De Persis and Pietro Tesi, "Controller Design for Robust Invarinace from Noisy Data", 2020. arXiv: https://arxiv.org/abs/2007.13181

• [DPT21] Claudio De Persis and Pietro Tesi, "Low-complexity learning of Linear Quadratic Regulators from noisy data", Automatica 2021.

09:00 - 10:20: Presentations and discussions of some relevant papers

10:20 - 10:40: Break

10:40 - 12:30: Lecture by Frank Allgoewer

14:00 - 15:30: Presentations and discussions of some relevant papers

15:30 - 16:10: Break

16:10 - 17:10: Lecture by Stephen Wright

A framework for data-driven control with guarantees: Analysis, MPC and robust control -- Abstract: In this lecture, a unitary framework for data-driven control theory is presented, which does not rely on explicit model knowledge but still allows to give desirable theoretical guarantees. The framework relies on a result from behavioral systems theory, where it was shown that one data trajectory can be used to parametrize all further trajectories of an unknown linear system. Three concrete applications of the framework are presented in the form of easily implementable yet mathematically sound system analysis and controller design techniques: 1) a simple method to verify dissipativity properties of linear systems based on input-output data, 2) data-driven model predictive control schemes with guaranteed closed-loop stability for linear and nonlinear systems, and 3) a robust control-based approach for data-driven analysis and control of linear and nonlinear systems.

Connections between Optimization, Learning, and Control: Past, Present, and Future -- Abstract: Optimization, learning, and control are interconnected disciplines, enriching each other with the complementary perspectives that they provide on many problems and issues. We start by tracing the historical engagement of optimization with control and learning, leading to the present situation in which optimization methods play a central role in such areas as deep neural networks, low-rank matrix problems, and model predictive control. We present some recent results on the use of neural networks in process control, and discuss the ways in which control and learning have influenced core optimization research. Finally we speculate on some possible future arenas of engagement between these areas.

Relevant work:

• Julian Berberich, Johannes Köhler, Matthias A. Müller and Frank Allgöwer "Data-driven model predictive control with stability and robustness guarantees", IEEE Transactions on Automatic Control, 2021.

• Julian Berberich, Johannes Köhler, Matthias A. Müller and Frank Allgöwer, "Linear tracking MPC for nonlinear systems part II: the data-driven case", 2021. arXiv: https://arxiv.org/abs/2105.08567

• Julian Berberich, Carsten W. Scherer and Frank Allgöwer, "Combining prior knowledge and data for robust controller design", 2020. arXiv: https://arxiv.org/abs/2009.05253

• Anne Koch, Julian Berberich, Johannes Köhler and Frank Allgöwer, "Determining optimal input-output properties: A data-driven approach",2020. arXiv: https://arxiv.org/abs/2002.03882

• Anne Koch, Julian Berberich and Frank Allgöwer, "Provably robust verification of dissipativity properties from data", 2020. arXiv: https://arxiv.org/abs/2006.05974

• Tim Martin and Frank Allgöwer, "Dissipativity verification with guarantees for polynomial systems from noisy input-

• state data", IEEE Control Systems Letters, 2020.

• Tim Martin and Frank Allgöwer, "Data-driven inference on optimal input-output properties of polynomial systems with focus on nonlinearity measures", 2021. arXiv: https://arxiv.org/abs/2103.10306

• Anne Romer, Julian Berberich, Johannes Köhler and Frank Allgöwer, "One-shot verification of dissipativity properties from input-output data", IEEE Control Systems Letters, 2019.

09:00 - 10:20: Presentations and discussions of some relevant papers

10:20 - 10:40: Break

10:40 - 12:30: Q&A session with Marco Campi (pre-recorded lectures are available here on YouTube)

14:00 - 15:50: Lecture by Nicolas Boumal

15:50 - 16:10: Break

16:10 - 17:30: Presentations and discussions of some relevant papers

Relevant work:

• [B20] Nicolas Boumal, "An Introduction to Optimization on Smooth Manifolds" 2020. link to the book: http://sma.epfl.ch/~nboumal/book/index.html Link to the toolboxes: https://www.manopt.org/ Link to the ManOpt forum: https://groups.google.com/g/manopttoolbox

• [CG18] Marco C. Campi and Simone Garatti, "Introduction to the Scenario Approach", SIAM 2018. Link: https://epubs.siam.org/doi/book/10.1137/1.9781611975444 (free pdf available upon request from the authors)

The summer school is also as an occasion to interact and share ideas. This is why we have reserved some time slots to discuss together some of the relevant papers in the field of data-driven control. Each slot consists of a short presentation + discussion. You are welcome to join via zoom and take part in the discussions!

.

The detailed schedule is presented in the NEWS! section.