IEEE CDC Workshop: Deterministic and Stochastic Hybrid Methods for Robust Learning and Optimization in Dynamical Systems
Monday 11th, December 2017, 9:00 am - 5:00 pm
University of Melbourne, Melbourne Law School, Level 6, Law Building, 185 Pelham St, Carlton VIC 3053
Jorge I. Poveda, Andrew R. Teel, Mouhacine Benosman, (Organizers and Speakers)

Kyriakos Vamvoudakis, Chris Manzie, Martin Guay  (Speakers)

In many optimization, learning, and control problems, it is difficult to have access to the precise mathematical model of the cost function or the plant to be optimized or controlled. To overcome this limitation, black-box and gray-box control, optimization, and estimation methods have been successfully developed during the last 60 years. In this workshop, we will study some recent results regarding the analysis and design of learning and estimation dynamics that, in contrast to most of the existing results in the literature, are characterized by set-valued hybrid dynamical systems, i.e., set-valued dynamical systems with states that can change continuously in time as well as instantaneously.  Set-valued hybrid dynamical systems emerge in a variety of applications such as purely continuous-time and purely discrete-time algorithms with a discontinuous right-hand side, parametric optimization, learning in games, event-triggered control optimization, global optimization under topological constraints, model-predictive control with adaptation, robust set-valued estimation, and global synchronization in networked systems, for example. Since the interaction between the discrete and continuous-time dynamics is in general not trivial, the design of hybrid learning and estimation mechanisms with stability guarantees is a challenging problem. In this workshop we will present some recent results in this area, aiming to illustrate the advantages of using hybrid dynamics in control, optimization and estimation methods. 

The goal of this workshop is to present novel results in the area of deterministic and stochastic hybrid systems applied to robust optimization and learning in dynamical systems. These include, but are not limited to, hybrid extremum seeking control, event-triggered sampled-data optimization, stochastic learning in asynchronous sampled-data games, robust set-based estimation, and event-triggered Q-learning optimal control. The workshop will present constructive approaches for the design of novel robust hybrid algorithms for black-box and gray-box optimization, as well as some engineering applications that motivate the study of hybrid optimization and learning.

Expected Outcomes
After finishing the workshop attendees will be familiar with a broad class of hybrid dynamics in the area of black-box and gray-box optimization, control, and estimation, as well as a basic understanding of how to design and analyze some of these hybrid algorithms. The merits of working with hybrid dynamics will be clearly illustrated throughout the workshop.

Workshop Outline

Morning Session:
  • 9:00 am - 10:30 am:  Introduction to hybrid dynamical systems: modeling, stability, and examples. Speaker: Andrew R. Teel. In this talk, a theoretical framework for modeling and analyzing set-valued deterministic and stochastic hybrid dynamical systems will be presented. This framework will be illustrated by means of several examples that highlight the importance of working with ``well-posed'' hybrid systems whose data satisfies some mild regularity properties. Different stability concepts, as well as Lyapunov conditions that can be used to certify the stability properties, will also be presented. The role of causality in set-valued stochastic systems will be discussed.  The talk will close by discussing some open research theoretical problems in the area of hybrid dynamical systems, needed for the development of advanced hybrid learning and optimization dynamics in complex environments.
  • 10:30 am - 11:00 am: Coffee Break. 
  • 11:00 am - 12:00 am: Distributed Nash Seeking in Asynchronous Sampled-Data Games.  Speaker: Jorge I. Poveda. Sample-data games are non-cooperative games where the players correspond to sampled-data systems. Since sample-data games describe a class of networked systems where each player has his own sampling clock, the problem of guaranteeing convergence to a Nash equilibrium in a decentralized way is, in general, a challenging task.  In this talk, we will present a stochastic hybrid approach to solve this problem by designing a class of set-valued hybrid synchronization mechanisms and a class of set-valued discrete-time learning dynamics, which combined, guarantee convergence in the mean-square sense to a neighborhood of the Nash equilibrium of the game.  Our framework allows for players to be in ``on'' and ``off'' modes, generating a rule that allows them to online decide what mode to chose at their next iteration step. By making use of recent results for input-to-state stability in set-valued stochastic systems, the effect of stubborn players whose state does not change along time is quantified. We also show the potential issues that may emerge when a causality condition (inside information in the game) is not satisfied by the updates of the players. In this case, an agent with inside information can induce a ``fake'' Nash equilibrium that generates a bigger revenue for him. This type of behavior seems to be unavoidable in set-valued stochastic learning dynamics.
Afternoon Session:
  • 1:00 pm - 1:45 pm. Hybrid Extremum Seeking Control: Analysis and Design.  Speaker: Jorge I. Poveda.  In this talk, we will present a novel class of deterministic extremum seeking controllers (ESC) with learning dynamics given by hybrid systems instead of standard Lipschitz continuous differential equations. Hybrid ESCs combine continuous and discrete-time dynamics during the seeking process, and their evolution in time is characterized by differential and difference inclusions rather than standard difference and differential equations. Examples of hybrid  ESCs include, but are not limited to, purely continuous ESC with optimizers described by set-valued mappings, ESC with arbitrarily fast and slow switching modes, ESC with weakly-jumping parameters, as well as distributed ESCs for multi-agent systems with time-varying graphs. We will show a step by step procedure to analyze and design this type of extremum seeking controllers, illustrating the role averaging and singular perturbation theory for hybrid systems in the stability proof.  We will present several examples of set-valued hybrid optimizers that are motivated by topological and performance constraints. 
  • 1:45 pm - 2:30 pm.  Novel Applications of Hybrid Extremum Seeking Control in Engineering. Speaker: Chris Manzie. In this talk, I will discuss three different applications which motivate different incarnations of extremum seeking algorithms, with either the underlying system or the controller exhibiting a hybrid structure. In the first example, it will be shown that applying extremum seeking (ES) to traffic networks requires consideration of the hybrid nature of the underlying system. Existing SPA stability results can be extended to enable ES to be applied to hybrid systems satisfying a singularly perturbed structure, as is the case in this application. In the second example drawn from the need to include feedforward control action in automotive ECUsto ensure good tracking of the optimal engine settings, multiplexed ES algorithms are proposed. It will be shown that switching between individual ES agents can lead to improved performance relative to a single agent in transient engine applications. In the final example, the need to consider averaged emissions constraints in the online optimization of fuel consumption in automotive engines necessitates the introduction of switched ES algorithms.
  • 2:30 pm - 3:00 pm: Coffee Break.
  • 3:00 pm - 3:55 pm: Robust Set-based Estimation for Learning and Optimization. Speaker: Martin Guay. Parameter identification is a key element of many robust adaptive control and optimization techniques. Traditionally, the estimation of uncertain model parameters can be used to improve the robustness of control systems. In optimization-based techniques such as model predictive control, the parameters can also be used as additional degrees of freedom to improve transient performance, their estimation providing optimality with respect to some prescribed cost functional. Beyond the precise estimation of parameters and the underlying requirement for conservative persistency of excitation conditions, the estimation of the uncertainty associated with a specific estimation mechanism can provide some additional robustness.  This presentation focusses on set-based estimation mechanisms that provide the joint estimation of the unknown parameters and an uncertainty set guaranteed to contain the true parameter values. Such mechanism, which fits in a broader class of hybrid learning approaches, can support adaptive learning in a wide range of applications such as robust optimization, robust control, and extremum-seeking control. They can also be applied to a variety of estimation approaches such as least-squares based techniques, dead-beat identifiers, and demodulation mechanisms. Some key applications of this approach are presented for specific applications such as extremum-seeking control.  Uncertainty estimation is proposed as an event-triggered mechanism that systematically reduces parametric uncertainty whenever valuable information is received. Uncertainty measures resulting from such estimation techniques provide suitable candidates for the design of real-time uncertainty minimization mechanisms. Such mechanisms are discussed in the design of self-regulating dither signal updates. 
  • 3:55pm - 4:50pm: Adaptive Hybrid Q-learning with Stability Guarantees. Speaker: Kyriakos Vamvoudakis. While the event and self-triggered literature continue to flourish, two fundamental issues remain overshadowed, the co-design of both the feedback law and the triggering scheme; and the performance guarantees by design. So far, only a few approaches have tried to simultaneously address these points. However, up to now methods rely on an offline computation of the Riccati or Hamilton-Jacobi-Bellman equation and depend on the full knowledge of the system dynamics, being vulnerable to exhaustive modeling and malicious attacks. Q-learning is a model-free reinforcement learning technique primarily developed for discrete-time systems where an optimal action is selected based on previous state and actions observations. This talk shall present a hybrid Q-learning scheme that is independent of the system matrices. Specifically, we will show the derivation of an actor/critic structure that adaptively tunes and approximates the optimal event-triggered controller and the Q-function respectively, to solve the problem online and forward in time. Because of the discrete and continuous time dynamics, we model the system as a hybrid model and prove that the equilibrium point of the flow and the jump dynamics is asymptotically stable. Finally, simulation results will show the efficacy of the proposed approach.
Expected Attendance

The workshop is intended to be a brief course on recent analysis and design tools for robust algorithms for model-free optimization and learning using deterministic and stochastic hybrid dynamics. It targets a broad audience in academia and industry, including graduate students looking for an introduction to a new and active area of research; control practitioners interested in novel design techniques and applications; and researchers in dynamical systems, optimization, learning in games, and control. The workshop audience is not expected to have any advanced background in hybrid systems or optimization. A basic knowledge of linear/nonlinear system theory and probability theory is useful.


Andrew R. Teel received the A.B. degree in engineering sciences from Dartmouth College, Hanover, NH, in 1987, and the M.S. and Ph.D. degrees in Electrical Engineering from the University of California, Berkeley, CA, USA, in 1989 and 1992, respectively. After receiving the Ph.D. degree, he was a  Postdoctoral Fellow at the Ecole des Mines de Paris in Fontainebleau, France. In 1992, he joined the Faculty of the Department of Electrical Engineering at the University of Minnesota, where he was an Assistant Professor until 1997. Subsequently, he joined the Faculty of the Department of Electrical and Computer Engineering at the University of California, Santa Barbara, CA, USA, where he is currently a Distinguished Professor and the Director of the Center for Control, Dynamical Systems, and Computation. His research interests include nonlinear and hybrid dynamical systems, with a focus on stability analysis and control design. Dr. Teel received the NSF Research Initiation and CAREER Awards, the 1998 IEEE Leon K. Kirchmayer Prize Paper Award, the 1998 George S. Axelby Outstanding Paper Award, and the first SIAM Control and Systems Theory Prize in 1998. He received the 1999 Donald P. Eckman Award and the 2001 O. Hugo Schuck Best Paper Award, both given by the American Automatic Control Council, and also received the 2010 IEEE Control Systems Magazine Outstanding Paper Award. In 2016, he received the Certificate of Excellent Achievements from the IFAC Technical Committee on Nonlinear Control Systems. He is a Fellow of the IFAC, and he is currently serving as Editor-in-Chief of Automatica.

MouhacinBenosman is a Senior Research Scientist at Mitsubishi Electric Research Labs (MERL) in Cambdrige, USA. Before joining MERL he worked at Reims University, France, Strathclyde University, Scotland, and National University of Singapore. HNational University of Singapore. His research interests include modeling and control of flexible systems, nonlinear robust and fault tolerant control, vibration suppression in industrial systems, multi-agent distributed control with applications to smart-grid, and learning and adaptive control for nonlinear systems. Mouhacine has published a monograph about learning-based adaptive control, more than 70 peer-reviewed journal articles and conference papers, andmore than 20 patents in the field of mechatronics systems control. He is a senior member of the IEEE and an Associate Editor of the Control System Society Conference Editorial Board.

Chris Manzie  is currently a full Professor and Head of Department of Electrical and Electronic Engineering at the University of Melbourne, and also the Director of the Melbourne Information, Decision and Autonomous Systems (MIDAS) Laboratory, which includes academics from multiple faculties including Engineering, Science and Law. Over the period 2003-2016, he was an academic in the Department of Mechanical Engineering, with responsibilities including Assistant Dean with the portfolio of Research Training (2011-2017), and Mechatronics Program Director (2009-2016). Professor Manzie was also a Visiting Scholar Science and Law. Professor Manzie was also a Visiting Scholar with the University of California, San Diego in 2007 and a Visiteur Scientifique at IFP Energies Nouvelles, Rueil Malmaison in 2012. His research interests are in model-based and model-free control and optimisation, with applications in a range of areas including systems related to energy, transportation, and mechatronics, and he has published over 170 refereed journal and conference articles. He is currently supported by industry collaborations with Toyota Motor Corporation, ANCA Motion, Mitsubishi Heavy Industries and the Defence Science and Technology Group, and has been an investigator on over $15M of grant funding and research contracts. He is currently an Associate Editor for Elsevier Control Engineering Practice; IEEE/ASME Transactions on Mechatronics; IEEE Transactions on Control Systems Technology; and Elsevier Mechatronics.

Kyriakos G. Vamvoudakis
was born in Athens, Greece. He received the Diploma (a 5 year degree, equivalent to a Master of Science) in Electronic and Computer Engineering from the Technical University of Crete, Greece in 2006 with highest honors. After moving to the United States of America, he studied at The University of Texas and he received his M.S. and Ph.D. in Electrical Engineering in 2008 and 2011 respectively.  During the period from 2012 to 2016 he was a project research scientist at the Center for Control, Dynamical Systems and Computation at the University of California, Santa Barbara. He is now an assist
ant professor at the Kevin T. Crofton Department of Aerospace and Ocean Engineering at Virginia Tech. His research interests have focused on game-theoretic control, network security, smart grid and multi-agent optimization.  Dr. Vamvoudakis is the recipient of several international awards including the 2016 International Neural Network Society Young Investigator (INNS) Award, the Best Paper Award for Autonomous/Unmanned Vehicles at the 27th Army Science Conference in 2010, the Best Presentation Award at the World Congress of Computational Intelligence in 2010, and the Best Researcher Award from the Automation and Robotics Research Institute in 2011. He is coauthor of one patent, more than 90 technical publications, and two books. He currently is an Associate Editor of the Journal of Optimization Theory and Applications, an Associate Editor of Control Theory and Technology, a registered Electrical/Computer engineer (PE) and a member of the Technical Chamber of Greece. He is a Senior Member of IEEE.

Martin Guay is a Professor in the Department of Chemical Engineering at Queen's University in Kingston, Ontario, Canada. He received his PhD from Queen's University in 1996. From 1995 to 1997, he was a research scientist with E.I. Dupont. Dr. Guay is Senior Editor for the IEEE CSS Letters and deputy Editor-in-Chief of the Journal of Process Control. He is also an associate editor for Automatica, the Canadian Journal of Chemical Engineering and Nonlinear Analysis & Hybrid Systems. He was the recipient of the Syncrude Innovation award from the Canadian Society of Chemical Engineers. He also received the Premier Research Excellence award. His research interests are in the area of nonlinear control systems including extremum-seeking control, nonlinear model predictive control, adaptive estimation and control, and geometric control.  Some notable industrial collaborators, past and present , include United Technologies Research Centre, Mitsubishi Electric Research Laboratory, Johnson Controls and Praxair.


Jorge I. Poveda received the B.S. degrees in Electronics Engineering and Mechanical Engineering in 2012, and the M.S. degree (Magna Cum Laude) in Electrical Engineering in 2013, all from University of Los Andes, Bogota, Colombia, and the M.S. degree in Electrical and Computer Engineering from the University of California, Santa Barbara, USA, in 2015, where he also received the 2013 Center for Control, Dynamical Systems, and Computation Outstanding Scholar Fellowship. He was a Research Intern with the Mitsubishi Electric Research Laboratories in Cambridge, MA, during the summers of 2016 and 2017. He is currently working towards the Ph.D. degree in the area of control and dynamical systems in the Department of Electrical and Computer Engineering at the University of California, Santa Barbara.