How do decision-making agents perform when the world changes?

Why is this tutorial being held at CPS-IoT week? 

AI-based approaches are increasingly being used for decision-making (both planning and control) in cyber-physical systems (CPS). However, CPS increasingly operate in environments that evolve over time due to drift in physical parameters, changing operating conditions, sensor degradation, and exogenous shocks. Yet most planning, control, and reinforcement learning methods rely on the assumption of stationary dynamics—an assumption that often breaks down in real-world deployments. This tutorial introduces a principled framework for decision-making under non-stationarity, equipping participants with the theoretical foundations, algorithmic tools, and practical experience needed to design adaptive decision-making systems that remain robust as the environment changes.

What will you learn? 

By the end of the tutorial, participants will be able to:

• Understand why stationarity assumptions fail in real-world CPS and IoT environments.

• Model evolving system dynamics using non-stationary MDPs, including abrupt changes, continuous drift, and periodic variation.

• Distinguish between known, detected, and hidden forms of non-stationarity.

• Evaluate why classical RL and online search-based methods struggle under non-stationarity and how adaptive planning can address these challenges.

• Design and experiment with non-stationary environments using NS-Gym, including CartPole, FrozenLake, CliffWalking, and Bridge environments.

• Apply adaptive decision-making techniques to CPS domains such as transportation, emergency response, and autonomous systems.