Synopsis

Agenda

Speakers

Below is an overview of the speakers, their backgrounds, and a summary of their talks.

 Session: Data-driven Dynamics and Control using the Koopman operator

Abstract: In recent years, the Koopman theory has emerged as a powerful tool for the data-driven analysis and control of complex systems. The talk will discuss recent advances in the theory and application of the operator theoretic framework for data-driven modeling, control, and safety of complex systems. We will also discuss the challenges in advancing the Koopman operator-based design methods and opportunities for the digital engineering of off-road autonomous vehicles.

 Session: Operator methods in complex mobility, manipulation and sensing 

Abstract: The talk will introduce a broad array of applications of operator methods to sensing and parameter estimation in unstructured environments, the construction of data-driven but physics-informed operator models and the use of such models for control, mobility and manipulation. Examples will be drawn from underwater sensing and estimation, control of underactuated swimming robots, stabilization of off-road robots, manipulation of particle swarms in fluids and conclude with some future possibilities for off-road ground vehicles.

 Session: Data-driven Modeling and Control for Off-Road Autonomy Using the Koopman Operator

Abstract: In recent years, Uncrewed Ground Vehicles (UGVs) have significantly advanced, with deployments growing in numbers and diversity across applications such as surveillance, exploration, payload transport, and logistics in increasingly challenging real-world settings. This expansion of operational regimes poses challenges in modeling and control due to non-linear wheel-terrain interactions and significant environmental perturbations. Traditional control paradigms, which rely on precise analytical models, often fall short as these models are difficult to derive and may not fully capture real-world complexities. While analytical and learning-based techniques have been applied, they either use lower fidelity models or sacrifice explainability with end-to-end neural network approaches. This talk presents a state-of-the-art data-driven modeling and control technique for UGVs using the adaptive Multi-Model Parameterized Koopman (MMPK) framework. Building upon the Koopman Extended Dynamic Mode Decomposition (KEDMD) algorithm, MMPK offers a flexible model and control adaptation in the presence of environmental uncertainty. Unlike traditional KEDMD-based approaches, MMPK captures multiple models spanning the operational space, factoring in real-time terrain perturbations to ensure adaptability, robustness, and generalizability. Additionally, we present an outer control loop incorporating a novel model-driven motion planner and path-tracking controller. This unified framework of offline modeling and an online planning-control loop has been extensively tested through simulations and hardware experiments, demonstrating superior path-tracking capabilities and the effectiveness of its local planning strategy for safe on-road and off-road traversal. Join us as we explore how the adaptive MMPK framework represents a significant advancement in UGV technology, providing robust and adaptable solutions for complex, real-world operational challenges. 

 Session: Modeling for Virtual and Agile Physical Prototyping of Full-scale High-speed Off-road Vehicles

Abstract: Despite the advances in high-fidelity simulation, full-scale prototyping is still necessary for validating and calibrating digital twins and as a basis for data collection supplying machine learning algorithms. This presentation covers the efforts of the Deep Orange vehicle prototyping program in creating a tracked high-speed off-road research vehicle platform for autonomy deployment and data collection. We will discuss the scope of modeling techniques utilized for the different phases of concept exploration, design, prototyping, and control of this 3-ton prototype skid steering vehicle with varying real-time capability and fidelity demands. Emphasis will be on the continued model refinement through data collection and calibration and how this iterative process of simulation and validation has led to an improved testbed and control framework. The presentation will highlight the instrumental role of data in developing high-fidelity models and how results from these models have been utilized by other research efforts to adapt existing control frameworks. Finally, the talk will introduce the comprehensive publicly available dataset emerging from the above efforts that will be shared with the research community.

 Session: Reinforcement Learning-Model Predictive Control (RL-MPC) for the Controls of Off-road Autonomous Vehicles 

Abstract: The talk will introduce a deep reinforcement learning enhanced model predictive controller for off-road autonomous driving in unknown environments. This controller adopts a model predictive controller as the backbone for the control and integrates deep reinforcement learning into the control process to address the challenge of modelling unknowns in off-road environments. The framework was firstly verified in Project Chrono simulation and then preliminarily tested on a full-scale off-road autonomous vehicle. The results and findings will be presented in detail. 

 Session: Autonomous Design in Piecewise Stationary Environments

Abstract: Key to autonomous design is the ability to control a system in non-stationary environments. While ready-made models may not be available for these environments, it is imperative to build models on the fly in a way where models can be compared, and learned controllers in environments can be repurposed to warm-start reinforcement learning algorithms in similar environments. In this talk, we will present a framework for autonomous design in piecewise-stationary environments, offer simulation results, and identify key theoretical analysis required to understand the computational considerations for said design. In addition, we will present two sets of theoretical results that define key elements of this design. Namely, they are characterization of computational complexity of sparse model learning in reproducing kernel Hilbert spaces and an analysis of change detection in transition kernels of Markov decision processes.

 Session: Perception and Control in the Wild for off-road Autonomous Vehicles

Abstract: The talk focuses on perception and planning algorithms for autonomous vehicles in off-road situations. A particular emphasis is on why off-road vehicles are different than on-road vehicles and how can we solve autonomy in the off-road domain. A major portion of the talk will be on applications of the above algorithms to real vehicles and the lessons that we have learned i.e. what worked and what didn’t and how we should go about building such systems.

Session: A Game-Theoretic and Learning-Based Approach for Improving the Collaborative Interactions of Mixed Human-Autonomous Vehicle Systems

Abstract: Human-autonomy collaboration is ever-growing, as engineered systems can provide support to perform tasks more efficiently and effectively than a human could do on their own. These tasks span various domains, from repetitive material handling operations in manufacturing to semi-autonomous vehicle navigation tasks, enabling humans to focus on secondary tasks simultaneously. A consequence of these interactions is that the autonomous system experiences bidirectional coupling, or interdependence, with the human during these collaborations. It is further important to consider that humans are capable of intelligent, intentional behaviors that adapt over time. Current approaches fail to adjust autonomous behavior to accommodate evolving human dynamics, thereby limiting overall performance. This talk explores the application of game-theoretic level-k thinking, which models strategies used by agents with bounded rationality, to inform autonomous system control in collaborative scenarios. By understanding and predicting human behavior through this framework, autonomous systems can make more informed decisions. Further, integrating iterative learning control into this framework for repeated tasks enables the autonomy to adapt to nuances between individuals and to changes in the individual due to factors such as learning, fatigue, etc., thereby optimizing team performance dynamically. 

 Session: Digital Engineering for Autonomous Off-Road Vehicles in the Automotive Research Center

Abstract: The Automotive Research Center (ARC) is the U.S. Army Center of Excellence in the area of modeling and simulation (M&S) of ground vehicles led by the University of Michigan. The future of ground systems in both commercial and military contexts is going through revolutionary change. The ARC is leading the way in creating this future. This presentation will discuss the technological gaps addressed in ARC research and recent advances that spearhead the most advanced digital engineering knowledge, technology and innovations to push forward the discovery and creation of ground vehicles. These efforts have seen a remarkable success for transformative new technologies over the past five years in the ARC. Because of the need to use digital engineering to develop and test autonomous systems, the ARC has placed a particular emphasis on fundamental discoveries and the creation of cutting-edge capabilities in autonomy.

The ARC is a collaborative team of researchers from fourteen universities and the U.S. Army CCDC Ground Vehicles Systems Center (GVSC). The ARC team has experts in multiple disciplines who converge on bridging fundamental knowledge gaps and addressing scientific challenges enhancing commercial and military vehicles. This research encompasses technological elements, human factors and social behavior that work in harmony in successful system-level designs. Such convergence of disciplines and integrative thinking are unique strengths of the ARC that will be discussed in this presentation. 

The ARC is the key hub for CCDC-GVSC, where new ideas are generated and translated into key technologies in autonomy of ground systems, including vehicle dynamics, control, autonomous behavior, human-autonomy teaming, high performance structures and materials, intelligent power systems, and fleet operations and vehicle system of systems integration. ARC research includes efforts to create synthetic environments with high-fidelity synthetic sensors and virtual vehicle prototypes (e.g., virtual proving grounds). These environments integrate virtual reality tools for human-autonomy teaming, which are implemented in simulation engines based on ARC-developed algorithms embedded in gaming platforms, such as virtual prototyping ADAS tools connected with Unreal Engine 4. Advanced running-gear-terrain interaction models (such as those created for the next generation NATO mobility model) are designed to augment synthetic environments. This includes enhanced M&S for off‐road mobility on deformable granular terrains. In addition, trust-based mobility for autonomous vehicles and optimal task allocation in human-autonomy teams are combined with M&S tools for terrain identification, vehicle thermal management, and models of other vehicle functions. Such efforts will be discussed in this presentation.

 Session: Pushing the Mobility Limits of Autonomous Off-Road Vehicles

Abstract: Safely exploiting the mobility limits of a vehicle on unknown deformable terrains is an essential capability for autonomous off-road driving in time and safety critical applications such as extreme maneuvers in military operations. However, the traditional approach of treating planning and control problems separately leads to overly conservative solutions and limits such exploitation. To address this gap, we propose a paradigm shift from separate to combined treatment of planning and control problems, which allows for solving both problems with a consistent understanding of the capabilities and limits of the vehicle for improving performance and safety. This is done in a model predictive control framework with special consideration for reliable and real-time performance throughout the formulation and solution stages. Augmenting this framework with real-time estimation of critical unknown parameters provides an adaptive solution that further increases its performance and reliability. The results show that the new framework leads to significant increases in safety, performance, and reliability. Experimental validation on a military vehicle on deformable terrain confirms the real-world implementability and benefits of the framework. Thus, this framework lays the foundation for future studies to push the mobility limits of autonomous driving to meet the complex autonomous navigation challenges in real-world applications. 

 Session: Multimodal Model-Based Reinforcement Learning

Abstract: Model-based reinforcement learning (MBRL) techniques have recently yielded promising results for real-world driving using high-dimensional observations. MBRL agents solve long-horizon tasks by building a world model and planning actions by latent imagination. This approach involves explicitly learning a model of the system dynamics and using it to learn the optimal policy for continuous control over multiple timesteps. As a result, agents may converge to sub-optimal policies if the world model is inaccurate. This talk describes Lucid Dreamer, an end-to-end multimodal MBRL agent that leverages egocentric LiDAR and RGB camera observations through self-supervised sensor fusion. The zero-shot performance of MBRL agents is empirically evaluated on a 1:10 scale rover in simulation for unseen conditions and in a real-world environment to demonstrate sim-to-real transfer. Although only trained against five static obstacles in simulation, Lucid Dreamer safely avoided collisions with a dynamic rule-based agent in a zero-shot manner. This talk illustrates that multimodal perception improves robustness of the world model without requiring additional training data. 

Organizers