Embedded Files
Shuo Yang1 Hongrui Zheng1 Cristian-Ioan Vasile2 George J. Pappas1 Rahul Mangharam1
University of Pennsylvania1 Lehigh University2
Abstract
Abstract
We study how to synthesize a robust and safe policy for autonomous systems under signal temporal logic (STL) tasks in adversarial settings against unknown dynamic agents. To ensure the worst-case STL satisfaction, we propose STLGame, a framework that models the multi-agent system as a two-player zero-sum game, where the ego agents try to maximize the STL satisfaction and other agents minimize it. STLGame aims to find the Nash equilibrium policy profile, which is the best case in terms of robustness against unseen opponent policies, by using the fictitious self-play (FSP) framework. FSP iteratively converges to a Nash profile, even in games set in continuous state-action spaces. We propose a gradient-based method with differentiable STL formulas, which is crucial in continuous settings to approximate the best responses at each iteration of FSP. We show this key aspect experimentally by comparing to RL-based methods to find the best response. Experiments on two standard dynamical system benchmarks, Ackermann steering vehicles and autonomous drones, demonstrate that our converged policy is almost unexploitable and robust to various unseen opponents' policies.
- STLGame Converges to Nash Policy Profiles
Car: Agent Trajectories During FSP Iterations
Car: Agent Trajectories During FSP Iterations
(Nash at Iteration 50)
Drone: Agent Trajectories During FSP Iterations
Drone: Agent Trajectories During FSP Iterations
(Nash at Iteration 100)
2. Control Policies Synthesized by STLGame are Robust against Unseen Opponents
2. Control Policies Synthesized by STLGame are Robust against Unseen Opponents
Car: Nash Agent Trajectories v.s. Unseen Opponents
Car: Nash Agent Trajectories v.s. Unseen Opponents
(STL formulas satisfied in all cases)
Drone: Nash Agent Trajectories v.s. Unseen Opponents
Drone: Nash Agent Trajectories v.s. Unseen Opponents
(STL formulas satisfied in all cases)
3. STLGame More Efficient Compared to RL in Continuous Environments
3. STLGame More Efficient Compared to RL in Continuous Environments
Using gradient descent with differentiable STL robustness produces much better exploitability results (better convergence to Nash).
Using gradient descent with differentiable STL robustness produces much better exploitability results (better convergence to Nash).
Using PPO as best response synthesizer does not produce enough improvement to agent control policies in the given compute budget.
Using PPO as best response synthesizer does not produce enough improvement to agent control policies in the given compute budget.
Page updated
Google Sites
Report abuse