Sponsor
National Science Foundation (NSF)
Abstract
In many distributed systems, a team of autonomous agents must collaborate in a complex environment, process massive amounts of streaming data, and simultaneously make optimal decisions. Traditional decision-making techniques can hardly tackle such a scenario, and reinforcement learning (RL) has been recently shown to be a promising decision-making technique for large-scale distributed systems. However, previous distributed RL models have failed to account for sensing and observing capabilities of agents, and thus rely on global information, which is not readily available in distributed environments. To fill this gap, this project aims to build a revolutionary, fully distributed RL system for large-scale networked systems without using global information. Toward this end, the project develops a novel theoretical framework, computational models, and scientific software tools needed to design, analyze, and test fully distributed RL algorithms. The algorithms will be further designed to be robust against dynamic environments and resilient to adversarial attacks, which will enable teams of multiple autonomous agents to reliably achieve their goals. The research will greatly impact real-world application areas where distributed machine learning algorithms and decision-making methods are needed. Typical examples include motion planning of teams of mobile robots, and coordination of networked smart devices in an IoT environment. The project promotes education and outreach activities, including broadening participation of female students in the field of machine learning, creating new courses, and designing research projects for K-12 students and undergraduates. The publications and software tools will be shared with the community to foster further research on distributed RL.
The central goal of this project is to establish theoretical foundations for fully distributed RL algorithm design, analysis, and applications over large-scale networks. The key technical challenges include bridging the gap between the global and local observability settings and achieving resiliency in the presence of dynamic and untrustworthy communications. To achieve the technical objective and tackle technical challenges, the project investigates three main thrusts. The first thrust establishes the fundamental novel theory for the design of fully distributed RL by approximating global information via distributed estimation. The second thrust develops robust distributed RL algorithms against time-varying communication and sensing capabilities, communication delays, and asynchronous updating. The third thrust designs distributed RL algorithms that are resilient to adversaries and malicious attacks capable of introducing untrustworthy information into the communication network, by first designing communication-efficient RL algorithms in which each agent can transmit only low-dimensional states, and then designing resilient information fusion/aggregation approaches for small- and even single-dimensional cases. The project provides a suite of novel distributed RL algorithms which can be used in any applied area where fully distributed decision making and learning with streaming data and in adversarial environments are needed. Concurrently with the three main thrusts, the project also designs, develops, and maintains a software framework for empirically validating and studying distributed RL algorithms that the entire distributed RL community can use.
Publications
Byzantine-Resilient Decentralized Multi-Armed Bandits [ arXiv ]
J. Zhu, A. Koppel, A. Velasquez, and J. Liu. Transactions on Machine Learning Research. accepted.
Resilient Multi-agent Reinforcement Learning with Function Approximation [ link ]
L. Ye, M. Figura, Y. Lin, M. Pal, P. Das, J. Liu, and V. Gupta. IEEE Transactions on Automatic Control. accepted.
Split-Spectrum Based Distributed State Estimation for Linear Systems [ link ]
L. Wang, J. Liu, B. D. O. Anderson, and A. S. Morse. Automatica. accepted.
Finite-Time Error Bounds for Distributed Linear Stochastic Approximation [ link ] [ arXiv ]
Y. Lin, V. Gupta, and J. Liu. Automatica. accepted.
Reaching a Consensus with Limited Information [ arXiv ]
J. Zhu, Y. Lin, J. Liu, and A. S. Morse. Systems & Control Letters, Art Krener Special Issue. accepted.
Distributed Multi-Armed Bandits [ link ]
J. Zhu and J. Liu. IEEE Transactions on Automatic Control, Special Issue on Learning for Control. accepted.
Heterogeneous Distributed Subgradient
Y. Lin, M. Gamarra, and J. Liu. American Control Conference, 2024.
Sampling-based Safe Reinforcement Learning for Nonlinear Dynamical Systems [ arXiv ]
W. A. Suttle, V. K. Sharma, K. C. Kosaraju, S. Sivaranjani, J. Liu, V. Gupta, and B. M. Sadler. 27th International Conference on Artificial Intelligence and Statistics, 2024.
A Resilient Distributed Algorithm for Solving Linear Equations [ arXiv ]
J. Zhu, A. Velasquez, and J. Liu. 62nd IEEE Conference on Decision and Control, 2023.
Split-Spectrum based Distributed Estimator for a Continuous-Time Linear System on a Time-Varying Graph
L. Wang, J. Liu, A. S. Morse, and B. D. O. Anderson. 62nd IEEE Conference on Decision and Control, 2023.
Resilient Distributed Optimization [ arXiv ]
J. Zhu, Y. Lin, A. Velasquez, and J. Liu. American Control Conference, 2023.
Information-Directed Policy Search in Sparse-Reward Settings via the Occupancy Information Ratio
W. A. Suttle, A. Koppel, and J. Liu. 57th Annual Conference on Information Sciences and Systems, 2023.
Subgradient-Push Is of the Optimal Convergence Rate
Y. Lin and J. Liu. 61st IEEE Conference on Decision and Control, 2022.
Reaching a Consensus with Limited Information
J. Zhu, Y. Lin, J. Liu, and A. S. Morse. 61st IEEE Conference on Decision and Control, 2022.