OVERVIEW

The operation of complex, connected infrastructures like the power grid, autonomous vehicles, and social media are guided by cyber systems that are informed by decisions of multiple agents. A compromised/ adversarial agent can act in a manner to impede the functioning of the cyber system. In many cases, actions of an adversary can disproportionately affect other agents and entities that rely on these infrastructures. The cyber system and the infrastructure that it influences also generate large amounts of data. This data can be used to interpret system operation, and to develop protocols and strategies that are resilient to adversaries.

Can this data be combined with domain-specific feedback to develop efficient and scalable algorithms to establish provable guarantees on performance and resilience when some decision makers are compromised?

To answer the above question, my research seeks to take a data-driven approach to interpret human-cyber interactions, and to develop resilient strategies for system-adversary interactions. My goal is to integrate and interpret feedback from human operators and data generated by cyber systems to develop viable algorithms for real-world applications that are scalable and provide provable performance guarantees. My long-term goal is to develop a learning-enabled, feedback-driven framework for the resilience of cyber and cyber-physical systems.

The results of my research have been disseminated in publications, which can be found here.

CURRENT PROJECTS

Illustration of project topic motivated by realistic traffic situations (merging, navigating a roundabout). Thrust 1 will devise methods to learn strategies by maximizing a cognizant utility that we will construct. Thrust 2 will design algorithms for multiple CPS to learn decentralized cognizant strategies, and verify that these strategies satisfy a desired objective.

NSF CRII: Cognizant Learning for Autonomous CPS

The objective of this project is to develop a cognizant learning framework for cyber-physical systems (CPS) that incorporates risk-sensitive and irrational decision making.

The necessity for such a framework is exemplified by two observations. First, CPS such as self driving cars will share an environment with other CPS and human users. Human drivers demonstrate a heightened sensitivity to changes in speed and can easily adapt to changes in the environment and road conditions, which makes it essential for a CPS to have an ability to recognize non-rational behaviors. Second, large amounts of data generated during their operation and limited access to models of their environments can make a CPS reliant on machine learning algorithms for decision making in order to meet performance requirements like reachability and safety.

Research efforts in this project will be grounded on improving the behaviors of autonomous vehicles in realistic traffic situations. Outcomes from this project will lead to the development of a research paradigm unifying control, learning, and behavioral economics.

Sponsor: National Science Foundation Computer and Information Science and Engineering Research Initiation Initiative (description)

Role: Sole PI

PREVIOUS RESEARCH

Schematic for FRESH (Feedback-based REward SHaping). A human operator is presented with trajectories of game-play from a replay buffer. At each state, the operator indicates whether the state and the action taken in that state is good or bad. This feedback is stored in a feedback buffer. A deep feedback network is used to allow the deep RL algorithm to generalize feedback signals obtained during training to unseen states and actions at test-time. The output of the feedback network is converted to a shaping reward, which is augmented to the reward provided by the environment.

Feedback-Based Reasoning in Reinforcement Learning

In reinforcement learning (RL) environments with sparse/ delayed rewards (e.g. computer games where a score is revealed only at the end of the game, and not at intermediate stages), human players perform much better than state-of-the-art RL algorithms. In these scenarios, the RL agent does not receive immediate feedback on the quality of its actions. This could result in sub-optimal actions that will not allow the agent to complete a task satisfactorily. This is termed as the credit assignment problem. Feedback signals, along with human-guided domain knowledge, can be used to modify or shape the reward to accelerate the learning process and accumulate higher rewards. Applications of these techniques include semi-autonomous driving modules, human-robot interactions, and multi-agent systems. My contributions in this area investigated providing informative feedback to facilitate credit assignment in learning-based cyber systems.

A human operator provided feedback on states and actions of the RL agent during training in FRESH (Feedback-based REward SHaping). The key insight in this work was to convert this feedback signal to a shaping reward, which was added to the reward provided by the environment. In environments with a large number of states and actions, a novel feedback neural network enabled FRESH to generalize feedback signals obtained during training to correlate with unseen states and actions at test-time.

I have also used shaping rewards described by potential functions to encode information about environment. This was added to the reward during training. Potential functions enable the incorporation of shaping rewards in a principled manner. An important guarantee offered by this method is that the total potential-based reward obtained when moving from a state and returning to the same state at a future time is zero. This property ensures that the agent will not be distracted from completing the original task specified by the reward provided by the environment.

Shaping rewards in either case could be easily specified by non-experts. Therefore, such a feedback-based learning strategy can be used in multiple domains that involve interaction of human operators with a cyber system.

Schematic representation of setting when a defender agent has to satisfy a temporal objective in the presence of an adversary. The environment of the agents is represented by the 5 x 5 grid. States in the grid are only partially observable to both agents. The defender aims to visit the green state infinitely often while always avoiding the obstacles in red. The adversary seeks to prevent the defender from accomplishing this objective. The goal is to maximize the probability of satisfaction of the temporal objective for the defender under any action taken by the adversary.

Satisfying Complex Objectives for Cyber Systems in Adversarial Environments

The growing complexity of cyber systems, and the roles that they play in supporting essential infrastructures like power systems and automobiles makes them vulnerable to attacks by intelligent and resourceful adversaries. My contributions in this domain sought to bridge the gap between the representation of properties like stability, reachability, and monitoring with requirements of safety and security. Although temporal logic frameworks enable succinct representation of objectives like safety, reachability, and priority, current research does not address the direct translation of security as a concept to these frameworks or to system models. My research established connections between game theory, formal methods, and optimization to dynamically ensure satisfaction of complex temporal objectives in the presence of an adversary.

The key insight to solve this problem was to model the interaction between the system and adversary as a two-player stochastic game. I studied this problem in diverse settings, including: i) partially observable environments; ii) satisfaction of time-critical temporal objectives, and; iii) when there were different types of adversaries, and an additional privacy requirement had to be met. In each case, I demonstrated that solving the corresponding stochastic game to equilibrium realized the desired temporal objective in a way that was resilient to actions of the adversary. I designed efficient (polynomial complexity), sound (every output was correct), and provably convergent (to an optimal solution) algorithms in each setting.

Since solution techniques for these problems involve the design of optimal protocols and strategies, they will be amenable for use in learning-based setups, where a model of the system may not be available or known.

Schematic representation of k-initial state opacity (k-ISO). Some trucks carrying money have to reach an ATM. Decoy trucks without money are employed for security purposes. The goal is to ensure that by merely observing the position of a truck, it should be impossible to determine whether the truck started from a location at which money was loaded into it. The verification of k-ISO was related to checking membership of the observation of the intruder in a set of reachable states.

A Unified Framework for Opacity

To gain illicit access to a system, a prospective attacker must be able to extract useful information, which can then be used to subvert nominal operation. Opacity is a property that captures whether an adversary can infer a secret of the system based on its observations of system behavior. The state-of-the-art studied opacity for discrete event systems, where all system parameters were discrete valued. However, in many practical systems, it is common for system variables to take continuous values. My contributions sought to broaden the scope of this security property to define opacity for discrete and continuous dynamical systems.

I bridged this gap in my Ph.D. work. I formulated k-initial state opacity (k-ISO) to characterize opacity for continuous and discrete system models. The key insight was to relate conditions to verify k-ISO in terms of sets of reachable states of the system. I proposed solutions to characterize k-ISO in multiple settings, including: i) single vs. multiple adversaries; ii) centralized vs. decentralized schemes, and; iii) CPSs modeled as switched linear systems. I additionally showed how to adapt techniques for computing under-approximations and over-approximations of these reachable sets to soundly approximate k-ISO.