Description and History

Human-Autonomy Teaming (HAT) describes situations where people cooperate with artificially intelligent autonomous agents to perform some function. In studying adaptive systems, we intend to look at capabilities that are developed using machine learning where adaptation can occur either in the lab or in “the wild” or both. This workshop will expand on the discussions begun during two previous IJCAI workshops:

· IJCAI 2017 Workshop on Impedance Matching in Cognitive Partnerships (https://sites.google.com/view/ijcai-17cognitivepartnerships/home)

· Autonomy in Teams: Joint Workshop on Sharing Autonomy in Human-Robot Interaction as part of the federated workshop program associated with IJCAI 2018 (https://sites.google.com/view/autonomy-in-teams/home).

In the 2017 workshop, we combined invited talks by Professor Stuart Russell of Berkeley, Dr. Peter Friedland of the US Air Force Office of Scientific Research, Professor Tim Miller of the University of Melbourne, and Dr. Mike Cox of the Wright State Research Institute with a series of presentations supported by papers, and a lively panel discussion. The first workshop explored cognitive partnerships among heterogeneous autonomous team members, whether they be human or artificial. People often struggle to work through the impedance mismatches caused by varying backgrounds, professional fields, and goals. This workshop targeted areas of impedance mismatch between humans and autonomous AI.

In the second workshop on HAT, we continued to focus on partnerships between humans and artificial intelligence. We held a full day workshop that combined invited talks, papers, and posters. In the morning, we had discussions on the definition and critical capabilities of the partnerships. In the afternoon we focused on measurement of team performance. Among the invited speakers were Roger Woltjer the Deputy Research Director at the Swedish Defence Research Agency.

In this third IJCAI workshop on Human-Autonomy Teaming, we intend to delve further into the problem of evaluation. Software engineering tools and techniques for evaluating the quality of traditionally developed software capabilities are well studied and many approaches are mature. System components developed through learning from data are less easily tested for mission critical systems. Reserving a portion of the data sample for testing provides a statistical measure of quality relative to the overall sample of the particular type of data for a particular component of what may be one in a series of learned models in a complex system. The quality of the overall system as it relates to a complex mission are less well known. Add to the problem the desire to allow models to adapt while deployed in lifelong learning approaches and complexity is further increased. Humans bring another element of quality assessment to the problem. Their interaction with the adaptive artificially intelligent components must also be evaluated in some way. Humans will adapt to the changing behaviors of the autonomy in ways that are currently unpredictable.