Program

Abstract: Large language models have revolutionized the way humans interact with AI systems, transforming a wide range of fields and disciplines. In this talk, I share two distinct approaches to empowering human-AI interaction using LLMs. The first one explores how large language models transform computational social science, and how human-AI collaboration can reduce costs and improve the efficiency of social science research. The second part looks at social skill learning via LLMs by empowering therapists and learners with LLM-empowered feedback and deliberative practices. These two works demonstrate how human-AI collaboration via LLMs can empower individuals and foster positive change.

Bio: Diyi Yang is an assistant professor in the Computer Science Department at Stanford University, also affiliated with the Stanford NLP Group and Stanford HCI Group. Her research focuses on natural language processing and computational social science. She is a recipient of  IEEE “AI 10 to Watch” (2020), Intel Rising Star Faculty Award (2021), Microsoft Research Faculty Fellowship (2021),  NSF CAREER Award (2022), and an ONR Young Investigator Award (2023).  Her work has received multiple paper awards or nominations at top NLP and HCI conferences (e.g., ACL, EMNLP, SIGCHI, and CSCW).

Website: https://cs.stanford.edu/~diyiy/ 

Abstract: AI systems are breaking into new domains and applications, and it is pivotal to center humans in contemporary AI systems and contemplate what this means. This discussion considers three perspectives or human roles in AI as users, contributors, and researchers-in-training, to illustrate the point.

Bio: Cecilia Alm (Ph.D., UIUC) is a professor at Rochester Institute of Technology (RIT) where she is the joint program director for the MS program in Artificial Intelligence, directs the Computational Linguistics and Speech Processing Lab, and serves as an associate director for the Center for Human-aware AI. In addition, she is the PI and director for the AWARE-AI NSF Research Traineeship program and has led two iterations of an NSF Research Experiences for Undergraduates Site about human sensing and AI. She has served as an ethics co-chair for NAACL 2024, a tutorial co-chair for NAACL 2022, a diversity and inclusion co-chair for ACL 2020, a co-chair in 2022 and 2018 for the Workshop on Human-centered Computational Sensing, and as a faculty advisor for the ACL 2017 Student Research Workshop. Her research interests center on responsibly developing human-inspired AI systems, AI systems offering interaction insights, and also AI research workforce preparation.

Website: https://www.rit.edu/directory/coagla-cecilia-alm 

Abstract: It is easy to assume that Language Model Models (LLMs) will seamlessly take over applications, especially those that are largely automated. In the case of conversational voice assistants, commercial systems have been widely deployed and used over the past decade. However, are we truly on the cusp of the future we envisioned? There exists a social-technical gap between what people want to accomplish and the actual capability of technology. In this paper, we present a case study comparing two voice assistants built on Amazon Alexa: one employing a human-in-the-loop workflow, while the other utilizes LLMs to engage in conversations with users. Firstly, we suggest that at its current state, there are issues that occur with human-in-the-loop systems which do not arise with LLMs, and vice versa. Secondly, there is a set of issues that arise for both systems, leading us to believe that focusing on the interaction itself is also crucial. Merely improving the performance of the worker or model may not necessarily solve the problem. This prompts the research question: What are the overlooked contributing factors when studying user interaction with voice assistants that might not be emphasized in prior research?

Bio: Ting-Hao 'Kenneth' Huang is a computer scientist deeply committed to creating technologies that help users. His research lies at the intersection of Natural Language Processing (NLP) and Human-Computer Interaction (HCI), building interactive and intelligent systems that support people in achieving their goals in day-to-day activities. His research pushes the boundaries of what can be achieved by people and computers together, imagining a future where people can achieve their creative and social goals more easily with the aid of computers.

Website: https://crowd.ist.psu.edu/ 

Abstract: We describe "Dr. A.I.", a virtual physician assistant that uses generative AI to conduct a pre-visit patient interview and to create a draft clinical note for the physician. We document the effectiveness of Dr. A.I. by measuring the concordance of the actual diagnosis made by the doctor with the generated differential diagnosis (DDx) list. This application demonstrates the practical healthcare capabilities of a large language model to improve efficiency of doctor visits while also addressing safety concerns for the use of generative AI in the workflow of patient care.

Bio: Geoff is a board certified Internist and Emergency Physician who has practiced and taught clinical medicine for more than 40 years. Geoff founded HealthTap to deliver the highest quality, most effective, most convenient, and least expensive healthcare to everyone on their mobile devices, from anywhere. As Chief Medical Officer, he is responsible for HealthTap's doctor network and medical groups, and he continues to practice in HealthTap's Virtual Primary Care clinic.
After medical school and residency training in Internal Medicine, Geoff earned a PhD in medical computer science from Stanford, was an NIH-funded researcher in medical AI, and served on faculty at Harvard, Stanford, and UCSD medical schools. After leaving academia, he created the first consumer health website and PHR at Healtheon/WebMD, was SVP of Clinical Transformation at First Consulting Group, CMIO at San Mateo Medical Center, and EVP, Product Development & Chief Medical Officer at Epocrates. He is an elected fellow of the American College of Medical Informatics.
Geoff and his wife enjoy adventures in their Glasair (experimental aircraft), and he is an avid pilot of rigid wing hang gliders, and a SCUBA diver, cyclist, and photographer.

Website: https://www.linkedin.com/in/geoffreyrutledge/ 

Abstract: Believable proxies of human attitudes and behavior can empower applications ranging from immersive environments to social policy interventions. However, the last quarter century has seen a slow recession of human behavioral simulation as a method, in part because traditional simulations have been unable to capture the complexity and contingency of human behavior. I argue that modern artificial intelligence models allow us to re-examine this limitation. I make my case through generative agents: computational software agents that simulate believable human behavior. Generative agents enable us to populate an interactive sandbox environment inspired by The Sims, where end users can interact with a small town of twenty five agents using natural language. Our generative agent architecture empowers agents to remember, reflect, and plan — enabling them to wake up, cook breakfast, and head to work; artists paint, while authors write; they form opinions, notice each other, and initiate conversations; they remember and reflect on days past as they plan the next day. Extending this line of argument, I explore how proxying human behavior and attitudes can help us design more effective online social spaces, understand the societal disagreement underlying modern AI models, and better embed societal values into our algorithms.

Bio: Michael Bernstein is an Associate Professor of Computer Science at Stanford University, where he is a Bass University Fellow. His research focuses on human-computer interaction and social computing systems. This research has been reported in venues such as The New York Times, Wired, Science, and Nature, and Michael has been recognized with an Alfred P. Sloan Fellowship, UIST Lasting Impact Award, and the Computer History Museum's Patrick J. McGovern Tech for Humanity Prize. He holds a bachelor's degree in Symbolic Systems from Stanford University, as well as a master's degree and a Ph.D. in Computer Science from MIT.

Website: https://hci.stanford.edu/msb/ 

Abstract: Model selection is a necessary step in unsupervised machine learning. Despite numerous criteria and metrics, model selection remains subjective. A high degree of subjectivity may lead to questions about repeatability and reproducibility of various machine learning studies and doubts about the robustness of models deployed in the real world. Yet, the impact of modelers' preferences on model selection outcomes remains largely unexplored. This study uses the Hidden Markov Model as an example to investigate the subjectivity involved in model selection. We asked 33 participants and three Large Language Models (LLMs) to make model selections in three scenarios. Results revealed variability and inconsistencies in both the participants’ and the LLMs' choices, especially when different criteria and metrics disagree. Sources of subjectivity include varying opinions on the importance of different criteria and metrics, differing views on how parsimonious a model should be, and how the size of a dataset should influence model selection. The results underscore the importance of developing a more standardized way to document subjective choices made in model selection processes.

Bio: Wanyi Chen is a Ph.D. candidate in computer science at Duke University. She received a B.A. in computer science and cultural studies from the University of North Carolina at Chapel Hill in 2019. She was a software engineer at Audible before joining graduate school. Her current research centers around Human-AI interaction and Responsible AI. She investigates the subjectivities involved in machine learning model creation, aiming to improve the repeatability and reproducibility of machine learning studies. She is also interested in the ethical aspects of AI, particularly its application in scientific discovery and healthcare. In the near feature, she plans to develop practical guidelines for addressing discrepancies between machine learning outcomes and established domain knowledge. Outside of work, she enjoys creating art and has several artworks displayed at the Duke student art gallery. She is also a competitive table tennis player and has been on women’s table tennis teams at both UNC and Duke.

Website: https://www.linkedin.com/in/wanyichen/ 

Abstract: This paper outlines an approach for assessing and quantifying the risks associated with integrating Large Language Models (LLMs) in generating naval operational plans. It aims to explore the potential benefits and challenges of LLMs in this context and to suggest a methodology for a comprehensive risk assessment framework.

Website: https://www.linkedin.com/in/simon-kapiamba-a6a376172/ 

Abstract: This study revisits the concept of hybrid forums proposed by Callon et al. (2011) in order to explore how risk perception and determination can be theorized and implemented to manage risks associated with autonomous human-AI collaborative systems. Drawing on the lessons learned from risk management of mundane systems, it develops a taxonomy of risk perception. It shows that the issue of risk perception and determination for such collaborative systems is an example of a tension between delegative democracy and dialogic democracy.

Bio: Mito Akiyoshi is Professor of Sociology at Senshu University. Her recent work has focused on human decision-making in the presence of complex technology and social drivers of Japanese women’s life choices. Her recent publications include “Trust in Things: A review of social science perspectives on autonomous human-machine-team systems and systemic interdependence” (Frontiers in Physics, 10, 2022) and “Affective aspects of parenthood and their intergenerational effects on fertility”’(with Mayumi Nakamura, International Sociology, in press). In the early 1990s, she worked for NTT Data as a systems engineer for financial services clientes. She received a Ph.D. in sociology from the University of Chicago in 2004.

Website: https://sites.google.com/senshu-u.jp/mitoakiyoshi/bio 

Abstract: Current risk frameworks such as probabilistic risk analysis methodologies do not take societal safety-related benefits into account. To inform human-AI collaborative system development, this presentation highlights the need for updated risk frameworks and suggestions for relevant considerations.

Bio: Ellen J. Bass is Interim Associate Dean for Research and Professor in the Department of Information Science at Drexel University’s College of Computing and Informatics. She holds affiliate status in the College of Nursing and Health Professions and the School of Biomedical Engineering, Science and Health Systems. She is Adjunct Professor of Anesthesiology and Critical Care at the University of Pennsylvania’s School of Medicine.
Bass has 40 years of human-centered systems engineering research and design experience. She develops quantitative modeling methodologies and measures to inform design and evaluation of human-automation interaction and human-human coordination in the context of total system performance.
Bass is a fellow of the Human Factors and Ergonomics Society.   

Website: https://cci.drexel.edu/faculty/ebass/ 

Abstract: In spite of the great progress of large language models (LLMs) in recent years, there is a popular belief that their limitations need to be addressed “from outside”, by building hybrid neurosymbolic systems which add robustness, explainability, perplexity and verification done at a symbolic level. We propose shape-charged learning in the form of Meta-learning/DNN -> kNN that enables the above features by integrating LMM with explainable nearest neighbor learning (kNN) to form the object-level, having deductive reasoning-based metalevel control learning processes, performing validation and correction of predictions in a way that is more interpret-able by humans.

Bio: Boris Galitsky contributed linguistic and machine learning technologies to Silicon Valley startups as well as companies like eBay and Oracle for over 25 years. Boris’ information extraction and sentiment analysis techniques assisted a number of acquisitions, such as Xoopit by Yahoo, Uptake by Groupon, Loglogic by Tibco and Zvents by eBay. His security-related technologies of document analysis contributed to acquisition of Elastica by Semantec.  As an architect of the Intelligent Bots project at Oracle, Boris developed a discourse analysis technique user for dialogue management and published in the book "Developing Enterprise Chatbots”. He also published a two-volume monograph “AI for CRM”, based on his experience developing Oracle Digital Assistant. Boris is Apache committer to OpenNLP where he created OpenNLP. Similarity component which is a basis for a semantically-enriched search engine and chatbot development. Galitsky’s exploration and formalization of human seasoning culminated in the book “Computational Autism” broadly used by parents of children with autistic reasoning and rehabilitation personnel. Boris's focus on medical domain led to another research monograph, “AI for Health Applications and Management”. An Author of 150+ publications, 50+ patents and 6 books, Boris’s focus now is on improving content generation quality of LLMs.

Abstract: How can a machine warn its human collaborator about an unexpected risk if the machine does not possess the explicit language required to name it? This research transfers techniques from creative writing into a conversational format that could enable a machine to convey a novel, open world threat. Professional writers specialize in communicating unexpected conditions with inadequate language, using overlapping contextual and analogical inferences to adjust a reader’s situation model. This paper explores how a similar approach could be used in conversation by a machine to adapt its human collaborator’s situation model to include unexpected information. This method is necessarily bi-directional, as the process of refining unexpected meaning requires each side to check in with each other and incrementally adjust. A proposed method and example is presented, set five years hence, to envisage a new kind of capability in human-machine interaction. A near-term goal is to develop foundations for autonomous communication that can adapt across different contexts, especially when a trusted outcome is critical. A larger goal is to make visible the level of communication above explicit communication, where language is collaboratively adapted.

Bio: Dr Beth Cardier is an Assistant Professor (Health Professions) at the Eastern Virginia Medical School and adjunct Principal Research Fellow at Griffith University, with an interdisciplinary PhD in Creative Writing / Information Systems from the University of Melbourne. She analyses narrative to understand the level of information above syntax, where distributed information can be integrated, and contextual information evolves. For the past 3 years, she has also been an Advance Queensland Fellow at the Trusted Autonomous Systems Defence CRC, where she supported the development of machines that can adapt to unpredictable situations. During this time, she collaborated with industry partner CAE under a grant from the Queensland Defence Science Alliance to produce a notation for cumulative influence in a prototype augmented reality military theatre. She has collaborated with Disney Research, Earl Technologies and the Virginia Modeling Analytics and Simulation Center, which developed a prototype immersive modelling environment based on her methods.

Website: http://bethcardier.com

Abstract: We explore a novel approach to complex domain modelling by emphasising primitives based on perception. The usual approach either focuses on actors or cognition associated with tokens that convey information. In related research, we have examined using effects and/or outcomes as primitives, and influences as the generator of those outcomes via categoric functors.
That approach (influences, effects) has advantages: it leverages what is known and supports the expanded logics we use, where we want to anticipate and engineer possible futures. But it has weaknesses when placed in a dynamic human-machine system where what is perceived or assumed matters more than what is known. The work reported here builds on previous advances in type specification and reasoning to ‘move the primitives forward’ more toward situation encounter and away from situation understanding.
The goal is in the context of shared human-machine systems where: (1) reaction times are shorter than the traditional ingestion/comprehension/response loop can support; (2) situations that are too complex or dynamic for current comprehension by any means; (3) there simply is insufficient knowledge about governing situations for the comprehension model to support action; and/or, (4) the many machine/human and system/system interfaces that are incapable of conveying the needed insights; that is, the communication channels choke the information or influence flows.
While the approach is motivated by the above un-friendly conditions, we expect significant benefits. We will explore these but engineer toward a federated decision paradigm where decisions by local human, machine or synthesis are not whole-situation-aware, but that collectively ‘swarm’ locally across the larger system to be more effective, ‘wiser’ than a convention paradigm may produce.
The supposed implementation strategy will be through extending an existing ‘playbooks as code’ project whose goals are to advise on local action by modelling and gaming complex system dynamics. A sponsoring context is ‘grey zone’ competition that avoids armed conflict, but that can segue to a mixed system course of action advisory. The general context is a costly ‘blue swan’ risk in large commercial and government enterprises.
The method will focus on patterns and relationships in synthetic categories used to model type transitions within topological models of system influence. One may say this is applied intuitionistic type theory, following mechanisms generally described by synthetic differential geometry. In this context, the motivating supposition of this study is that information-carrying influence channels are best modelled in our challenging domain as perceived types rather than understood types.

Bio: Ted Goranson is presenting as a research scientist with Sirius-beta, working with the Australian Advanced Strategic Capabilities Accelerator. He has a 50 year history leading research in the US at NSA, DARPA and JFCOM Joint Experimentation Command (J9). He was an original member and later manager of the group within the US intelligence community chartered with expanded reasoning foundations. He spent a decade seconded to DARPA as office scientist to create the disciplines of enterprise modelling and integration, now widely used and anchoring a trillion dollar sector.
In the mid-noughts, he supported J9 in mapping deep theory in alternative, situated reasoning to fielded capability, primarily in sensor-centric reasoning. This led to interest in intelligent-presenting systems where no agent can be seen as intelligent, focusing on the central and gut nervous systems. Currently, he is Research Professor at Griffith University, Chief Scientist at xDNA, a Downer company, and Chief Scientist at Sirius-beta — a focused defence research collaborative.
Ted lives in Brisbane.

Website: https://sirius-beta.com/ 

Abstract: The paper presents a conceptual analysis of the notion of Trust as a agent-base cognitive function of a Trustor towards a set of Trustees so as to accomplish a mission X. Trust  emerged as a meta-process sensitive to varieties of information and risky actions. Based on a social sciences survey, an experimental campaign allowed to compare the decisions of a human Trustor facing different information Trustees and allowed to assess that in modern socio-technic environment, the automation bias fades considering the continuum of agents: mere machines, AIs, human beings. Then, following a formal approach, an incremental agent-base Trust-module is built so as to define each step of knowledge representation of a Symmetrical Trust Model.

Bio: Former director of the French national aerospace research center, he is CEO of Theorik-Lab. Graduated from Ecole Polytechnique: Promotion”X1981”. Toulouse University: PhD 1989 ‘Approximate Reasoning’; Habilitation Thesis 2005:  ‘Simple Structures for Complex Knowledge’. Kendo teacher and national instructor (martial art of Japanese fencing, 6th dan Renshi). Certified Educator for Persons with Disabilities. Lifeguard Master. Cajon percussionist. Research topics: Formal modeling, Human Reasoning, MetaCognition, Decision, Virtual Minds, Continuous Learning, Trust Modeling, Cooperative and Conflicting Agents, Self-Orientation, Rehabilitation, and Peace Engineering.

Website: https://www.researchgate.net/profile/Laurent_Chaudron 

Abstract: This present theoretical work is deliberately placed in the context capable of defining the requirements expressed by machine decision-making calculations. The informational nature of a decision requires abandoning any invariant preserving the structure but on the contrary switching into total chaos, a necessary and sufficient condition for exploiting the symmetries allowing the calculation to converge. Decision arithmetic is the best way to precisely define the nature of these symmetries.

Bio: Olivier Bartheye is an assistant professor in computer science at the French Air Force and Space Academy at Salon-de-Provence since 2019. His research concerns explainable artificial intelligence, symbolic artificial intelligence, autonomous agents, machine decision, action planning, representations of sense-making and very recently quantum computing. Formerly, he was an assistant professor in computer science at the French Ground Force Academy in Brittany, in Guer-Coëtquidan from 2001 to 2019. After his PhD about action planning, carried out from 1989 to 1994, at the Central Commission of Nuclear Energy, he started in 1995 his industrial carrier as a software engineer until 2001.

Website: https://crea.ecole-air-espace.fr/ 

Abstract: Artificial intelligence (AI) technologies have been increasingly integrated into human workflows. For example, the usage of AI-based decision aids in human decision-making processes has resulted in a new paradigm of AI-assisted decision making---that is, the AI-based decision aid provides a decision recommendation to the human decision makers, while humans make the final decision. The increasing prevalence of human-AI collaborative decision making highlights the need to understand how humans engage with the AI-based decision aids in these decision-making processes, and how to promote the effectiveness of the human-AI team in decision making. In this talk, I'll discuss a few examples illustrating that when AI is used to assist humans---both an individual decision maker or a group of decision makers---in decision making, people's engagement with the AI assistance is largely subject to their heuristics and biases, rather than careful deliberation of the respective strengths and limitations of AI and themselves. I'll then describe how to enhance AI-assisted decision making by accounting for human engagement behavior in the designs of AI-based decision aids. For example, AI recommendations can be presented to decision makers in a way that promotes their appropriate reliance on AI by leveraging or mitigating human biases, informed by the analysis of human competence in decision making. Alternatively, AI-assisted decision making can be improved by developing AI models that can anticipate and adapt to the engagement behavior of human decision makers.

Bio: Ming Yin is an Assistant Professor in the Department of Computer Science, Purdue University. Her current research interests include human-AI interaction, crowdsourcing and human computation, and computational social sciences. She completed her Ph.D. in Computer Science at Harvard University and received her bachelor’s degree from Tsinghua University. Ming was the Conference Co-Chair of AAAI HCOMP 2022. Ming is a recipient of the NSF CAREER award, and her work was recognized with multiple best paper (CHI 2022, CSCW 2022,  HCOMP 2020) and best paper honorable mention awards (CHI 2019, CHI 2016).

Website: https://mingyin.org/ 

Abstract: This work presents a hierarchical approach to explaining poisoned artificial intelligence (AI) models. The motivation comes from the use of AI models in security and safety critical applications, for instance, the use of AI models for classification of road traffic signs in self-driving cars. Training images of traffic signs can be poisoned by adversaries to encode malicious triggers that change trained AI model prediction from a correct traffic sign to another traffic sign in a presence of such a physically realizable trigger (e.g., sticky note or Instagram filter). We address the lack of AI model explainability by (a) designing utilization measurements of trained AI models and (b) explaining how training data are encoded in AI models based on those measurements at three hierarchical levels. The three levels are defined at graph node (computation unit), subgraph, and graph representations of poisoned and clean AI models from the TrojAI Challenge.

Bio: Peter Bajcsy received his Ph.D. in Electrical and Computer Engineering in 1997 from the University of Illinois at Urbana-Champaign (UIUC) and a M.S. in Electrical and Computer Engineering in 1994 from the University of Pennsylvania (UPENN).  He worked for machine vision, government contracting, and research and educational institutions before joining NIST in June 2011. At NIST, Peter has been leading efforts focusing on the application of computational science in metrology, and specifically live cell and material characterization at very large scales. Peter’s area of research is large-scale image-based analyses and syntheses using mathematical, statistical, and computational models while leveraging computer science fields, such as image processing, machine learning, artificial intelligence, computer vision, and pattern recognition. Peter has authored more than 45 papers in peer reviewed journals, co-authored 10 books or book chapters, and more than 100 conference papers.

Website: https://www.nist.gov/people/peter-bajcsy 

Talk info: Systems Engineering (SE) based design, test and evaluation (DT&E) approaches have proven crucial for the successful realization of most modern-day complex systems and system-of-systems solutions. The DT&E of engineered systems with many technologically advanced and complex components hinges on well-structured integration and life cycle evolution process models, clearly defined requirements, along test and validation of components that transcend to an overall improved, trust-worthy, and certifiable system. However, the integration of machine learning (ML), deep learning (DL), and artificially intelligent (AI) components poses new challenges as the DL algorithms are primarily data-driven with opaque decision-making constructs. In recent AI/ML literature, four primary challenges— namely Robustness, Monitoring, Alignment, and Systemic Safety— are identified for test, evaluation, and safety of DL algorithms. This presentation will explore SE foundations for addressing these challenges through the system life-cycle process modeling, functional interaction design, and SE-based test and evaluation of systems with embedded AI components. In particular, we will introduce different SE tools that can help address the four challenges described above and demonstrate an in-depth example of SE-based test and evaluation methodology applied to a reinforcement learning problem.

Bio: Ali K. Raz is a professor in the Department of Systems Engineering and Operations Research at George Mason University's College of Engineering and Computing. His research focuses on understanding the collaborative nature of autonomy and developing systems engineering methodologies for integrating autonomous systems.

Website: https://seor.gmu.edu/profiles/araz

Talk info: TBA

Bio: James Llinas is an Emeritus Professor and Founder, Director of the Collaborative Institute for Multisource Information Fusion at the State University of New York at Buffalo that addresses systemic and technological research in the broad field of sensor, data, and information fusion.  Current research topics at the Institute include Space Situational Awareness, Drone and Swarm Surveillance, Nuclear Materials Detection and Tracking, and Natural Disaster Response.

Website: https://www.buffalo.edu/cmif/people/llinas.html