Mastering HCI Experiments with Intelligent Virtual Agents:
A Comprehensive Guide to Technical Setup, Affect Recognition, Behaviour Management, and Evaluation
This tutorial is a hands-on experience for researchers at any career stage (early-career researchers specifically welcome!) in designing, implementing and evaluating experiments in which a user is interacting with a human-controlled IVA - led by an interdisciplinary team of computer scientists and social scientists. The team will guide the tutorial participants in designing and implementing their own exemplary experiments using a combination of sophisticated open source software tools (Visual SceneMaker - StudyMaster - AffectToolbox).
Why do Experiments with Intelligent Virtual Agents?
Embodied Intelligent Virtual Agents (IVAs) are used in social training systems such as for stress regulation training [1], job interview training [2], and conflict resolution training for teachers [3]. When these systems are evaluated in studies to measure their effectiveness, the IVAs embody and take on the role of confederates with the added benefit of controlled reproducible behavior, which is difficult to achieve with human confederates. For example, IVAs can be employed as a controlled and consistent method to elicit emotions and social behavior [4], [5], [6]. To achieve realistic human-IVA interactions, interdisciplinary teams are required to design and evaluate the IVA's behavior. This tutorial aims to support such interdisciplinary teams through the use of open source tools such as AffectToolbox [7] and Visual SceneMaker [8].
How to set up the IVA System for Experiments?
Monitoring users’ affective states is of great interest in theoretical research scenarios as well as practical software design. Psychological studies often include automated analysis and recordings of user behavior. The AffectToolbox [7] is a novel software system that aims to support researchers in developing affect-sensitive studies and prototypes. The architecture encompasses a variety of pre-trained deep learning models for emotion recognition on multiple affective channels and modalities, as well as an elaborate fusion system to merge multi-modal assessments into a unified result. Emphasizing ease of use, the AffectToolbox requires no programming knowledge and offers its functionality through an accessible graphical user interface. It includes networking capabilities to broadcast its results in several formats (JSON, XML, etc.) and protocols (UDP, Kafka, etc.) to allow study and system designers easy integration into the broader infrastructure without the need to allocate resources to the implementation of the affect recognition pipeline.
To author the interactions between an IVA and a human user, the Visual SceneMaker (VSM) toolkit [8] provides a GUI with fine-grained control over customizing the actions, both verbal and non-verbal, that are to be executed in a human-IVA interaction. The customisation can be in accordance with the experimenter's goal of measuring a particular effect such as whether the IVA can induce emotions in a human [4]. Due to its modular design, the VSM is also interfaced with the AffectToolbox.
Wizard-of-Oz (WoZ) study setups offer a standard procedure to implement natural interactions between human and IVA. In a WoZ study, participants assume that they are interacting with an autonomous IVA, while in fact the steps to be performed by the IVA system are controlled by (the evaluations of) a human experimenter, the so-called Wizard. The VSM Toolkit supports this with StudyMaster, an extension that provides just such a control setup for real-time control of the IVA.
How to evaluate the IVA System?
The goal is for the IVA to be perceived and treated as a natural social interaction partner. To evaluate this, we investigate the users’ social behavior and experienced emotions during the interaction with the IVA, as in [4], [5], [6], which will be presented during the tutorial.
What awaits the participants?
This tutorial is a hands-on experience in designing HCI experiments with IVAs – led by an interdisciplinary team of computer scientists and social scientists. The aim is for the participants to implement different experimental scenarios in which a user is interacting with a human-controlled IVA. Example WoZ studies, realized in VSM and evaluated by social scientists will also be presented. The wizard can be supported by the results of real-time user behavior analysis coming in from the AffecToolbox. The setup in this tutorial is intended to be implemented by a team of two people on a single laptop using a combination of sophisticated open source software tools (Visual SceneMaker - StudyMaster - AffectToolbox). The participants will be encouraged to come up with and evaluate a WoZ-based IVA system with the support of the organizers in the workshop.
What will participants gain?
After the tutorial, participants should have a good understanding of the design of automated and/or WoZ experiments based on social signal analysis. Furthermore, the hands-on session will enable them to have implemented a working study environment on their own devices using our open source tools and provide them with the knowledge to evaluate the implemented system.
Who can attend?
This tutorial is aimed at researchers (at any stage in their career) in the field of IVAs, including computer scientists, social scientists, designers and programmers. We specifcally welcome early career researchers! Also people that have no related research done (but have gerenal interest and research ideas) are welcome! No specific in-depth knowledge is required, but a general affinity to technology is assumed.
Attendees are encouraged to bring their own Windows laptops to set up their own IVA experiment environment. Potential software downloads will be announced beforehand and will also be available on portable storage devices during the tutorial. The organizers also provide laptops with all necessary software pre-installed.
Software Tools
Optional: VisualStudio BuildTools
References.
[1] Schneeberger, T., Sauerwein, N., Anglet, M. S., & Gebhard, P. (2021, April). Stress management training using biofeedback guided by social agents. In 26th International Conference on Intelligent User Interfaces (pp. 564-574).
[2] Heimerl, A., Mertes, S., Schneeberger, T., Baur, T., Liu, A., Becker, L., ... & André, E. (2022). " GAN I hire you?"--A System for Personalized Virtual Job Interview Training. arXiv preprint arXiv:2206.03869.
[3] Chirag Bhuvaneshwara, Manuel Anglet, Bernhard Hilpert, Lara Chehayeb, Ann-Kristin Meyer, Daksitha Withanage Don, Dimitra Tsovaltzi, Patrick Gebhard, Antje Biermann, Sinah Auchtor, Nils Lauinger, Julia Knopf, Andreas Kaiser, Fabian Kersting, and Gregor Mehlmann (2023). MITHOS-Mixed Reality Interactive Teacher Training System for Conflict Situations at School. In Proceedings of the International Conference of the Learning Sciences (ICLS).
[4] Schneeberger, T., Scholtes, M., Hilpert, B., Langer, M., & Gebhard, P. (2019). Can social agents elicit shame as humans do?. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 164-170). IEEE.
[5] Schneeberger, T., Hladký, M., Thurner, A. K., Volkert, J., Heimerl, A., Baur, T., ... & Gebhard, P. (2023). The Deep Method: Towards Computational Modeling of the Social Emotion Shame driven by Theory, Introspection, and Social Signals. IEEE Transactions on Affective Computing.
[6] Hladky, M., Schneeberger, T., & Gebhard, P. (2021, September). Understanding Shame Signals: Functions of Smile and Laughter in the Context of Shame. In 2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) (pp. 01-07). IEEE.
[7] Mertes, S., Schiller, D., Dietz, M., André, E., & Lingenfelser, F. (2024). The AffectToolbox: Affect Analysis for Everyone. arXiv preprint arXiv:2402.15195.
[8] Patrick Gebhard, Gregor Mehlmann, and Michael Kipp (2012). Visual SceneMaker—a tool for authoring interactive virtual characters. Journal on Multimodal User Interfaces 6, 1-2 (2012), 3–11.