Neighbor-Environment Observer: An Intelligent Agent for Immersive Working Companionship
Neighbor-Environment Observer: An Intelligent Agent for Immersive Working Companionship
Zhe Sun, Qixuan Liang, Meng Wang, Zhenliang Zhang*
The 36th Annual ACM Symposium on User Interface Software and Technology (UIST), 2023
Human-computer symbiosis is a crucial direction for the development of artificial intelligence. As intelligent systems become increasingly prevalent in our work and personal lives, it is important to develop strategies to support users across physical and virtual environments. While technological advances in personal digital devices, such as personal computers and virtual reality devices, can provide immersive experiences, they can also disrupt users' awareness of their surroundings and enhance the frustration caused by disturbances. In this paper, we propose a joint observation strategy for artificial agents to support users across virtual and physical environments. We introduce a prototype system, neighbor-environment observer (NEO), that utilizes non-invasive sensors to assist users in dealing with disruptions to their immersive experience. System experiments evaluate NEO from different perspectives and demonstrate the effectiveness of the joint observation strategy. A user study is conducted to evaluate its usability. The results show that NEO could lessen users' workload with the learned user preference. We suggest that the proposed strategy can be applied to various smart home scenarios.
Introduction of NEO agent
The NEO system jointly observes the physical and virtual environments and takes action in the two environments simultaneously. It is designed to deal with disruptions for people when they are immersed in a virtual working environment.
Why does NEO provide better experience?
External stimuli and disruptions may break the immersion. We propose an intelligent agent NEO that could perceive user's demands, help them to handle the disruptions, and preserve their immersion in virtual environments.
System structure
Information and decision transfer process in our framework. Joint observation information is transmitted to a data manager via WiFi. The data manager activates the decision manager, from which the decision is transmitted to the two types of embodiments.
Hardware
Hardware implementation of NEO. The mobile robots serve as movable physical embodiments of NEO. Together with trays and the camera, they construct the desktop robot system that provides object transportation. The mobile robot connected with a speaker forms the ground robot system that could help the users to receive visitors.
Joint observation for virtual-real spaces
The And-Or Graph of the joint observation. The perception of the agent is categorized into two sets: user state and environment state. The user state node consists of four components, representing occupation methods, which are further divided based on method classes and specific methods. The environment state node consists of two components: the virtual part and the physical part. Colors of the Terminal-nodes denote whether the nodes belong to the physical environment or not.
Comparison between different settings
NASA-TLX results. (a) The overall workload of the whole task (including both discussing with the professor and handling disruptions ). (b) The workload of handling the disruptions. Note that the points of performance in this figure are aligned so that all questions share the same trend: "the lower the point is, the better".
Cite
@inproceedings{sun2023neighbor,
title={Neighbor-Environment Observer: An Intelligent Agent for Immersive Working Companionship},
author={Sun, Zhe and Liang, Qixuan and Wang, Meng and Zhang, Zhenliang},
booktitle={Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology},
pages={1--14},
year={2023}
}