Symmetrical Reality
On the Emergence of Symmetrical Reality
Zhenliang Zhang, Zeyu Zhang, Ziyuan Jiao, Yao Su, Hangxin Liu, Wei Wang, Song-Chun Zhu
Beijing Institute for General Artificial Intelligence & Peking University
Paper link: https://arxiv.org/pdf/2401.15132.pdf
Abstract
Artificial intelligence (AI) has revolutionized human cognitive abilities and facilitated the development of new AI entities capable of interacting with humans in both physical and virtual environments. Despite the existence of virtual reality, mixed reality, and augmented reality for several years, integrating these technical fields remains a formidable challenge due to their disparate application directions. The advent of AI agents, capable of autonomous perception and action, further compounds this issue by exposing the limitations of traditional human-centered research approaches. It is imperative to establish a comprehensive framework that accommodates the dual perceptual centers of humans and AI agents in both physical and virtual worlds. In this paper, we introduce the symmetrical reality framework, which offers a unified representation encompassing various forms of physical-virtual amalgamations. This framework enables researchers to better comprehend how AI agents can collaborate with humans and how distinct technical pathways of physical-virtual integration can be consolidated from a broader perspective. We then delve into the coexistence of humans and AI, demonstrating a prototype system that exemplifies the operation of symmetrical reality systems for specific tasks, such as pouring water. Subsequently, we propose an instance of an AI-driven active assistance service that illustrates the potential applications of symmetrical reality. This paper aims to offer beneficial perspectives and guidance for researchers and practitioners in different fields, thus contributing to the ongoing research about human-AI coexistence in both physical and virtual environments.
The conceptual video demos for symmetrical reality.
Topological symmetry of symmetrical reality. Humans, as components of the physical world, are capable of perceiving both the physical and virtual worlds, including themselves. Similarly, AI agents, as components of the virtual world, are capable of perceiving both the physical and virtual worlds, including themselves. The physical and virtual worlds are heterogeneous with respect to their forms of existence, while entities within the same world are homogeneous.
Human-AI coexistence in symmetrical reality. The AI agent can influence the human's long-term living paradigm on five levels, enhancing the human's overall experience. Conversely, humans can also affect the AI agent on three levels.
Overview of the inverse virtual reality system. The left figure depicts the physical world, while the right figure depicts the virtual world. The AI agent is immersed in the physical world and can manipulate physical objects using a robot arm.
Overview of the inverse augmented reality system. In the left figure, the human and objects outlined in green are mapped to the virtual world shown in the right figure. The AI agent can see both the virtual environment and physical objects.
Illustration of the potential applications. The figure at the top provides a schematic representation of the system’s architecture. The middle and bottom figures serve as practical demonstrations of the system’s capabilities, specifically in the context of beverage service and child location. This application example illustrates that the AI agent is capable of assimilating human behavioral patterns through daily observation, subsequently offering reasonable assistance as per human requirements. This holds true irrespective of whether the need arises in the physical or virtual world.
Conclusion
This paper introduces the emergence of symmetrical reality and presents a comprehensive framework by analyzing its essential components, with dual-center perception and bidirectional interaction identified as key features that demonstrate the value of symmetrical reality. Technical challenges in developing symmetrical reality systems are discussed, followed by an analysis of human-AI coexistence within symmetrical reality. Demonstrations of symmetrical reality systems and proof-of-concept scenarios provide insights for researchers on the practical application of symmetrical reality.
@article{zhang2024emergence,
title={On the Emergence of Symmetrical Reality},
author={Zhang, Zhenliang and Zhang, Zeyu and Jiao, Ziyuan and Su, Yao and Liu, Hangxin and Wang, Wei and Zhu, Song-Chun},
journal={arXiv preprint arXiv:2401.15132},
year={2024}
}