Physical Metaverse

Problem

The metaverse is a digital world where users can interact with each other and virtual objects. While it's possible to manipulate virtual objects in the metaverse, physical actions cannot be performed on real-world objects. To illustrate, let's say a user in Genoa wants to pour a drink for a friend in Siena, who has upper limb impairments. In the metaverse, it's not possible to physically pour water into a glass for someone in the real world. Avatars cannot touch or manipulate physical objects outside of the virtual environment. Therefore, we need to find a way to enable physical interaction with real-world objects and environments in the metaverse.

The Idea

The Physical Metaverse is a revolutionary concept that enables users to physically interact with objects in the virtual world. By using a robot in a teleoperation scenario, a user's avatar can perform physical actions such as pouring water. What's remarkable is that the robot's physical appearance is hidden from the streaming video, so other users do not perceive its presence.

To achieve this, the Physical Metaverse introduces a new type of avatar called "Avatarms". The Avatarms has the ability to manipulate and perceive both physical and digital objects, thanks to a robotic manipulator that physically interacts with real-world objects in the metaverse. This elevates metaverse interaction to a whole new level, unlocking a wide range of new research and applications.

Robot Cancellation

Robot cancellation is a powerful technique that removes the appearance of a robotic manipulator from a streaming video. This is achieved by creating and applying a dynamic chromatic mask on the camera streaming of the environment.

t=0: a shot of the background without the robot is acquired.

t>0: the binary mask is computed in real-time using the CAD of the robot and combined with the background image to obtain the chromatic mask.

The superimposition operation (denoted with the § operator) of the chromatic mask on camera frames returns a scene diminished of the robot.

Comparison of frames before and after applying the robot cancelling algorithm. A final evaluation of the cancelling percentage is post-performed.

Performing the Physical Action

The physical action is executed by a robot, which is controlled by a human through teleoperation.

The position of the center of mass of the virtual object is mapped in the end-effector pose to pair the movement of the object's physical counterpart.

The closure state of virtual digits is mapped into the gripper, while the internal forces measured by sensors on the jaws of the gripper are projected on human digits enabling haptic interfaces.

The manipulability measure of the robot, in form of a vibrating cue, aware the user of the singularity state of the latter.

Haptic Feedback

The force data are registered by the ATI sensors integrated into the robot gripper, which grasps the real object, and reproduced by the haptic device to the fingers of the user.