MR

Tutorials: Mixed Reality

Concepts:

Operator Endpoint: a Windows PC to run Unity exe software
Robot Control Endpoint: the control computer for the robot arm, in our case, it is Raspberry Pi
HuBotVerse Endpoint (Robot): Runs on the control computer to serve as HuBotVerse client, sync command between robot and server
HuBotVerse Endpoint (Operator): Runs on the operator computer or standalone PC to serve as HuBotVerse client, sync command between robot and server
Data Recorder (optional): Locally record data for evaluation

First-time Setup

Operator Endpoint
- Download Oculus App: https://www.meta.com/gb/quest/setup/
  - Version: Oculus App 60.0.0.162.352 (60.0.0.162.352)
- Connect your MR device to your PC, normally Windows, using Quest Link (recommended) or AirLink
- Enable developer mode: https://developer.oculus.com/documentation/native/android/mobile-device-setup/?locale=en_GB
- Oculus App -> Setting -> General ->Turn on "unknown source"
- Oculus App -> Setting -> Beta -> Turn on "Runtime", "Oculus Link Passthrough (required)"
- Launch Quest Link : https://www.meta.com/en-gb/help/quest/articles/headsets-and-accessories/oculus-link/connect-link-with-quest-2/
- Check network IP
- Disable Windows firewall (required): https://support.microsoft.com/en-us/windows/turn-microsoft-defender-firewall-on-or-off-ec0844f7-aebd-0583-67fe-601ecf5d774f
- Setup this PC as hotspot
- Ensure this PC connect internet
Robot Control Endpoint
- Download our code, and install the required packages following readme.md: [link to code repo]
- Install ROS (Ubuntu 20.04 + ROS Noetic, recommended)
- Check network IP
- Disable Ubuntu firewall (required): sudo ufw disable
HuBotVerse Endpoint (Robot)
- Install required packages
HuBotVerse Endpoint (Operator)
- You can use standalone Ubuntu PC (recommanded), or Windows subsystems
- Install required packages
- Setup ROS networking between two PCs: https://wiki.ros.org/ROS/Tutorials/MultipleMachines
- Connect to operator PC's hotspot
For using Geomagic touch: install Windows driver: https://support.3dsystems.com/s/article/Haptic-Device-Drivers?language=en_US

Bring-ups

Setup network, the roscore is recommended to run on the robot side.

Operator Endpoint
- Plugin MR device using Quest Link and launch Quest Link
- Connect to HuBotVerse Endpoint (Operator) network (wired or wireless) and internet
- Start one mode of teleoperation software (mrpt-unity.exe, or mrsc-unity.exe, or touch-unity.exe)
- Enter HuBotVerse Endpoint (Operator) IP (192.168.xxx.xxx:10000), then press connect
HuBotVerse Endpoint (Operator)
- Connect to operator PC's hotspot
- Start unity-ros-endpoint, the tcp_ip is the Operator Endpoint IP:
  - roslaunch ros_tcp_endpoint endpoint.launch tcp_ip:=192.168.xxx.xxx tcp_port:=10000
- python teleop_320_ros2server.py
Robot Control Endpoint
- Start camera node: roslaunch realsense2_camera rs_camera.launch (For MR-SC and Touch)
- HuBotVerse Endpoint (Robot)
  - Start HuBotVerse client node, change IP to the HuBotVerse server IP if necessary
  - python client_320_unity.py
Data Recorder (optional)
- Folder:~/Desktop/imitate_gen3/mr_teleop_scripts/
- python3 record_mycobot320.py --participant <name>--experiments <VR,MR,or Touch> --data-path ~/Desktop/ziniu_data