Implementation

Components

Initial schematic of the networked work environment.

Network Setup:

We built the central server on AWS, using a t3.micro EC2 instance. This cloud-hosted Linux instance, with its high uptime and global availability, allowed us to abstract away the details of running our ROS master node in some way that would be accessible to all users. While such a low-power compute instance was sufficient for this proof of concept, in development we would aim to upgrade to more powerful computers, and perhaps scalable clusters if our focus were on growth.

To enable users to communicate from their local computers as nodes in a networked ROS graph, we established a VPN. This allowed secure bi-directional communication between the server and client nodes. We used Husarnet, a VPN service from a company called Husarion which implements back-end services for robotics. With the VPN in place and running on our individual computers as well as on the AWS instance, we could synchronize ROS topics, services, and messages between all users.

Unity Robot Simulation:

While this VPN was being set-up initially, the other half of our team was working with Unity. First and foremost, we needed to be able to somehow model the robots we wanted to work with. Since we anticipated that this program would be used for existing robots (aka primarily for controls development, not ground-up robotics hardware development), we did not want to model the TurtleBot from scratch. Instead, we wanted to be able to somehow "import" the model. Our research indicated that most robots are simulated using URDF's and so we chose to look into methods for translating URDF files into Unity models. Here we first ran into the two primary code suites for robotics simulations in Unity: ROS# (aka ROS Sharp) by the Siemens research group and Unity-Technologies by the Unity developers themselves.

Unity-ROS Communications and Controls Set-Up

To allow for communications between the ROS environment and our Unity environments (which would eventually move off a computer to a phone app), we needed to establish some type of connection to stream data. We ended up choosing to use a WiFi protocol since it is the most common wireless connection protocol on devices and has a larger range than other protocols. Additionally, to avoid potentially losing critical data, we opted to use a TCP connection over UDP connection. For ROS-Unity TCP communications, there again already existed two different dominant code suites: ROS# and Unity-Technologies. Furthermore, even for controlling imported URDFs, we had to select between the two options (or so we initially thought, see below for final decision).

Software Nodes

Network Demonstration

As a proof of concept of our network we hacked together the launch files from several in-class labs to create a multiplayer turtlebot simulator — an arbitrary number of users on the network could spawn and control a turtlebot using keyboard teleop and, for example, chase other users’ turtlebots around a STDR map.

AR Environment Creation

In Unity, we first developed an app that used the AR Foundation library and ARCore to detect planes and create a point cloud of the environment from the motion of the phone and the back camera. In this app, the user has the ability to draw walls, place objects, and place robots in the AR scene to build an environment. If the iPhone 12 were the source device, there would also be the option to reconstruct the scene in AR using Lidar from the ARKit library.

We also looked into and developed a separate app intended for multiple users, where two phones in the same environment can simulate the same scene based on a shared anchor point. This was implemented using Unity Multiplayer and the cloud anchors API.

Robot Control in AR Environment

As discussed above, we had two choices of existing code suites to handle importing URDFs, ROS-Unity communication, and Unity robot model joint control. Initially, we tried to just use Unity-Technologies' URDF Importer and ROS-TCP-Connector. However, Unity-Technologies' code did not appear to work for all three needed functions:

We could import a URDF creating a gameObject model
But we couldn't control the model (at least for TurtleBots, we could import and control Niryo bots properly for some reason....).
We could send signals between ROS and Unity using the TCP-Connector.

As a result of this failure, we then tried to use only ROS# instead. However, we again ran into complications.

We could import a URDF.
But we couldn't send signals between ROS and Unity through ROS#'s Ros Bridge.
So we couldn't test if the joint control worked...

Therefore, in the end, we had to write our own C#/python code to integrate the two code suites which was challenging as we had neither C# experience or great intimacy with these massive libraries. Fortunately, either way, we were able to successfully merge the two in the end.

We could use either library for URDF importing.
We used ROS#'s teleop program for generating control messages in ROS but had those signals transmitted to Unity through the Unity-Technologies TCP-Connector.
Then we adapted the ROS#'s C# joint controller to accept messages from Unity-Technologies' TCP-Connector rather than from the ROS# Ros Bridge.

Ubuntu 64-bit - VMware Workstation 2020-12-10 18-17-59.mp4

Successful test of ROS network where two people miles apart control different turtlebots in shared STDR environment

20201211_002341.mp4

Demonstration of ability to detect surface and have bot respond to gravity

Cloud anchors being synchronized across two different devices

Final Integration

This assumes two things that we were unable to finish: a control system that allows us to move the TurtleBot and successful integration of the cloud anchor functionality for multiple user environment synchronization. Discussion of complications that prevented us from completing the project at this time can be found in the Results and Conclusions pages.

Single User functionality:

1. The user first launches the app on their phone. In the background, the app tries to connect to a computer at a hard-coded IP address. If it gets a response from that IP, the app and computer initiate a TCP connection for streaming data between the two devices.
2. The app will then automatically detect planar surfaces.
3. The user can use the UI to place objects on a detected surface. From the dropdown, one of two objects can be selected and then placed. The code for creating and manipulating objects in the environment is written in C# and hosted on the app itself.
  - If “Walls” is selected, they can be placed and act as obstacles in the AR space that the turtlebot would have to account for. They can be removed by pressing the “Clear” button for a fresh environment.
  - If “Turtlebot” is selected, the user can place a single turtlebot at a designated location. On subsequent screen touches, the user can instantly translate the robot around the space.
4. Ideally, a user would then be able to control the robot and drive it around the surface using a tele-op program or an autonomous controls program on their computer.
  - The tele-op ROS node publishes the desired joint angles to a topic in the ROS environment.
  - A separate ROS node subscribes to said joint topic and translates the messages into a data package to be transmitted over the TCP connection to the users phone.
  - On the phone, this package is received, used by the C# code to move the robot's modelled joints and display the changed pose of the robot. Unfortunately, we were unable to get this control scheme working with a Turtlebot placed in the AR environment though we can control a robot placed in an entirely simulated environment on a phone.
5. Ideally, the robot would be able to sense objects and avoid collisions.
  - Simulated objects should theoretically be able to be "seen" by the simulated robot and therefore relatively easily avoided.
  - Real world objects are slightly more challenging to detect as the simulated robot doesn't have any real sensors to detect the objects. Therefore, we would have to rely on the phone's camera to not only visualize the robot in the real world but also to calculate what the robot would "see" from it's perspective.

Multi-user functionality:

One person can first launch the app and create an environment (ie. place walls or other objects on detected surfaces relative to an anchor) as outlined above.
- The location of all created objects would be sent from the phone app to the users computer (published on a ros topic) and therefore (because of the established ROS network) to the AWS ROS instance available for others to also see.
When the second person launches the app, they must place an anchor. Then, all objects created by the first person would show up relative to said anchor.
- On the backend, their app would send a request through the second users computer to the AWS server and receive the location of all placed objects.
Now, each person can place a robot in the shared environment and have the robots navigate and interact with each other.
- For example, the robots could each be controlled by a separate program designed by each user to race through the environment to a destination point (and therefore compare path-planning algorithms) while avoiding collisions with the environment or each other (and therefore compare vision-processing algorithms).

Report abuse