COVID-19 has forced educators, students, and researchers alike to learn and work from home. For many in software-related fields, this doesn’t present a huge obstacle — Internet-based repositories and version control make delivering code from a distance a solved problem. However, in the subfield of robotics, being able to test code on physical robots in a lab is often a critical step in development. Participation in such assessments with teammates present can be vastly informative. Given that physical access to robots and large meetings of teammates are prohibited by the pandemic, our group looked to recreate the physical, shared robotic development experience in some remote-work friendly medium. Robotic simulation technology has come a long way in the past decade, but by exploiting the power of augmented reality (AR) we are able to more accurately represent the true environment and interactions between multiple (autonomous or user controlled) agents.
Primarily, we wanted to find an alternative medium to physical meetings that could still allow teams to simultaneously deploy their code onto robot systems and evaluate the resulting performance tangibly. We also needed to find a way to network this system reliably and synchronize results between users. This requires the resolution of multiple problems:
How do we set up the simulated AR environment? How do we get simulated robots to interact with the real world? (the simulation)
How do we collect data in this environment, and use it to do useful tasks, such as path planning around objects, or grasping things in AR space? This should be possible with user given controls, or with autonomous algorithms that the robots follow. (the sensing, planning, and actuation)
How do we get multiple users to interact with one another in the same environment from any physical location using the internet? (multi user collaboration)
How can we communicate all the underlying ROS messages from our simulation, and vice versa, over the network? (networking, server status)
Integration of all of the above problems in a manner that has as low of latency as possible.
The work from our project could be deployed in research labs in industry and academia. It should be simple enough that hobbyists could use it as well. It would allow principal investigators to meet virtually with researchers and watch in real-time, in shared space, as a model of a robot executes tasks. This would help such groups benchmark and iterate more quickly and with greater fidelity to real-world performance.
This project could also be used in educational settings. Running ROS on AWS and simulating in Unity allows students without access to machines that are able to handle computationally expensive tasks to experiment with and explore.
The multi-robot communication also has uses from a teleoperation perspective. It would allow operation of robots safely from a distance if they are physical, as well as the ability to test interactions between robots.