Following from our self assessment on the proposal (the standards of “Hello”, “Hello world”, etc), we were partially able to complete the Hello World tasks. The “Hello” task was to be able to publish and subscribe to ROS topics from multiple locations from an AWS server, which was completed within the first few days of working on the project. The "Hello World" task was to be able to control robot motion through a UI, whether through an autonomous algorithm completing a task, or through user controls. This assessment also including updating the AR space from multiple users in real time. We were able to set up the environment in a unity scene and create an app on our phones that was able to eventually place and minimally move the robot using commands, but extending this any further proved difficult and put a stop to our ability to test on the tasks that we had envisioned (such as path planning around real and objects created in-simulation). While we were able to update the scene in real time from multiple users, we were not able to get multiple robots to properly interact with one another by the deadline.
We had a lot of parts that worked individually regarding the AR simulation, but integration was too ambitious and ultimately unsuccessful in the given time frame. Some of these parts were:
Turtlebot placement (via URDF importer and ROS#) into AR space, and minimal motion.
Slight motion control of robot using linux mouse method
Point cloud creation and ability to make simulated obstacles (as shown in our demo and presentation)
Multi user robot/object placement in the same AR space
In terms of sensing, planning, and actuation, we were able to create a system which relies on all three features:
We use our phone cameras as sensors successfully (they also obviously fed data about the real world environment back to Unity/ROS), and were able to create point clouds that we wanted to extract as features. The documentation on this feature extraction was sparse, and we ran into roadblocks while trying to take our generated point clouds out of the phone app., despite being able to update it in real time.
Our planning models were to use those features to do basic things such as pathfinding algorithms around the environment. We did also need to create a smart controller to handle data flow between our computers , phones and the AWS server.
Finally, the actuation was for the robot to implement these pathfinding algorithms, but the most we were able to do was have minimal user controls (which was shown in the demo with the wiggling turtlebot that fell off the table) though we did experiment with having the robots move autonomously (by avoiding wall collisions) in the STDR network demonstration. In our final product though, we still had the phone app intelligently visualize the features we wanted in the AR space.
Initiating contact between remote users on the ROS network was difficult. ROS has robust networking protocols for computers on the same local network, but has little support for multiple machines on remote networks. Setting up the VPN allowed us to abstract this away and operate as if on the same local network.
As stated previously, the control systems (user controlled and through autonomous algorithms), were particularly troublesome. One critical aspect of this was being able to accurately represent surfaces using the unity GameBoard gameobject, and having proper mesh colliders that allowed robots to interact with the real world. An example of this is in one of our demo videos, where we placed a turtlebot in the environment and you can see it wiggling around and staying stable on the GameBoard that represents the table’s surface, but we had major difficulties on getting it to move properly beyond that. While we were able to control individual joints on the turtlebot, getting it to move beyond that in the AR space is still a work in progress, and provided much frustration.
Bridging the gaps between the self-contained engines (AR, Unity, local ROS, server ROS) were the points of major development and difficulty. Each of these links took time to establish and presented their own unique challenges.The integration was one of the most difficult aspects of this project, and the amount of effort required to put all these individual components together was underestimated in our initial proposal.
We would need more time to integrate everything together, and also have a more robust system for robotic motion. Making a more robust controls system would probably require a lot more time, but seems much easier than the full integration. These were two of our biggest frustrations during this project, and it seems we needed more time and resources to get this all together.
In terms of what we would have changed, our code base is spread out over many folders and GitHub repositories because we underestimated how fragmented all of our individual, independently developed components for this project were (server setup, ros-to-unity communications, AR environment setup, turtlebot sensing + path planning, etc..). It was hard to have predicted this from the start, but maybe we should have been more cognizant of how to integrate things together and have put more of an emphasis on that during project development.
We recognized that this project should be about the robots and not the networking aspect, so we wanted to minimize the time spent on networking set-up. We figured if we knocked the networking stuff out of the way first, then we wouldn't have any trouble when it came time to test the multi-robot communication. If we were to do this project again in the time frame, maybe it would have been better to put off the networking stuff until it was time for integration, to give us that much more time to flesh out controls, and other robotics/AR related things that we got stuck on.
In the end though, we would like to continue pursuing and even expand upon our initial goals! Nearly all our testing was done simply with the TurtleBot but we want to explore how easy or challenging it is to use other robots in our ecosystem. We want to explore real-world object detection rather than just AR object (ie. wall) placement. We could even look into human-robot interactions where the AR-simulated robot will detect humans and adapt it's behavior in response to human interaction.