Evaluating Our Results

Our final solution somewhat met our design criteria. We were able to generate a structure, take images of the structure, and reconstruct the corners of the structure from these images. We were also able to (somewhat) rebuild the structure, but with a few changes to our design (as explained below).

We encountered many difficulties during our project, both with the CV side and the kinematics side. On the CV side, we found that there was a lot of error with triangulation - even if we got really good corner matches and set a very low epipolar error threshold, the 3d points reconstructed often did not match the actual corners of the blocks. We got around this by designing our filtering algorithm to filter out incorrect corners, and by simply using more images. So, we ended up using 8 pairs of images for our structures, which is more than we had originally planned to use.

On the kinematics side, there were a lot of problems we had to solve! One big issue was picking up the blocks, because they kept slipping out of Tiago's gripper. We tried adjusting the friction coefficient/mass of the blocks, but we found that adjusting the physical properties of the block did not help. Eventually, we just modified the blocks so that they were easier to pick up by changing them to a dumbbell shape. Another difficulty we faced was that when putting down blocks, our robot would often hit the other blocks that had been placed with the block in its hand, even after adding obstacle constraints. To solve this problem, we simplified our design and assumed that blocks wouldn't be placed on top of each other or close to each other (by adding spacing between the corners returned from the CV). The last major issue we faced was that gmapping was not very accurate, so given the same goal location, the robot might go to slightly different places every time when placing a block. To fix this, we added a new aruco marker every time a new block was to be placed on the structure, so that the robot would know where to place the new block next.

With additional time, we would like to use fewer images by potentially finding better angles from which to take images. We want to be able to pick up any type of block, modifying shape and size without it slipping out of the gripper. We would also refine our placing routine to allow for more accurate positions, allowing us to place blocks both closer to each other and on top of each other. Other improvements could be allowing for obstacles on the map and using structures with blocks of different shapes and sizes, placed in any orientation.

Where Does Our Project Fit In?

Our project, in a sense, is pretty similar to 3D printing - the user specifies the design of their model, and then it gets reconstructed (whether that be in simulation or real-life). Extensions of this project would allow users to see anything they can imagine get rebuilt - as long as they can design it in a Gazebo simulation. Practically, this has applications in construction (Robob building a house!), and it is useful for designers to visualize models that they build in simulation. Unfortunately, in order to use this project you have to have a $60,000 TIAGo robot - which not many would want to buy! However, a construction or design company might be able to buy the robot to help its employees visualize their designs/build larger structures. There could also be smaller versions that are able to build smaller models.

In addition to construction, if this were modified to simply pick up and place objects in certain locations, it could be applied to cleaning or organizing a house or some other space. Finding objects on the ground and placing them in a certain area would be perfect for such an application. Again, this would be quite expensive, but the idea could still be used.