We wanted to create a programming by demonstration system that would be able to grasp toys with Aruco markers attached to them after witnessing a demonstration of how it should be done. Because we want the user to easily perform a demonstration for the robot, we first added the video feed from the realsense camera on the Stretch onto our interface (picture below). We also implemented the abliity to teleoperate the Stretch's joints using our interface this week as that is crucial to perform a demonstration.
Next, we started creating the programming by demonstration system. For our initial demonstration we just used an aruco marker placed on medicine as we didn't have a better object and the grasp for this object seemed the best to test out at first (however we will retest our PbD system with a marker placed on a more suitable object like a toy). The first video linked below is how we demonstrated the task we wanted to playback. The second video linked below is the playback. As you can see from the video, the robot successfully moves its base and gripper to grasp the object like we did in the demonstration. This will be extremely useful for our project as we can demonstrate the robot how to pick up certain toys optimally and using this PbD system, it will be able to recreate the task of picking up those toys.