Assignment 4
Due: Apr 26 Wednesday, 11:59pm | Canvas link (rubric & submission) | Points: 3
For this assignment you will think more about perception capabilities needed for your project and you will get started on implementing perception capabilities on the Stretch robot.
Post #9: Last week in Post # 8 we asked you to identify the perception capabilities needed for your project by answering the following questions: What objects, people, or landmarks does the robot need to be able to detect? At what level of precision and temporal continuity do they need to be detected? Based on your answers (which you are welcome to revise now), make a plan for your project for how you will enable the Stretch robot to perceive the necessary items at the necessary precision/frequency. Include at least two alternative approaches as noted below, for each different perceptual capability you need (e.g., you might need different solutions for perceiving a human user versus perceiving an inanimate object in the environment).
Target solution: What would be the simplest feasible perception system for your needs, without making any modifications in the environment? What models/libraries can you use? What data do you need to collect? What training procedures can you use?
Minimum viable fallback solution: Implementing autonomous, generalizable robot perception capabilities is challenging; it is hard to know in advance if an approach will work, and even if it works in some/most situations, there will always be failure cases. It is therefore important to think about consequences of a failing perception system and have a fallback solution to perception failures. We recommend two approaches:
Modify the environment for perception: You might have control on how the robot's environment or interacted objects are designed to make the perception problem easier for the robot. The canonical example for this approach is attaching AR Markers (a.k.a. fiducials, or ArUco Markers) to relevant landmarks in the environment and take advantage of existing algorithms for robustly detecting them. Others use bright, unique colors to make perception easier (e.g., green screen). For this approach, be specific about the proposed modifications and make an argument about their feasibility.
Human-in-the-loop/interactive perception: Another approach is to ask a human to "help" with perception, for instance, by clicking on the target object on an image or creating a bounding box around the object. For example, see the demo of Segment Anything by Meta. For this approach, be specific about the type of human input you will need and the feasibility of obtaining that input from the user or a caregiver.
Post #10: After completing labs on Week 5, write a post demonstrating the perception capabilities you implemented on the Stretch within the context of a task that is relevant to your project (e.g., attach an ArUco marker to an object relevant to your project and place it in a natural relative pose to the robot).
Submit your response on Canvas as a link to the latest post on your team's website.