Assignments

WEEK #8

Labs

This week you can work to catch up on any labs from previous weeks.

Assignment 8

Due: May 24 Tuesday, 5:00pm | Canvas link (rubric & submission) | Points: 6

In this week's update you will present a proposal and plan for your projects.

PART 1: Project proposal (2 points)

Building on last week's ideation about project directions, your increasing experience with the robot and task domain, and discussions with the teaching staff decide on your specific project topic and write a post to answer the following questions.
- General motivation: Write your own version of the motivation for building on picking robots. Explain why it is challenging. You can reference the problem description on the course home page.
- Specific motivation: Describe the particular failure of the baseline approach to robotic picking that you would like to address in your project, referencing the common failure modes. Describe which item attributes or item configurations lead to this type of failure (e.g. "bagged items", "non-rigid items", "items with indistinct packaging", "bins that are +75% full", etcetera). Provide evidence for the problem based on a quantitative analysis of the baseline approach and/or a specific video or image illustrating the problem.
- Technical approach: Describe your approach for addressing the problem described in your specific motivation, referencing the topics of interest from last week (i.e., incremental container modeling, data-driven perception, environment co-design, extraction strategies, remote human assistance, or other). Provide details on how you will implement your approach.

PART 2: System (2 points)

Draw a ROS-level system figure for your proposed project. Be sure to indicate:
- Which modules are already implemented, which need to be developed, and who will work on developing which module;
- How each module will communicate with one another;
- Which parts constitute the minimal viable product, and what parts are stretch goals.
Draw a Finite State Machine or storyboard that illustrates how your system will work.

PART 3: Plan (2 points)

Write detailed deliverables for three upcoming milestones:
- Week 9: Demonstrate feasibility of proposed approach
- Week 10: End-to-end pick with proposed approach
- Week 11: Improved end-to-end pick with proposed approach and formal evaluation of approach

Write a self contained update that would make sense to someone who reads it independently of this assignment description. Submit a link to your update on Canvas.

WEEK #7

Labs

This week we have one last lab, Lab 34, which will provide high-level guidance for closing the loop to pick up an item detected through perception.

Assignment 7

Due: May 17 Tuesday, 5:00pm | Canvas link (rubric & submission) | Points: 3

In this week's update you will demonstrate your first perception-driven item picking from the shelf.

PART 1 (2 points)

After completing Lab 34 make a video of the robot attempting to pick up at least two different items that it is able to detect, in at least two different scenarios (i.e., different shelf compositions and configurations). Post a video describing your system.

PART 2 (1 points)

Next week you will start working as a team to focus your projects on a particular sub-problem. To that end, this week you should start team discussions to identify topics of interest for specialization. A few possible directions are presented below, but you might come up with others. Write a paragraph describing your interests (pick at least two) and particular approaches you might take for each. Give motivation for why you would like to pursue these directions, preferably based on the failures of the baseline picking approach you have implemented so far. Your specialized solution can focus on a subset of the objects (based on attributes) or a subset of failure modes. Reference the Project Scope documentation which consolidates your efforts to identify performance metrics, failure modes, and object attributes to take inspiration. You will write a more detailed proposal next week, so this is the initial ideation and exploration phase. Possible topics of specialization:

Incremental container modeling: Assume the robot can watch a person stow things on the shelf over time and use this temporal information to build better models of how objects are configured on the shelf.
Data driven perception: Collect larger object datasets with ground truths to develop better item perception algorithms.
Environment co-design: Fabricate end-effector, shelf, or workcell enhancements that facilitate picking.
Extraction strategies: Explore alternative strategies for extracting objects from the shelf, different from the classical approach of rigid grasping based pick-and-place. For example you can pull the object closer to the shelf edge or tip it over to create an overhang before attempting to pick; or you can try to push other objects away from the target object to create space around it.
Remote human assistance: Build interfaces to enable a remote operator to efficiently assist the robot with in challenging scenarios (e.g., when perception fails).

Write a self contained update that would make sense to someone who reads it independently of this assignment description. Submit a link to your update on Canvas.

WEEK #6

Labs

We are half way through the quarter and this week we will continue implementing basic perception skills to detect an item on the shelf. Lab 32 will explore different clustering methods to segment the pointcloud within a shelf bin and Lab 33 will involve simple feature extraction from the segments to try to recognize them.

Assignment 6

Due: May 10 Tuesday, 5:00pm | Canvas link (rubric & submission) | Points: 4

In this week's update you will demonstrate new perception capabilities and discuss what specific problem you will focus on in your project.

PART 1 (2 points)

After completing Lab 32 provide an analysis of the three different clustering algorithms that you tried for segmentation. Show the segmentation results for each method in at least five different bin configurations involving different items, at different packing densities, in different arrangements. Then quantify the segmentation success for each method by counting correctly segmented items and errors (false negatives: over segmented or undetected items; false positives: segments that do not correspond to an item). Write up these results in your update (under subsection "Segmentation Results") including images for segmentation results. Discuss which method you might use in your project.

PART 2 (2 points)

After completing Lab 33 provide a similar analysis for item recognition. Be sure to report accuracy (% of correctly identified objects) as well as a confusion matrix that characterizes the errors that your recognizer makes. You can do this analysis in the same scenes as the ones from Part 1. Write up these results in your update (under subsection "Recognition Results"). Discuss which method you might use in your project.

Write a self contained update that would make sense to someone who reads it independently of this assignment description. Submit a link to your update on Canvas.

WEEK #5

Labs

This week we will jump into robot perception, one of the key challenges of making robots do useful tasks autonomously. As perception algorithms tend to be computationally expensive, due to the high dimensionality of sensor data, many perception algorithms are implemented in C++. In Lab 27 we will create our first C++ package and then learn to deal with point clouds obtained from Fetch’s PrimeSense camera in Labs 28-29. Next we will get a very useful perception library in ROS, namely AR marker detection, working with our point cloud data in Lab 30. Finally, we will integrate perception and arm motion by developing a Programming by Demonstration (PbD) system in Lab 31. This PbD system will allow you to: (a) save end-effector poses relative to an AR marker and (b) move the robot’s arm to the same relative poses after the marker has been moved. Using this system you will be able to define different actions relative to the shelf (which will have markers attached to it) for manipulating objects that are on the shelf. You will do this simply by saving a sequence of end-effector poses and gripper states (open/closed) relative to the marker on the shelf.

Assignment 5

Due: May 3 Tuesday, 5:00pm | Canvas link (rubric & submission) | Points: 6

In this week's update you will demonstrate the perception capabilities and the PbD system you developed in the labs.

PART 1 (2 points)

At the end of Lab 30 you will have a smart point cloud cropper that cuts out part of the robot's point cloud that corresponds to a particular bin on the shelf, based on its pose relative to an AR marker attached to the shelf. Make a video that demonstrates the smart cropper by visualizing the cropped point cloud in RViz and showing how the cropped point cloud updates over time when you move the shelf around, successfully capturing the contents of a particular bin in different shelf poses. Post this video together with a brief description of what it demonstrates.

PART 2 (4 points)

At the end of Lab 31 you will program (by demonstration) three robot actions applied on objects placed on the shelf (see image below) to demonstrate the capabilities of the PbD system you developed. Make a video that demonstrates (a) the process of programming at least one of those actions, and (b) the successful execution of all three actions on the real robot. Post this video with a description of what it demonstrates.

Write a self contained update that would make sense to someone who reads it independently of this assignment description. Submit a link to your update on Canvas.

WEEK #4

Labs

This week we will continue to work with RViz and get started on robot manipulation. Labs 19-22 will introduce you to Inverse Kinematics (IK) and motion planning with obstacles and constraints. Labs 23-25 will introduce you to coordinate frame transforms using TF and transformation matrices. Lab 26 will give guidance for implementing two RViz tools to command robot manipulation with interactive markers, as part of Assignment 4 described below.

Assignment 4

Due: Apr 26 Tuesday, 5:00pm | Canvas link (rubric & submission) | Points: 6

In this week's update you will demonstrate the teleoperation interface that you developed in the labs.

PART 1 (3 points)

In Lab 26 you will first develop an RViz-based interface that involves an InteractiveMarker, shaped like the robot's gripper, which can be moved around to specify a target gripper pose and send the robot to that pose. When moved around, the marker should change color to reflect whether the corresponding gripper pose is reachable by the robot or not. After testing it in simulation, you will test this interface on the real robot to grasp a soft object from the shelf. To do this, use your interface to (i) send the robot gripper to a "pre-grasp" pose near the object, (ii) then send it to a "grasp" pose, (iii) close the gripper, and (iv) send the gripper to a "lift" pose. Make a video that shows (a) the use of your RViz interface and (b) the successful grasping the object by the robot. See below example videos (in simulation and on the real robot) of such an interface. Post your video with a brief description of what it demonstrates.

PART 2 (3 points)

Next you will develop an RViz-based interface that involves an InteractiveMarker corresponding to a world object which can be moved around. You will use transform arithmetic to compute "pre-grasp", "grasp", and "lift" poses for the gripper relative to the current pose of the object. You can use any type of visualization to indicate whether an object pose has reachable gripper poses associated with it (e.g., change color of the object and attach text or separately visualize the associated gripper poses in different colors). After testing it in simulation, you will test this interface on the real robot to grasp a soft object from the shelf. To do this, use your interface to (i) place your interactive object marker approximately where the object appears in the robot's pointcloud stream, and (ii) send the robot to computed pre-grasp, grasp, and lift poses one at a time (close the gripper before lifting). Make a video that shows (a) the use of your RViz interface and (b) the successful grasping the object by the robot. See below example videos (in simulation and on the real robot) of such an interface. Post your video with a brief description of what it demonstrates.

Write a self contained update that would make sense to someone who reads it independently of this assignment description.

Submit a link to your update on Canvas.

WEEK #3

Labs

This week we will work on ROS visualizations and robot navigation through Labs 11 through 18 (more details in the assignment description).

Assignment 3

Due: Apr 19 Tuesday, 5:00pm | Canvas link (rubric & submission) | Points: 7

In this week's update you will demonstrate the teleoperation interface that you developed in the labs and discuss teamwork strategies.

PART 1 (2 points)

Labs 11-15 will introduce you to a visualization tool in ROS, called RViz, and teach you how to create custom interactive visualizations different types of information. By the end of Lab 15 you will have implemented the following functionalities in RViz: (a) custom “Markers” that provide a visualization of the robot’s path as you drive it around around, and (b) custom “InteractiveMarkers” that allow you to trigger robot actions, such as moving or turning the robot using odometry. Make a screen capture video of your RViz interface demonstrating both capabilities: drive the robot around using the InterativeMarkers you created and demonstrate the Marker-based visualization of the path it takes. Post this video with a short description of what it demonstrates.

PART 2 (2 points)

Labs 16-17 will introduce you to basics of robot navigation and teach you how to make a map, localize in it, and navigate to a target configuration in it through RViz and through code. By the end of Lab 17 you will make a tool that allows you to annotate a map by driving the robot to a desired location in the map and using command line inputs to trigger the annotation process and name the location. The same tool should allow you to send the robot to a previously annotated location on the map. Make a screen capture video that demonstrates driving the robot around to annotate distinct locations on the map and sending the robot to those locations. Post this video with a short description of what it demonstrates.

PART 3 (3 points)

Lab 18 involves putting together what you learned in the previous exercises to build a tool for annotating a map through a web/RViz interface and sending the robot to annotated locations. Make a screen capture video demonstrating the use of this tool to annotate distinct locations on the map and sending the robot to those locations. Post this video with a short description of what it demonstrates.

Write a self contained update that would make sense to someone who reads it independently of this assignment description.

Submit a link to your update on Canvas.

WEEK #2

TODOs

Come up with a team name and a name for your version of the robot, to replace our current placeholders "Team GIX" and "Fetch." Think of when you will present your project at the end of the quarter; you'll want to say something like "Hello, we are Team Warebots and this is our first robot, Pickie!" Update your team page with this information. It is okay to change these in later weeks, but try to pick something for now.

Labs

This week we will start with Lab 8 where you will learn to inspect the current state of the robot. Then in Lab 9 and 10 you will learn how to make a browser-based user interface integrated with ROS. By the end of these labs, you will be able to develop a tele-operation interface that lets you remotely control different parts of the robot. You are welcome to complete these tutorials as a team, in pairs, or individually.

Assignment 2

Due: Apr 12 Tuesday, 5:00pm | Canvas link (rubric & submission) | Points: 6

In this week's update you will demonstrate the teleoperation interface that you developed in the labs and discuss teamwork strategies.

PART 1 (4 points)

After completing Lab 10, use the browser-based teleoperation tool you developed to control the simulated Fetch robot to do the following tasks.

- Drive from the starting pose to the table
- Raise the torso and lower the head to look at the table
- Move the arm (and possibly the base and torso) and open/close the gripper to pick up the blue cube on the table, without colliding with the table.
The tele-operator should be able to do these tasks without looking at the simulation (Gazebo) screen and without relying on their memory. In other words, the teleoperation interface should display sensory information to help guide the operator.
Make a screen capture video of the teleoperation process. At the end, open the Gazebo window to show that the object was successfully picked up by the Fetch robot. Upload your video to Youtube and include a link in your update that describes all elements on your tele-operation interface.

PART 2 (2 points)

Assign team roles to everyone in your project team, considering common potential roles in this class. Write a blog post that describes everyone's roles and concrete responsibilities. Consider weekly responsibilities as well as quarter-term project responsibilities. In addition, write a paragraph about strategies you will follow to ensure everyone in the team acquires the knowledge and skills they hope to get out of the class (e.g., everyone in the team learns ROS).

As before, try to write a self contained update that would make sense to someone who reads it independently of this assignment description.

Submit a link to your update on Canvas.

WEEK #1

TODOs

Create a Google Sites page for your team. The main page should be structured like a blog where you will post your weekly updates. Add another page titled "Team" and add a photo and short bio of each team member there.
Everyone on the team should have a GitHub account.
Carefully read the project scope description on the Home page.

Labs

This week we will start learning about the Fetch mobile manipulator and how to control it through ROS. Your goal is to complete Lab 1 through Lab 7 where you will control the robot's base, gripper, torso, head, and arm, both in simulation and on the real hardware. We recommend doing these tutorials individually or in pairs so that everyone can get a strong foundation for working with the Fetch robot.

If you are new to ROS you might also want to go through the Getting set up tutorial and Lab 0 which involves going through the basic official ROS tutorials. The following video tutorials might be helpful: Video-1, Video-2, Video-2.1, Video-2.2.

Assignment 1

Due: Apr 5 Tuesday, 5:00pm | Canvas link (rubric & submission) | Points: 5

For this assignment you will get more familiar with the Amazon Picking Challenge (APC), which is closely related to the projects you are doing in this class. As a starting point read the official APC rules from the 2016 challenge (you can focus on the section "PHASE 4").

Part 1 (2 points): Based on your understanding of the goals of the project, create a list of performance metrics with which to evaluate a robotic picker. Identify as many metrics as you can think of and give a precise definition of each metric. As an example, here is one performance metric you should include: "success percentage/ratio is the fraction of pick attempts that result in the requested item to be picked and placed into a tote without any error". As part of this process make a list of all types of errors you can anticipate (the rule book already identifies a few "penalties").
Part 2 (1 point): Find a video from a picking challenge team demonstrating their robot performing the task on at least five different objects. Choose two metrics from your list and measure them (to the best of your ability) based on the video to obtain a rough baseline number for those performance metrics.
Part 3 (2 points): Go over the full list of objects that were used in APC 2015. Create a list of object attributes that may influence the best strategy or the difficulty of the picking task. Identify as many attributes as you can think of, give a precise definition, and provide examples of objects that differ in terms of each attribute. As an example, here is one attribute you should include: "rigid/non-rigid: whether the object maintains its shape while being manipulated; e.g. the soap bar is rigid, the dog toy is non-rigid.

Write your first weekly update as a team with the information from these three exercises. Be sure to include links (or embed) any external information, such as the video you use for Part 2. Try to write a self contained update that would make sense to someone who reads it independently of this assignment description. To that end, you are welcome to copy or rephrase parts of this assignment description.

Submit a link to your update on Canvas.

Report abuse