Projects

PROJECT SCOPE

Metrics

At the beginning of the quarter, project teams worked independently to identify different metrics for assessing robotic pickers. Here we consolidate the set of metrics that will be used to measure the performance of the robotic pickers developed in this class. Like many robotic systems, the key performance metrics for picking are related to the efficacy and efficiency of the system. We propose the following key performance metrics:

• Efficacy (% success, %failures): The fraction of successful and failed attempts out of a total number of pick requests. To better understand how our system fails, we will have a detailed categorization of failure types as described below.

• Efficiency (task completion time): The average time it takes for the robot to complete a successful pick.

• Pick rate (items per hour): To capture efficacy and efficiency with a single metric, we will measure overall system performance with the average rate of target objects successfully picked within a unit time. This will be computed as the number of successful picks over the total time taken to complete a total number of pick requests, measured as items per hour. 

Successful attempts end when the picked object is dropped into the red tote and the robot returns to its start pose. Failed attempts end when the robot or an experimenter observing the robot declares one of the failures listed above and the robot returns to its start pose.

Failures

We categorize the ways in which the robotic picker may fail as follows:

• Pass (P)*: The robot did not attempt to pick the requested item, due to a perception error (could not detect item) or manipulation error (failed to find a confident grasp or a feasible motion plan)

• Stuck (S)*: The robot arm was overloaded or got stuck while executing plan due to a collision with an external obstacle. 

• No pick (N)**: Attempted but could not pick up anything due to perception or manipulation error

• Fall (F)**: The robot dropped the requested item on the floor or in to the tote from higher than 0.3 meters. 

• Damage (D): The robot damaged the requested item with its manipulator. 

• Wrong item (W): Picked up wrong item, due to a perception or manipulation error

• Additional unwanted items picked (U): The robot picked additional unwanted item(s) in addition to the target item due to perception or manipulation error.

• Fall other item (F2): The robot dropped other items while extracting the target item. 

• Damage other idem (D2): The robot damaged other items while extracting the target item.

• Container damage (C): The robot displaced or damaged the shelf with its manipulator.

The failures marked with * are ones that the robot will be aware of by design (i.e., the software will decide whether to pass and will receive a signal when the arm refuses to move due to an overload). The failures marked with ** are ones that the teams will try to implement detectors for using the robot's sensor. All other ones can be identified by an experimenter observing the system.

Object types

Project teams also looked closely at a set of objects from the Amazon Picking challenge to identify the different dimensions in which items can vary. Here is a subset of those item attributes that we will pay attention to as we specialize picking strategies for different types of objects. You can assume that the "listing" for an item, which involves information related to an item that the robot has access to when an item is requested, will involve a categorization of the item in these dimensions.

• Box (yes/no): Whether the item is a rectangular prism or not. For items that are boxes, the dimensions will also be available.

• Cylinder (yes/no): Whether the item is (mostly) cylindrical or not. For items that are cylinders, the dimensions will also be available.

• Rigid (yes/no): Whether the item maintains its shape when placed on the shelf in different configurations.

• Compliant (yes/no): Whether the item can flex when grasped without being damaged.

• Bagged (yes/no): Whether the item has plastic film packaging around it.

• Fragile (yes/no): Whether the object could be damaged from a drop or not.

In addition to these attributes teams can augment the item listings with additional information and data, such as weight of the item, canonical images or pointcloud segments of the item, feature representation of the item (e.g. color or shape histogram).

PROJECT TEAMS & BLOGS

• Team Bounty, video

  • Jakub Filipek, Linden Gan, Jacob Lee, Steven Su

• Team "Escalating to SIGTERM", video

  • Matthew Arnold, Tudor Fanaru, Kaelin Laundry, Shikuang Zhou

• Team 3, video

  • Aaron Jenson, Ioannis Lefkaditis, Noah Ponto, Victor Shan

• Team Nark, video

  • Mrigank Arora, Tanish Kapur, Long Thanh Nguyen, Ritadhwaj Roy Choudhury