Quantiative Success Metrics

Success Metric

In order to objectively compare the performance of different methods, we define success metrics for every task. The success metric is designed to maximize clarity and reproducibility.

When comparing different methods initial object and arm configurations are matched as closely as possible for different methods.

In all of our comparisons we use 10 different runs for every method where object and arm positions are varied every time. In all runs the arm starts at around 3-5cm distance from the target object.

Lift object Tasks

Examples: "Lift Bowl"

For tasks that involve lifting an object (such as "lift pot" or "lift bowl") we count whether the container has been lifted by more than 5mm off the ground at any point during the trajectory.

Lever Turning Tasks

Examples: "Turn Faucet Vertical to Front"

For tasks that involve turning the lever from vertical to front we count a rollout successful if the lever ended up between between 80° and 90° from the vertical orientation (vertical is 0°).

Put object in container tasks

Examples: "put sweet potato in pot", "put pear in bowl",

For tasks that involve placing an object into a container, we count the rollout as successful if the object ended up inside the container after the trajectory has finished, or if the object ended up exactly above the container such that it will fall into the container after the grippers open.

Put object on another flat object tasks:

Examples: "Put carrot on plate"

For tasks that involve placing an object on another flat object, such as a plate (or cutting board) we count success if the object center was moved on top of the flat object (entering the silhouette of the object when viewed from above).

Flip cylindrical objects upright tasks:

Examples: "Flip Orange Pot Upright", "Flip (Metal) Pot Upright"

We define as success if the cylindrical object is transferred from the round side to the flat side, and the opening is at the top.