Benchmarking in Robotic Manipulation

Benchmarking in robotic manipulation is critical for driving scientific progress but is a divisive topic in the community. We would like to bring together researchers who work in the following different sub-communities on the same platform: robot learning, simulators, embodied AI, grasping, motion planning, perception, sim2real, amongst several others. We hope that with our invited talks and panel debate, we will discuss the crucial topics such as the following:

• The role of simulators in benchmarking

• Benchmarking in the real world

• Standardization (or lack thereof) in datasets, objects and hardware

• Quantitative protocols that the community can agree on

• Which robotic tasks can and should be benchmarked?

• How do we incentivize researchers to benchmark their learnt models?

• Do we even need benchmarking in robot manipulation? When should researchers be disincentivized to benchmark?

We believe that this workshop will benefit the robot learning community in several ways. First, we can hope to replicate the progress made in the computer vision community (e.g. ImageNet, MS COCO, KITTI, etc.) in robotics, if we can come to a resolution on common benchmarks and metrics that several research labs could use. Second, if we could have some constructive feedback on the pitfalls of existing benchmarking efforts (such as being too focused on making incremental gains on a leaderboard), this will allow the community to propose novel benchmarking contributions in future conferences, which will further incentivize good science in the field.