New Benchmarks, Metrics, and Competitions for Robotic Learning

A RSS Workshop – Pittsburgh, 29 - 30 June 2018

Ask your questions for the panel discussion here

Rooms: June 29 DH 2315 -- June 30 GHC 4303


This workshop will discuss and propose new benchmarks, competitions, and performance metrics that address the specific challenges arising when deploying (deep) learning in robotics. Researchers in robotics currently lack widely-accepted meaningful benchmarks and competitions that inspire the community to work on the critical research challenges for robotic learning, and allow repeatable experiments and quantitative evaluation. This is in stark contrast to computer vision, where datasets like ImageNet and COCO, and the associated competitions, fueled much of the advances in recent years.

This workshop will therefore bring together experts from the robotics, machine learning, and computer vision communities to identify the shortcomings of existing benchmarks, datasets, and evaluation metrics. We will discuss the critical challenges for learning in robotic perception, planning, and control that are not well covered by the existing benchmarks, and combine the results of these discussions to outline new benchmarks for learning in robotic perception, planning, and control.

The new proposed benchmarks shall complement existing benchmark competitions and be run annually in conjunction with conferences such as RSS, CoRL, ICRA, NIPS, or CVPR. They will help to close the gap between robotics, computer vision, and machine learning communities, and will foster crucial advancements in machine learning for robotics.


Researchers in Robotics often lack standardized realistic benchmarks to conduct repeatable large-scale experiments in order to evaluate and quantitatively compare the performance of their algorithmic approaches and overall systems. This is in stark contrast to the computer vision community where datasets such as Pascal VOC, ImageNet, or COCO, and the associated evaluation protocols, fueled much of the advances in object recognition, object detection, semantic segmentation, image captioning, and visual question answering in recent years.

The lack of a standardized benchmarks is a significant roadblock for meaningful progress in robotics, especially in robotic learning for perception and action. It currently causes researchers to conduct non-comparable and non-repeatable experiments and ultimately compromises the overall validity of evaluations in our field of research. The goal of the workshop is to provide a forum where the community can discuss and propose new benchmarks, competitions, and performance metrics addressing the specific challenges of robotic learning.


The workshop will cover 1.5 days. The first day (29 June) is dedicated to invited talks, panel discussions, and contributed paper poster presentations with spotlight talks. The organisers, invited speakers, and interested participants will get together in the morning of the second day (30 June) to consolidate the discussions and work on a document that summarizes the outcomes of the workshop. This can be extended further into a survey paper to be submitted to a journal.

Schedule Day 1 (Friday, 29 June) Room DH 2315

Andrea Censi, Liam Paull, Jacopo Tani, Thomas Ackermann, Oscar Beijbom, Berabi Berkai, Gianmarco Bernasconi, Anne Kirsten Bowser, Simon Bing, Pin-Wei David Chen, Yu-Chen Chen, Maxime Chevalier-Boisvert, Breandan Considine, Justin De Castri, Maurilio Di Cicco, Manfred Diaz, Paul Aurel Diederichs, Florian Golemo, Ruslan Hristov, Lily Hsu, Yi-Wei Daniel Huang, Chen-Hao Peter Hung, Qing-Shan Jia, Julien Kindle, Dzenan Lapandic, Cheng-Lung Lu, Sunil Mallya, Bhairav Mehta, Aurel Neff, Eryk Nice, Yang-Hung Allen Ou, Abdelhakim Qbaich, Josefine Quack, Claudio Ruch, Adam Sigal, Niklas Stolz, Alejandro Unghia, Ben Weber, Sean Wilson, Zi-Xiang Xia, Timothius Victorio Yasin, Nivethan Yogarajah, Julian Zilly, Yoshua Bengio, Tao Zhang, Hsueh-Cheng Wang, Stefano Soatto, Magnus Egerstedt, Emilio Frazzoli

Schedule Day 2 (Saturday, 30 June) Room GHC 4303

  • 9.00 Welcome and Introduction
  • 9.10 Break into working groups. Each group discusses and starts drafting concrete proposals for benchmarks, metrics, and competitions.
  • 10.00 Coffee Break until 10.30, continue discussions
  • 11.00 Finalise proposals.
  • 11.30 Get together in a big group, discuss and consolidate proposal drafts. Plan next steps for after RSS.
  • 12.00 Workshop Conclusions

Call for Participation

Our workshop puts a very strong emphasis on developing new benchmarks that address the challenges arising when deploying deep learning for robotics in complex real-world scenarios, current gaps in our collective knowledge in this area, and the necessary new research directions to close these gaps.

We therefore invite authors to contribute extended abstracts or full papers that:

  • identify the shortcomings of existing benchmarks, datasets, and evaluation metrics for robotics
  • propose improved datasets, evaluation metrics, benchmarks, and protocols for robotics that foster repeatable evaluation and motivate research in important areas not well covered by existing benchmarks
  • address specific robotics learning-related research challenges like coping with open-set conditions, uncertainty estimation, incremental / continuous learning, active learning, active vision, transfer learning

Papers on benchmarks and datasets should be guided by the following questions:

  • Where do you see the shortcomings in existing benchmarks and evaluation metrics?
  • What are important research challenges for robotic learning that are not well covered by existing benchmarks?
  • What characteristics should new benchmarks have to allow meaningful repeatable evaluation of approaches in robotic vision, while steering the community to addressing the open research challenges?

Instructions for Authors

Papers can be submitted as extended abstracts (2-3 pages plus references) or full papers (6-8 pages plus references) using this form.

Important Dates

  • June 8, 2018 : Deadline for submission. (Extended)
  • June 15, 2018 : Acceptance notification.
  • June 29 (full day) + June 30 (morning) : Workshop at RSS in Pittsburgh


With support by

  • Trung T. Pham (Postdoctoral Fellow, University of Adelaide)
  • Vijay Kumar (Postdoctoral Fellow, University of Adelaide)
  • Gustavo Carneiro (Associate Professor, University of Adelaide)
  • Peter Anderson (PhD Researcher, ANU)
  • Ingmar Posner (Associate Professor, University of Oxford)
  • Michael Milford (Professor, QUT)
  • Anton van den Hengel (Professor, University of Adelaide)
  • Ken Goldberg (Professor, UC Berkeley)
  • Peter Corke (Professor, QUT, and Director, Australian Centre for Robotic Vision)