Datasets and Benchmarking Tools for Advancing and Evaluating Robotic Manufacturing

Robotics: Science and Systems (RSS) 2023 Conference Workshop

This is a half-day workshop to be held during the Robotics: Science and Systems (RSS) 2023 conference on July 10, 2023 in Daegu, Republic of Korea.

Note: this workshop will run in the morning with a complimentary workshop in the afternoon, "Forum to Develop an Open-Source Ecosystem for Robotic Manipulation."

NIST Assembly Task Board (NIST-ATB) #1 raffle to be held during the workshop! See details below!

NIST-MOAD v0.1 featuring NIST-ATB #1 is available now!
https://www.robot-manipulation.org/nist-moad

Workshop Overview

This workshop is a follow-on to a previous workshop held at RSS:

Benchmarking Tools for Evaluating Robotic Assembly of Small Parts (2020)

This workshop will bring together academia and industry to share research and perspectives on data-driven artificial intelligence (AI) / machine learning (ML) solutions for robotics performing manufacturing tasks including bin picking, kitting, and assembly, among others. Specifically, the workshop will focus on the use of datasets and benchmarking tools for evaluating robotic capabilities in these applications, components of this domain that are lacking in terms of standards and common, systematic methodologies. While some object datasets for robot manipulation exist (e.g., YCB Object and Model Set, Fraunhofer IPA Bin-Picking Dataset), almost none of the objects represented by current datasets are representative of industrial manufacturing tasks. The same can be said about applicable benchmarking practices for robotics in manufacturing; a lack of common practices and metrics. The content of this workshop will cover existing and new efforts that seek to fill these gaps.

To this end, one of the main features of the workshop is the promotion of a new dataset, the NIST Manufacturing Objects and Assemblies Dataset (NIST-MOAD), which features a physical set of standard objects for experimentation and a corresponding set of high-resolution RGB images, RGB-D images, point clouds, segmentation masks, and texture mapped 3D-mesh models, per object assembly. The dataset will feature several example manufacturing object assemblies relevant to a variety of industries including automotive, furniture, and electronics. The NIST-MOAD also includes the NIST Assembly Task Boards (NIST-ATB), fabricated test artifacts that feature generalized assembly tasks including peg insertions, gear meshing, flexible part handling, and nut threading. NIST-ATB have previously been used in several Robotic Grasping and Manipulation Competitions (ICRA 2023 and 2022; IROS 2021, 2020, and 2019), with over 50 NIST-ATB distributed internationally.

A preliminary version of the NIST-MOAD will be prepared for the workshop, and NIST-ATB #1 will be provided for free (a $500 value) to invited and accepted workshop speakers, with 10 NIST-ATB #1 kits to be raffled off during the workshop (to be mailed to the winners after the workshop), and several in-hand at the workshop for audience inspection during breaks and discussion periods. Conducting a workshop in this manner is a similar approach to how the YCB Object Set was initially proliferated, which is the most widely used benchmarking tool for robot manipulation research (over 250 sets distributed).

NIST-MOAD data collection
rig scanning NIST-ATB #1

Discussion Topics

Requirements for relevant datasets and benchmarking tools to use for evaluating robotic manufacturing and proposals for new assets to meet these requirements.
Burgeoning data-driven robotic manufacturing capabilities that are driven by AI/ML which can benefit from and/or supporting the generation of new datasets.
Feedback on NIST-MOAD dataset, including its objects, data types included, data collection methods, how the data can be used, test protocols and metrics for benchmarking using the dataset, and the potential to utilize it in competitions.

NIST-MOAD Dataset (v0.1)

An initial version of the NIST Manufacturing Objects and Assemblies Dataset (NIST-MOAD) is available now and includes the following data from cameras positioned at five different angles captured in 5° rotational increments for the isolated individual components of NIST-ATB #1 as well as the entire task board in disassembled and assembled state:

High resolution RGB images
Colored point clouds (transformed so all point clouds for an object align)
Metadata of the scan (object, date, LUX, camera focal length, depth camera transform matrices)

Access v0.1 of the dataset here: https://www.robot-manipulation.org/nist-moad

UR5 with Robotiq 2F-85 manipulation a gear on NIST-ATB #1

Additional data is planned to be collected in the future, following a similar methodology to the YCB Object and Model Set:

Segmentation masks for images
Texture mapped 3D mesh models
Scripts for downloading and processing the data

Additional relevant manufacturing assembly objects will be added to the dataset based on feedback received from the workshop.

NIST Assembly Task Board (NIST-ATB) #1 Raffle

During the workshop, all attendees will have the chance to win a NIST-ATB #1 (a $500 value). During the scheduled break period, interested attendees will provide their name, organization, and a single sentence of the research area they would use the NIST-ATB for. At the end of the workshop, we will randomly select 10 submissions who will be mailed a NIST-ATB #1 kit after the workshop (all shipping and import fees paid for by the workshop organizers).

Speakers

Megan Zimmerman

National Institute of Standards and Technology (NIST)

Minas
Liarokapis

University of Auckland

Yashraj
Narang

NVIDIA

Elena
Messina

Prospicience LLC

Andra
Keay

The Association For Manufacturing Technology (AMT)

Schedule

All times given are in Korean Standard Time (GMT+9)

A YouTube playlist of the workshop presentations can be found here: https://www.youtube.com/playlist?list=PLfUzSIwyYwvXvf3Ya7_xUsGHLkDxOqLM5

9:00 Workshop and participant introduction
9:10 NIST Manufacturing Objects and Assemblies Dataset (MOAD), Megan Zimmerman [youtube]
9:30 From simulation to sim-to-real transfer for robotic assembly, Yashraj Narang [youtube]
9:55 Developing data-driven robotic solutions for assembly tasks, Minas Liarokapis [youtube]
10:15 Discussion
10:35 Coffee break
11:00 Reproducibility and generalizability of AI/ML for robotics, Elena Messina [youtube]
11:20 The role of standards in advancing AI and robotics in manufacturing, Andra Keay [youtube]
11:40 Discussion
12:00 Workshop end

Contributions

Extended abstracts (2 pages excluding references, following RSS style guidelines, anonymization not required) to cover proposed research that will leverage the NIST-MOAD are sought. Examples of possible proposed research include:

Machine learning techniques for robotics manufacturing applications
Development of perception methods for object identification
Evaluation of manipulation capabilities using the task boards
Other uses of the NIST-MOAD dataset for development and evaluation of robotics

Authors of accepted contributions will be given a NIST-ATB #1 for free (a $500 value) that will either be shipped to them or can be picked up at the workshop.

April 3, 2023: Call for submissions open
May 1, 2023, 23:59 Anywhere on Earth (AoE): Early submission deadline for extended abstract to ensure decision by RSS 2023 early registration deadline (May 7)
May 3, 2023: Notification of acceptance for early workshop submissions
June 12, 2023, 23:59 AoE: Submission deadline for extended abstracts
June 16, 2023: Notification of acceptance for workshop submissions
June 30, 2023: All NIST-ATB #1 kits shipped out to authors of accepted submissions; additional kits will be brought to the workshop for in-person dissemination (if requested) and inspection by workshop participants
July 10, 2023: Date of workshop at RSS 2023 and initial NIST-MOAD dataset (v0.1) consisting of NIST-ATB #1 will be released

Submissions should be e-mailed to adam_norton@uml.edu with the text “[RSS 2023 Workshop Submission]” in the subject line. Authors of accepted submissions are encouraged to upload their papers to arXiv.org; they will also be hosted on a publicly accessible Google Drive folder and linked on the workshop website along with presentation slides and videos of presentations.

Participation

The workshop will be run in a hybrid format, utilizing the Pheedloop platform provided by the RSS conference organizers. Questions will be collected from the online chat during each talk and asked by one of the workshop organizers to the speaker. The virtual hand-raising feature will be utilized during the discussion and monitored by one of the workshop organizers.

Organizers

Adam Norton, University of Massachusetts Lowell
Megan Zimmerman, National Institute of Standards and Technology (NIST)
Kenneth Kimble, National Institute of Standards and Technology (NIST)
Craig Schlenoff, National Institute of Standards and Technology (NIST)
Berk Calli, Worcester Polytechnic University

Contact

Please contact Adam Norton with any questions or comments via e-mail: adam_norton@uml.edu

Funded by the National Institute of Standards and Technology (NIST) under award 70NANB22H114

Page updated

Google Sites

Report abuse