SynTable - Pipeline

Synthetic Data Generation Pipeline

About

At the time of our work (in Isaac Sim version 2022.1.0), NVIDIA Isaac Sim Replicator Composer does not natively support the functionality required to generate an amodal instance segmentation dataset. In light of this limitation, we have undertaken extensive modifications to the Replicator Composer's original source code, thereby introducing new features that enable the creation of a unified dataset generation pipeline.

SynTable Synthetic Data Generation Pipeline

Our dataset generation pipeline is illustrated below. We use a YAML file to store the parameters and configurations of the scenes to be rendered. The objects and settings required to render the scene are retrieved based on the instructions in the configuration file. We collectively term the objects, materials, and light sources used in our pipeline as assets. Thereafter, the tabletop scene with objects floating above the table is rendered in Isaac Sim. We then run a physical simulation to drop the rendered objects onto the table. For every view within a scene, camera viewpoints and lighting conditions are re-sampled. Subsequently, ground truth annotations are obtained and systematically recorded to create the dataset. Users are able to customize the pipeline as required.

The process of capturing the ground truth annotations for a scene. For each viewpoint, we capture the following: (a) RGB and depth with all objects displayed, (b) object visible masks & bounding box, (c) object amodal masks, (d) object occlusion masks and occlusion rate, (e) occlusion order adjacency matrix.

Customizable variables to build your own dataset

No. of Scenes
No. of Images per Scene
Objects & Table Composition
Material Composition
Lighting Conditions
Viewpoint angles

Data Generated

RGB Images
Depth Images
Object Amodal Masks
Object Occlusion Masks
Object Visible Masks & Bounding Boxes
Object Occlusion Adjacency Matrix (OOAM)
Background Mask (Including Table and Room)

Sample SynTable Annotations: (a) RGB Image, (b) Depth Images, (c) Object Amodal Masks, (d) Object Visible Masks, (e) Object Occlusion Masks, and (f) Object Visible Bounding Boxes

Sample SynTable Annotations: Using Occlusion Order Directed Acyclic Graph (OODAG) to visualise Occlusion Order Adjacency Matrix (OOAM) of a scene.

Domain Randomization for background assets

Page updated

Google Sites

Report abuse