Spatial Reasoning as Object Graph Energy Minimization

Nikolaos Gkanatsios, Ayush Jain, Zhou Xian, Yunchu Zhang, Christopher Atkeson, Katerina Fragkiadaki

Carnegie Mellon University

* = Equal Contribution

Paper

Code

Real World Results

In each of the videos below, we directly evaluate our model trained on simulation in real-world. In addition to robot execution, we also visualize all intermediate outputs including visual grounding and energy minimization for goal generation.

circle_inside_a_plate.mp4

Rearrange all small objects in a circle inside the plate

composition_group.mp4

Put the green cube above the banana and put the strawberry below the blue cube and put the blue cube above the banana and put the banana left of the strawberry

composition_single.mp4

Put the strawberry above the banana and left of the green cube and right of the blue bowl

circle.mp4

Rearrange all the red objects into a circle

line.mp4

Rearrange all cubes into a line

Simulation Results

In addition to robot execution, we also visualize all intermediate outputs including visual grounding and energy minimization for goal generation.

composition_group_1.mp4

Put the yellow cube left of the blue bowl and put the blue bowl right of the gray bowl and put the gray bowl left of the yellow cube

composition_group_2.mp4

Put the yellow cylinder below the gray cylinder and put the blue cylinder right of the gray cylinder and put the brown cube above the yellow cylinder and put the gray cylinder right of the brown cube

composition_single_2.mp4

Put the blue ring below the gray bowl and right of the yellow bowl and below the blue blue bowl

composition_single_1.mp4

Put the red ring above yellow cylinder and right of yellow bowl

circle_1.mp4

Rearrange cyan cubes into a circle

circle_2.mp4

Rearrange green cubes into a circle

line_1.mp4

Rearrange green rings into a line

line_2.mp4

Rearrange red cubes into a line

Simulation with Closed Loop Feedback

In this experiment, our model generates a goal based on the task description and tries to execute it. We then check if the objects managed to reach their predicted goal or not. If it failed, the model retries by regenerating a goal and trying to execute the task.

In the videos below, not all the constraints got satisfied the first time, however after retrying they were completed successfully.

good_2.mp4

good_1.mp4

Put the blue cube right of the blue ring and put the yellow cube right of the blue cube and put the blue ring left of the yellow cube

Put the brown ring left of the cyan bowl and put the yellow bowl left of the brown ring and put the cyan bowl right of the yellow bowl

EBM Visualizations

Below we visualize GIFs of the energy minimization

Inside

Left

Right

Above

Below

(Zero-Shot) Green box inside blue box and blue box right of red box and cyan box left of blue box and red box above magenta box and blue below yellow box

Circle

Line

(Zero-shot) Circle + Inside

EBMs with Pose Information

Below we show that EBMs can be easily extended to also perform tasks which involves pose information

Above, we show the Pizza! task. Here we have pizza slices abstracted as triangles which represents object's center, size and pose information. These objects are first optimized for "circle" concept using circle EBMs and then optimized by pose ebms which are each trained to point to the center of the circle formed.

Next, we show above the Tracker! task. Here the pizza slices keep track of the moving "red cherry" by being trained to always align itself with the cherry.

EBM Energy Landscape Visualization

Below we visualize the energy landscape of our various EBMs. In this visualization, the boxes you see remain fixed, and we score the energy of another box relative to the fixed box by moving it all along the workspace. Energy decreases from Red to black in this heatmap.

Page updated

Google Sites

Report abuse