3D-VTAMP

3D Visual Task and Motion Planning

3D Visual Planning framework to solve long horizon, multi-step rearrangement tasks solely from demonstrations

Submitted to CoRL 2024

Learn a task, generalize it to novel scenarios

1. From task demonstrations:

- Lean suitable placements for each object

- Retrieve the goal configuration of the task

2. Given a novel initial observation:

- Propose multiple placements for each object

- A* search over the suggestions until the goal is reached

Example Task:

Place the blue mug inside the microwave and close it.

(In the initial configuration of the task, the microwave is closed, and the mug is blocking it from opening)

Training

- Trained on 20 demonstrations of the 1 mug version of the task (shown in the figure above).

-The mug placements and door opening angles vary within demonstrations.

Planning

- Apply the 3D-VTAMP method, knowing that the microwave is articulated.

- The search can be visualized as a search tree.

Execution

- Once a plan is found, plan each robot's motion ahead to find collision-free paths.

- Use the robot model and 3D observations to check for collisions.

Planning & Execution with 1 mug

Input:

RGBD observation
Names of the objects relevant to the task

Task planning

Task Execution

execution_0.mp4

Planning & Execution with 2 mugs (trained with 1 mug)

Input:

RGBD observation
Names of the objects relevant to the task

Task Planning

Task Execution

execution_1.mp4

Planning & Execution with 3 mugs (trained with 1 mug)

Input:

RGBD observation
Names of the objects relevant to the task

Task Execution

execution_2.mp4

Baselines and Ablations Comparison

Planning Succes Rate

Failure Examples

Greedy Expansion Ablation

This example on the 3 mugs task fails to find the optimal path. It suggests a placement for the red mug that does not move it away from the microwave door opening region

Random Rollouts Ablation

This example of the 3 mugs task also fails. This rollout shows how the selected suggestion for the red mug collides with the green mug in the right.

Some Failure Cases From DP3

We trained an end-to-end policy using DP3 with 22 expert demonstrations to finish the task as a baseline. Below shows some of its failure rollouts.

vtamp_2.mp4

Reached joint limit

vtamp_3.mp4

Failed to put mug in proper place and grasp the handle

vtamp_0.mp4

Failed in grasping the mug

vtamp_1.mp4

Confused two stages (put mug in / open microwave)

one mug case in the real world

two mug case in the real world

three mug case in the real world

failure cases

Page updated

Google Sites

Report abuse