DiMSam: Diffusion Models as Samplers for

Task and Motion Planning under Partial Observability

[paper]

Xiaolin Fang1 , Caelan Reed Garrett2, Clemens Eppner2,

Tomás Lozano-Pérez1, Leslie Pack Kaelbling1, Dieter Fox2

MIT CSAIL1 NVIDIA2

Overview

Difficulties in Task and Motion Planning

Continuous constraints

Sampled trajectories reach the goal set, collision-free, IK feasible, etc.

Partial observability

Microwave door occludes the base

Unknown dynamics

Door dynamics depend on an unknown articulation model

Our approach

Diffusion models as parameterized samplers in TAMP
Conditional sampling of diffusion models for different constraints
Sampling in latent space without explicit state estimation or object models

Experiment

TAMP: Joint searching and sampling

Task1 | Close

Initialization
- A microwave is located on the table.
- An obstacle located between the door and the base of the microwave.
Goal
- The microwave is closed.

sim_close_clip1.mp4

sim_close_clip2.mp4

Final plan

Move(obstacle)

Push(microwave)

Sampled values

Pose_o obstacle placement

Traj_o obstacle moving trajectory

Traj_m microwave moving trajectory

Traj_r robot moving trajectory

Task2 | Stow - Close

Initialization
- A cup and a microwave are located on the table.
Goal
- The cup is in the microwave.
- The microwave is closed.

stowclose_clip2.mp4

stowclose_clip1.mp4

Final plan

Stow(cup)

Push(microwave)

Sampled values

Pose_o cup placement

Traj_o cup moving trajectory

Traj_m microwave moving trajectory

Traj_r robot moving trajectory

Task3 | Stow - Close - Blocked

Initialization
- A microwave is on the table.
- A bowl is located near the microwave that may block the door from being fully opened.
Goal
- The bowl is in the microwave.
- The microwave is closed.
Additional constraint
- The microwave needs to be fully opened before stowing the object.

stowcloseblock_clip1.mp4

stowcloseblock_clip2.mp4

Final plan

Move(bowl)

Push(microwave)

Stow(bowl)

Push(microwave)

Sampled values

Pose_o_1 bowl intermediate placement

Pose_o_2 bowl final placement

Traj_o_1 bowl moving trajectory

Traj_o_2 bowl stowing trajectory

Traj_m_1 microwave opening trajectory

Traj_m_2 microwave closing trajectory

Traj_r_1 robot door opening trajectory

Traj_r_2 robot door closing trajectory

The start of Traj_m_2 must be the same as the end of Traj_m_1 (microwave latent state). The planner searches for the plan skeleton and samples values jointly that satisfy such constraints. Same for Traj_o and Traj_r.

Guided Sampling

Real world deployment

We apply the model trained in simulation directly to the real world without finetuning on door closing and opening tasks. In both tasks, conditional samplers are used. The sampled trajectories are further rejected by IK and motion sampler, with collision checking to the observed partial point cloud.

In the door-opening task, the robot is holding a stick. The predicted waypoints are set as the target location of the stick endpoint.

Despite the partial observability due to 1) self-occlusion, 2) missing depth reading of reflective material, 3) missing depth of the glass door, our model is still able to perform the task. See the attached videos for observed point cloud.

close3_info.mp4

close1_info.mp4

close2_info.mp4

open1_info.mp4

open2_info.mp4

open3_info.mp4

Failure cases

close_fail1.mp4

Predicted trajectory falls short. The microwave door is not closed to the end.

open_fail1.MOV

Collision due to 1) stick slips in gripper, and 2) sparsity of the point cloud.

@inproceedings{fang2023dimsam,

title={{DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability}},

author={Xiaolin Fang and Caelan Reed Garrett and Clemens Eppner and Tomás Lozano-Pérez and Leslie Pack Kaelbling and Dieter Fox},

booktitle={arXiv preprint arXiv:2306.13196},

year={2023},

}