SculptDiff: Learning Robotic Clay Sculpting from Humans With Goal Conditioned Diffusion Policy

Alison Bartsch¹ Arvind Car¹ Charlotte Avra¹ Amir Barati Farimani¹

¹Carnegie Mellon University Mechanical Engineering

Abstract

Manipulating deformable objects remains a challenge within robotics due to the difficulties of state estimation, long-horizon planning, and predicting how the object will deform given an interaction. These challenges are the most pronounced with 3D deformable objects. We propose SculptDiff, a goal-conditioned imitation learning framework that works with point cloud state observations to directly learn clay sculpting policies for a variety of target shapes. To the best of our knowledge this is the first real-world method that successfully learns manipulation policies for 3D deformable objects.

Video Results

X

Line

Cone

Final Shapes

The final shapes created by the policies trained with point clouds inputs. For the target point cloud on the left-most column, the lightness of each point is correlated with the point's z-value to visualize depth. While both human oracles create the best shapes, point cloud diffusion policy is able to successfully create the closest matches to the human demonstrations.

SculptDiff Pipeline

The pipeline of SculptDiff. The state and goal point clouds are encoded with PointBERT and a linear projection head to create a latent conditioning observation along with the previous action executed by the robot. The latent state and goal observations as well as the previous action are the conditioning information used to condition the denoising diffusion process for diffusion policy to generate the predicted action sequence.

Visualization of Sculpting Sequences

The intermediate 3D point cloud states as the clay is sculpted with the SculptDiff policy compared to the target shape point cloud. The lightness of each point is correlated with the point's z-value to visualize depth.