DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
NeuRIPS 2023
Abstract
Combining gradient-based trajectory optimization with differentiable physics simulation is an accurate and efficient technique for solving soft-body manipulation problems. Using a well-crafted optimization objective, the solver can quickly converge onto a valid trajectory. However, writing the appropriate objective functions requires expert knowledge, making it difficult to collect a large set of naturalistic problems from non-expert users. We introduce DiffVL, a framework that integrates the process from task collection to trajectory generation leveraging a combination of visual and linguistic task descriptions. A DiffVL task represents a long horizon soft-body manipulation problem as a sequence of 3D scenes (key frames) and natural language instructions connecting adjacent key frames. We built GUI tools and tasked non expert users to transcribe 100 soft-body manipulation tasks inspired by real-life scenarios from online videos. We also developed a novel method that leverages large language models to translate task language descriptions into machine-interpretable optimization objectives, which can then help differentiable physics solvers to solve these long-horizon multistage tasks that are challenging for previous baselines. Experiments show that existing baselines cannot complete complex tasks, while our method can solve them well.
Graphical User Interface For Soft Body Manipulation
Method
We propose DiffVL to tackle the challenging tasks in SoftVL100. DiffVL utilizes large language models to compile the natural language instructions into a machine-interpretable optimization program. The program comprises of the names of visible elements and Python functions, effectively capturing the essence of the language instructions.
Single Stage Tasks
Language：
Put the knife above the white object. Cut the knife down to match the target shape of the white object. Ensure that the knife does not rotate.
Code:
sample("knife", above(get("white"))),
optimize(
require(similar(get("white"), goal("white"))),
keep(touch("white")),
keep(roll() < 0.1),
keep(pitch() < 0.1),
keep(yaw() < 0.1)
)
Language：
Use the gripper to first lift up the white cube above the black wall and then move it to the target location
Code:
white_cube = get("white cube")
black_wall = get("black wall")
sample("gripper", grasp(white_cube)),
optimize(
require(y(com(white_cube))>y(max(pcd(black_wall))), end=0.5),
require(l2(com(white_cube), com(goal("white cube")))),
keep(touch(white_cube))
)
Language：
Grasp the front of the rope and grasp the back of the rope. Deform the rope into its target shape.
Code:
sample("gripper", grasp(backend(get("rope")))) optimize(require(emd(get("rope"), goal("rope")))) sample("gripper", grasp(frontend(get("rope")))) optimize(require(similar(get("rope"), goal("rope"))))
Multistage Tasks
Language:
1.Put the board above the white objects vertically. Manipulate the white plasticine into the goal directly and do not move other objects.
2.Grasp the objects in the front cube and the back cube. Pick and place the objects the front cube and the back cube into their goal place. Keep touching the objects the front cube and the back cube and do not move other objects.
3.Use the large Gripper to manipulate the object white vertically. Wrap the white to the goal shape as well as the front cube and the back cube in it.
Language:
1.Put the board above the purple objects vertically. Manipulate purple plasticine into goal directly.
2.Put the board above the green objects vertically. Manipulate green plasticine into the goal directly.
3.Grasp the objects yellow. Pick and place the objects yellow into their goal place. Keep touching the objects yellow and do not move other objects.
4.Use the large Gripper to manipulate the object purple vertically. Wrap the purple to the goal shape as well as the yellow in it.
5.Grasp the objects in yellow and purple. Pick and place the objects yellow into its goal place. Keep touching
Language:
1.Grasp the front-end objects of the rope. Deform the rope into its goal shape and please do not break it. Make the pink_ball and black_ball at their place.
2.Grasp the front-end objects of the rope. Deform the rope into its goal shape and please do not break it. Make the pink _ball and black ball in their place.
3.Grasp the back-end objects of the rope. Deform the rope into its goal shape and please do not break it. Make the pink _ball and black ball in their place.
4.Grasp the back-end objects of the rope. Deform the rope into its goal shape and please do not break it. Make the pink _ball and black ball in their place.
More demos
Language:
1.Grasp the right end objects of the rope. Deform the rope into its goal shape and please do not break it.
2.Grasp the ball. Pick and place the ball into its goal place. Keep touching the objects yellow and do not move other objects.
Language:
1.Grasp pink and red vertically. Grasp the pink and reds then move them to the goal place. Make sure other objects are keeping their place and move the tool away in the end.
2.Use the large Gripper to manipulate the white vertically. Wrap the white to the goal shape and the others in it.
3.Put the board above the objects vertically. Manipulate all plasticine into the goal directly.
Language:
1.Put the board above the yellow vertically. Manipulate the plasticine into the goal directly.
2.Grasp the objects yellow. Pick and place the yellow into its goal place. Keep touching the yellow and do not move other objects.
3.Grasp the object upper white. Lift it up first then pick and place the object into its goal place. Keep touching it and do not move other objects.
4.Grasp small balls. Pick and place them into their goal places. Keep touching them and do not move other objects.
Language:
1.Grasp all the small bags and pick and place them on the big one.
Language:
1.Put the board above the white vertically. Manipulate the plasticine into the goal directly.
2.Grasp the red ball and pick and place it into goal place.
3.Grasp the front end of the rope. Wind the rope with the red ball in it.
4.Grasp the red ball and the blue rope. Pick and place them into goal place.
Language:
1.Use the large Gripper to manipulate the white vertically. Wrap the white to the goal shape and the others in it.
2.Grasp the left back end of the rope. Wind the rope into the goal shape.
3.Grasp the right back end of the rope. Wind the rope into the goal shape.
Language:
1.Grasp all the objects. Pick and place them into their goal shape.