Adapting Skills to Novel Grasps:
A Self-Supervised Approach
Embedded Files
On this anonymous webpage, we provide further videos for our CoRL 2023 paper submission.
Abstract. In this paper, we address the problem of adapting manipulation skills involving grasped objects (e.g. tools) learned for a single grasp pose to different novel grasp poses. Most robot learning methods address this by learning skills over a range of grasps explicitly, but this is highly inefficient. Instead, we propose a method to adapt such skills directly while only requiring a period of self-supervised data collection during which a camera observes the robot's end-effector moving with the object rigidly grasped. Importantly, our method requires no prior knowledge of the grasped object (such as a 3D CAD model). Through a series of real-world experiments, we show that this outperforms a number of baselines for both RGB and depth modalities. Code and videos of these experiments are available on our anonymous webpage at https://sites.google.com/view/self-supervised-adapt-skills and in our supplementary material.
Experiments Overview
Adapting skills on everyday tasks
Hammer
Skill grasp. Demonstration
Deployment grasp.
No adaptation
Deployment grasp. Adapted
Screwdriver
Skill grasp. Demonstration
Deployment grasp.
No adaptation
Deployment grasp. Adapted
Bread
Skill grasp. Demonstration
Deployment grasp.
No adaptation
Deployment grasp. Adapted
Spoon
Skill grasp. Demonstration
Deployment grasp.
No adaptation
Deployment grasp. Adapted
Wrench/Spanner
Skill grasp. Demonstration
Deployment grasp.
No adaptation
Deployment grasp. Adapted
Glass
Skill grasp. Demonstration
Deployment grasp.
No adaptation
Deployment grasp. Adapted
The videos above demonstrate the 6 real-world tasks taught to the robot using imitation learning.
1st row. For each task the demonstration given to the robot for a specific skill grasp is shown. All skills are taught using the one-shot imitation learning method DOME.
2nd row. For each task, the object is grasped at a random novel deployment grasp. The videos demonstrate that if the skill is not adapted to the corresponding deployment grasp, then, the skill fails to complete the task (as it was taught to the robot in the 1st row).
3rd row. For each task, using our method we adapt the demonstrated skill to the novel deployment grasp. The videos demonstrate how the skill is adapted to successfully complete the task using our method.
Videos for the 6 everyday tasks and peg-in-hole insertion are x2.
Adapting skills for the peg-in-hole task
Skill grasp. Demonstration
Deployment grasp.
No adaptation
Deployment grasp. Adapted
Adapting skills for multiple tasks zero-shot for a single object (NEW VIDEO)
zero_shot_uncut.mp4Paper Overview Video
video.mp4Method Overview
This part provides an overview of our method. A more detailed, step-by-step explanation can be found in Supplementary Material: Algorithms 1-3.
All the videos below are real-time
Section 1: Hammer Object
(A) Self-supervised Data Collection
This section demonstrates a potential reference grasp, reference pose and how we collect our dataset in a self-supervised manner
The EEF at the reference pose with the Hammer grasped at the reference grasp
The robot begins to move around the reference pose in a self-supervised manner, with the hammer rigidly grasped at the reference grasp. This way we emulate potential novel deployment grasps and collect our training dataset
data_collection_aspect.mp4(B) Skill Learning
This section demonstrates:
(1) how our method is deployed before a skill is learned to determine the pose that aligns a potential skill grasp to the reference grasp
(2) The skill learning process. The are many ways a skill can be learned; in this case we teach a skill to the robot with the skill grasp using the one-shot imitation learning method DOME
Skill grasp. Before skill learning begins the EEF grasps the hammer and moves to the reference pose
skill_grasp_align_aspect.movUsing our method, we align the skill grasp to the reference grasp via visual servoing. Then, we store the EEF transformation that aligns the skill grasp to the reference grasp
demo_aspect_hammer.movA Hammering skill with the object grasped at the skill grasp is demonstrated to the robot using the one-shot imitation learning method DOME
(C) Novel Deployment Grasp Skill Execution (Not Adapted)
This section demonstrates what happens if the hammer is grasped at a different pose in the EEF, during skill deployment. That is, the robot fails to complete the hammering task
Deployment grasp. The novel deployment grasp to which we would like to adapt the skill to
This video demonstrates that if the skill is not adapted to the novel deployment grasp, the skill fails
dep_failure_aspect.mov(D) Adapt Skill to the Novel Deployment Grasp
This section demonstrates:
(1) How we use our method to first align the novel deployment grasp to the reference grasp
(2) How we use our method to adapt the skill to the novel deployment grasp
dep_alignment_aspect.movObtain the EEF transformation that aligns the novel deployment grasp to the reference grasp via visual servoing. Using this transformation and the EEF transformation that aligns the skill grasp to the reference grasp, we can obtain the corrective transformation
adapted_T_aspect.movUsing the corrective transformation we can adapt the skill to the novel deployment grasp to successfully complete the Hammering task!
(E) Another Example
This section demonstrates another example of a different deployment grasp, and how our method is used to adapt the hammering skill. The same process is followed as shown above (excluding the data collection part).
Deployment grasp. Another example of a novel deployment grasp to which we must adapt a skill.
Skill Failure
dep_failure_2_aspect.movIf the skill is not adapted to the novel deployment grasp, the skill fails
Skill Adaptation
dep_align_2_aspect.movObtain the EEF transformation that aligns the novel deployment grasp to the reference grasp via visual servoing
dep_adapt_2_aspect.movAdapt the skill to the novel deployment grasp
Section 2: Peg-in-hole
(Videos are x2 speed)
This section shows the same process demonstrated for the Hammer, but for the insertion task using the peg
(A) Self-supervised Data Collection
This section demonstrates a potential reference grasp, reference pose and how we collect our dataset in a self-supervised manner
The EEF at the reference pose with the peg grasped at the reference grasp
The robot begins to move around the reference pose in a self-supervised manner, with the peg rigidly grasped at the reference grasp. This way we emulate potential novel deployment grasps and collect our training dataset
data_collection_aspect.mov(B) Skill Learning
This section demonstrates:
(1) how our method is deployed before a skill is learned to determine the pose that aligns a potential skill grasp to the reference grasp
(2) The skill learning process. The are many ways a skill can be learned; in this case we manually program a sequence of poses the robot tracks for the skill grasp
Skill grasp. Before skill learning begins the EEF grasps the peg and moves to the reference pose
skill_align_aspect_T.movUsing our method, we align the skill grasp to the reference grasp via visual servoing. Then, we store the EEF transformation that aligns the skill grasp to the reference grasp
demo_aspect.movWe manually program a sequence of EEF poses to track to equip the robot with a peg-in-hole insertion skill given the skill grasp
(C) Novel Deployment Grasp Skill Execution (Not Adapted)
This section demonstrates what happens if the peg is grasped at a different pose in the EEF, during skill deployment. That is, the robot fails to complete the peg-in-hole insertion
Deployment grasp. The novel deployment grasp to which we would like to adapt the skill to
This video demonstrates that if the skill is not adapted to the novel deployment grasp, the skill fails
dep_fail_1_aspect.mov(D) Adapt Skill to the Novel Deployment Grasp
This section demonstrates:
(1) How we use our method to first align the novel deployment grasp to the reference grasp
(2) How we use our method to adapt the skill to the novel deployment grasp
dep_align_1_aspect_T.movObtain the EEF transformation that aligns the novel deployment grasp to the reference grasp via visual servoing. Using this transformation and the EEF transformation that aligns the skill grasp to the reference grasp , we can obtain the corrective transformation
dep_adapt_1_aspect.movUsing the corrective transformation we can adapt the skill to the novel deployment grasp to successfully complete the peg-in-hole insertion task!
(E) Another Example
This section demonstrates another example of a different deployment grasp, and how our method is used to adapt the peg-in-hole insertion skill. The same process is followed as shown above (excluding the data collection part).
Deployment grasp. Another example of a novel deployment grasp to which we must adapt a skill.
Skill Failure
dep_fail_2_aspect.movIf the skill is not adapted to the novel deployment grasp, the skill fails
Skill Adaptation
dep_align_2_aspect_T.movObtain the EEF transformation that aligns the novel deployment grasp to the reference grasp via visual servoing
dep_adapt_2_aspect.movAdapt the skill to the novel deployment grasp
Thank you for watching
Page updated
Google Sites
Report abuse