Vision-Based Manipulators Need to Also See from Their Hands

Visualization of the table height distribution shift used in the cube grasping experiments, along with the distribution of initial object and end-effector positions.

 Visualization of the distractor objects distribution shift used in the cube grasping experiments, along with the distribution of initial object and end-effector positions. The distribution shift "3 black (test)" is not depicted below due to space constraints. Note that as in the other experiment variants, the task is to grasp and lift the dark brown textured cube.

Visualization of the table textures distribution shift used in the cube grasping experiments, along with the distribution of initial object and end-effector positions.

DAgger and DrQ results for cube grasping. The first, second, and third rows respectively contain results for the table height, distractor objects, and table textures experiment variants.  Compared to the third-person perspective (dashed lines), the hand-centric perspective (solid lines) leads to better out-of-distribution generalization performance across all three distribution shifts for both DAgger and DrQ. For DrQ, we also see appreciable improvements in sample efficiency when using the hand-centric perspective.

DAC results for cube grasping. Left: base variant (initial object and end-effector position randomization) with no distribution shift between demo collection and training. Center: base variant with table height shift between collection of 25 demos and training. Right: base variant plus three distractor objects with no distribution shift between demo collection and training. Across the three experiment variants, the hand-centric perspective (solid lines) enables the agent to generalize in- and out-of-distribution more efficiently and effectively than the third-person perspective (dashed lines).

Rollouts of the learned policy on the train and test conditions for the table height experiment variant. For DAgger, we train on z_shift = 0 and test on z_shift = {-0.10, -0.05, +0.05, +0.10}. The agent achieves higher success rates given the hand-centric view. The images appear more blurry than before as these are the actual 84x84 image observations given to the agent.

Rollouts of the learned policy on the train and test conditions for the distractor objects experiment variant. For DAgger, we train on a mix of distractors (1 red, 1 green, 1 blue) and test on {3 red, 3 green, 3 blue, 3 brown, 3 white, 3 black}. The agent achieves overall higher success rates given the hand-centric view, but the differences in performance are less drastic: the hand-centric perspective leads to better generalization performance only for the "3 brown" (depicted below) and "3 black" (not depicted) distribution shifts.

Rollouts of the learned policies on the train and test conditions for the table textures experiment variant. We train on a set of 5 table textures from the describable textures dataset (DTD) (Cimpoi et al., 2014) and test on a set of 20 held-out table textures. The agent achieves higher success rates given the hand-centric view.

Visualization of the train and test distributions in six tasks adapted from the Meta-World benchmark (Yu et al., 2020). The last two, reach-hard and peg-insert-side-hard, are custom-made. The hand-centric observability decreases from top to bottom (high, moderate, low). The train and test distributions are disjoint since we resample initial object positions at test time if any of them could have been seen during training.

DrQ-v2 results for Meta-World. Each row contains results for two manipulation tasks that roughly exhibit the same level of hand-centric observability, which decreases from top to bottom (high, moderate, low). Using the proposed approach (both perspectives with a VIB on the third-person perspective's representation, represented by orange curves) leads to the best out-of-distribution generalization performance for all levels of hand-centric observability (though it is matched by the hand-centric perspective when hand-centric observability is high).

Rollouts of the learned DrQ-v2 policies on the train (left two columns) and test conditions (right two columns) for the six Meta-World tasks. We compare the vanilla combination of both hand-centric and third-person views with the version that regularizes the third-person information stream with a variational information bottleneck (VIB). Each GIF shows 3 episodes. The VIB-regularized agent is often able to solve tasks with initial object configurations that confuse the other (vanilla) approach.

Note that these videos are different than the ones shown in the cube grasping environment because here we show policies trained with both hand-centric and third-person perspectives combined (whereas in cube grasping the policies were trained with a single perspective). Therefore, to compare the gifs below, one must compare different columns to each other (e.g., first vs. second column, and third vs. fourth column) rather than different rows.