Results

Scenario 1: Performance on Various Tasks

We showcase successful trajectories collected by our method and compare them to unsuccessful ones collected by other methods. For success rates, see the quantitative results section below. Left: V-PTR (ours). Right: (clockwise, from top left) PTR (Kumar et al. 2023), VIP (Ma et al. 2022), R3M (Nair et al. 2022), masked visual pre-training (Xiao et al. 2022)

Croissant from bowl

V-PTR (ours)

final_croissant_ego4d_left_rear_shift_2_success.mp4

PTR

final_croissant_scratch_left_front_shift_2_failure.mp4

Masked Visual Pre-training

final_croissant_bc_mae_left_front_shift_1_failure.mp4

VIP

final_croissant_rl_baselines_vip_left_rear_under_failure.mp4

R3M

final_croissant_bc_r3m_right_front_shift_2_failure.mp4

Sweet potato on plate

V-PTR (ours)

final_potato_plate_ego4d_right_rear_under_success.mp4

PTR

final_potato_plate_scratch_right_front_shift_1_failure.mp4

Masked Visual Pre-training

final_potato_bc_mae_left_rear_shift_2_failure.mp4

VIP

final_potato_rl_baselines_vip_right_rear_shift_1_failure.mp4

R3M

final_potato_bc_r3m_left_rear_shift_2_failure.mp4

Knife in pot

V-PTR (ours)

final_knife_pan_ego4d_left_front_under_success.mp4

PTR

final_knife_pan_scratch_left_front_shift_2_failure.mp4

Masked Visual Pre-training

final_knife_bc_mae_left_rear_shift_2_failure.mp4

VIP

final_knife_rl_baselines_vip_right_front_shift_1_failure.mp4

R3M

final_knife_bc_r3m_right_front_under_failure.mp4

Cucumber in pot

V-PTR (ours)

final_cucumber_ego4d_right_rear_under_success.mp4

PTR

final_cucumber_scratch_right_rear_shift_2_failure.mp4

Masked Visual Pre-training

final_cucumber_bc_mae_left_front_shift_1_failure.mp4

VIP

final_cucumber_rl_baselines_vip_left_rear_shift_2_failure.mp4

R3M

final_cucumber_bc_r3m_right_rear_shift_2_failure.mp4

Open Microwave

V-PTR (ours)

vptr_open.mp4

PTR

scratch_open.mp4

VIP

vip_open.mp4

Sweep Beans

V-PTR (ours)

vptr_sweep.mp4

PTR

scratch_sweep.mp4

VIP

vip_sweep.mp4

Even when successful, VIP has qualitatively worse performance (i.e. smaller swept area)

Scenario 2: Performing tasks with novel distractors

We introduce various distractors to test whether the robot is able to identify the correct object to pick up. The order of videos is the same as before.

Croissant from bowl with distractors

V-PTR (ours)

final_croissant_distractors_ego4d_right_rear_shift_2_success.mp4

PTR

final_croissant_distractors_scratch_right_rear_shift_2_failure.mp4

Masked Visual Pre-training

final_croissant_distractors_bc_mae_right_front_shift_1_failure.mp4

VIP

final_croissant_distractors_rl_baselines_vip_right_front_under_failure.mp4

R3M

final_croissant_distractors_bc_r3m_right_front_under_failure.mp4

Sweet potato on plate with distractors

V-PTR (ours)

final_potato_distractors_ego4d_right_rear_shift_1_success.mp4

PTR

final_potato_distractors_scratch_right_rear_shift_1_failure.mp4

Masked Visual Pre-training

final_potato_distractors_bc_mae_left_front_under_failure.mp4

VIP

final_potato_distractors_rl_baselines_vip_left_front_shift_1_failure.mp4

R3M

final_potato_distractors_bc_r3m_left_rear_shift_1_failure.mp4

Knife in pot with distractors

V-PTR (ours)

final_knife_distractors_ego4d_left_rear_shift_1_success.mp4

PTR

final_knife_distractors_scratch_right_rear_under_failure.mp4

Masked Visual Pre-training

final_knife_distractors_bc_mae_right_front_shift_1_failure.mp4

VIP

final_knife_distractors_rl_baselines_vip_right_front_under_failure.mp4

R3M

final_knife_distractors_bc_r3m_left_front_shift_1_failure.mp4

Cucumber in pot with distractors

V-PTR (ours)

final_cucumber_distractors_ego4d_left_rear_shift_2_success.mp4

PTR

final_cucumber_distractors_scratch_right_rear_under_failure.mp4

Masked Visual Pre-training

final_cucumber_distractors_bc_mae_left_rear_shift_1_failure.mp4

VIP

final_cucumber_distractors_rl_baselines_vip_left_front_shift_1_failure.mp4

R3M

final_cucumber_distractors_bc_r3m_left_front_shift_2_failure.mp4

Our training data and evaluation setups use a variety of initial positions and distractor objects for each target task. Here are some examples of the data diversity in our training data for the "take croissant out of colander" task:

traj0.mp4

traj14.mp4

traj7.mp4

traj19.mp4

Scenario 3: New target objects

We replace the target object in one of the tasks (croissant out of bowl) with various other objects. Our method is still able to achieve some success, while others struggle.

Various objects in colander

V-PTR (ours)

final_colander_targets_ego4d_right_rear_under_carrot_success.mp4

PTR

final_colander_targets_scratch_right_rear_under_carrot_failure.mp4

Masked Visual Pre-training

final_colander_targets_bc_mae_left_front_shift_1_failure.mp4

VIP

final_colander_targets_rl_baselines_vip_left_rear_shift_1_ice_cream_failure.mp4

R3M

final_colander_targets_bc_r3m_left_front_under_failure.mp4

Quantitative

Page updated

Google Sites

Report abuse