Robosuite Experiments

On the left a human generates successful demonstrations that start in various states and take various paths to the goal state. They share an implicit discrete plan of mode sequence: Reaching -> Scooping -> Transporting -> Dropping. Scooping mode is defined by the spoon making contact with any marbles; transporting mode is defined as the spoon holding at least one marble; dropping mode is defined as the spoon reaching above the target bowl location such that rotating the spoon will drop the marbles into the target bowl; everything else is in the reaching mode. The demonstration we collected consists of robot end-effector x, y, z, quaternion, robot joint states, and a mask of the spoon from the wrist camera view as shown above on the right. In particular, we prompt GPT4-Vision (vision language foundation model) to inform the system that tracking if there are marbles on the spoon is critical for this task. The labels "Spoon" and "Red Marbles" are given to SAM (segment everything - vision foundation model) to segment the spoon mask. Note that the mask is a complete ellipse when the spoon is empty. Otherwise, it is a partial ellipse due to marble occlusions. We use this binary mask as the state representation for the wrist camera. 

On the left a human demonstrates various ways perturbations can be added to fail a successful scooping trajectory, which we use as counterfactuals to the successful demonstrations shown above. On the right the corresponding wrist camera view is shown. Note that the labels "No Marble" and "Has Marble" is not given to the learning system and are only displayed for visualization purposes.

We show that without mode abstraction, the robot cannot reason that it is in a different mode when perturbations cause the scoop to drop all marbles. Consequently, the task execution fails even when motion imitation is successful.

We show that the learned grounding classifier allows the system to map from pixel observations to a discrete symbolic space where LLM can replan when perturbations derail the original plan that's been demonstrated.

We prompt LLM to generate a subset of features relevant to predicting task success: X, and Y locations of the robot end-effector in the robot base frame as well as the wrist camera mask. Due to a lack of contact sensors, we omit the scooping mode, and prompt the LLM to generate a plan: Reaching -> Transporting -> Dropping (assuming scooping is always successful when transitioning from the reaching mode to the transporting mode. The corresponding feasibility matrix is F3. In the top figure, we plot demonstration data in X and Y and use the color of the scattered plot to indicate ground truth modes (reaching is red,  transporting is green, and dropping is blue). Examples of logged spoon masks along these trajectories are shown at the top. At the bottom, we visualize the learned classifier, which has correctly learned three modes (indicated by three distinct colors) by partitioning the space according to X and Y locations as well as the masks. Note that (1) while the shape of the learned dropping mode is not perfect, which should be improved as we collect more demonstrations and counterfactuals, the location of the mode corresponds to that of the dropping bowl location; (2) the classifier successfully learns the threshold function that turns the continuous mask values to the discrete information that differentiates reaching mode from transporting mode given the same X, Y locations.