SCONE: A Food Scooping Robot Learning Framework with Active Perception

The ability to successfully scoop up food items presents a significant challenge for existing robot systems due to the complex states and physical properties of food, including deformability, fragility, fluidity, or granularity, pose significant challenges for existing representations. In this paper, we investigate the potential of active perception for learning meaningful food representations in an implicit manner to improve close-loop robot policy learning. We present SCONE, a food SCOoping robot learNing framEwork that leverages the representations gained from active perception to provide the food-scooping model. 

SCONE utilize active perception to improve the target manipulation tasks in a coarse plus fine manner,  consisting of three main feature extraction modules: 

(a) Global temporal encoder, which is able to capture the overall environment information from the current observations. 

(b) Interactive encoder (coarse), which effectively gathers information and integrates it to form a holistic understanding of the physical properties of the food items. 

(c) State retrieval module (fine), which explores and extracts the task-related state information based on current state to facilitate manipulation. 

Visualization

Real World Experiment

Experiment 1: Single Scooping

The robot must scoop up food items and transfer them to the target container without causing any spillage or damaging the food in a single attempt. Overall, our proposed method achieves a task success rate of 71% across three difficulty levels, outperforming other baselines. 

Experiment 2: Continuous Scooping

The robot needs to repetitively transfer food from one bowl to another until it fails (scoops up nothing). From the results, our framework showcases the ability to successfully scoop food across intricate variations in food states, with higher task success rate and the lower proportion of food/setting spillage and damage.

Experiment 3: Out-of-Distribusion Food Scooping

To validate the generalization ability of our method, we applied the proposed approach to scoop up the black tea and the pudding. This task is highly challenging as our training data did not include any liquids or semi-solid foods. From the results, he proposed method can generalize to unseen foods and achieve a success rate of over 50%.