In-Mouth Robotic Bite Transfer

With Visual and Haptic Sensing

Lorenzo Shaikewitz*, Yilin Wu*, Suneel Belkhale*, Jennifer Grannen, Priya Sundaresan, Dorsa Sadigh

ICRA 2023


Paper,    Video,    Appendix

Abstract

Assistance during eating is essential for those with severe mobility issues or eating risks. However, dependence on traditional human caregivers is linked to malnutrition, weight loss, and low self-esteem. For those who require eating assistance, a semi-autonomous robotic platform can provide independence and a healthier lifestyle. We demonstrate an essential capability of this platform: safe, comfortable, and effective transfer of a bite-sized food item from a utensil directly to the inside of a person’s mouth. Our system uses a force-reactive controller to safely accommodate the user’s motions throughout the transfer, allowing full reactivity until bite detection then reducing reactivity in the direction of exit. Additionally, we introduce a novel dexterous wrist-like end effector capable of small, unimposing movements to reduce user discomfort. We conduct a user study with 11 participants covering 8 diverse food categories to evaluate our system end-to-end, and we find that users strongly prefer our method to a wide range of baselines.



Supplementary Video

Wrist Mount

Our custom wrist mount is designed to work with a Franka Emika Panda robot end effector. Here's what you need to build your own:

Sample Videos (Full Videos Link)

Human Oracle involves the user being fed by another human, which serves as a reasonable upper bound for our robotic feeding system.

tony_pineapple_human_1.MOV

pineapple chunk

tony_tomato_human_1.MOV

cherry tomato

20_broccoli_human_trial_1.MOV

broccoli piece

20_tofu_human_trial_1.MOV

tofu chunk

Fixed Pose is not an in-mouth bite-transfer method from prior work. Instead, the robot moves the fork to a position outside the user’s mouth and allows the user to take a bite on their own.

20_cheesecake_fixed_trial_1.MOV

cheesecake

20_broccli_fixed_trial_2.MOV

broccoli piece

5_tomato_fixed_trial_2.MOV

cherry tomato

5_blueberry_fixed_trial_2.MOV

blueberry

 Our Method involves a novel wrist-like end-effector, capable of small, unimposing movements to reduce the user discomfort and a phased force-reactive controller to safely accommodate users' motions. It is substantially preferred over baselines across 11 participants and 8 different bite-sized food items in the user study.

carrot_our.mov

carrot

pineapple_our_1.mov

pineapple chunk

5_tofu_our_trial_2.MOV

tofu chunk

strawberry_our_1.mov

strawberry

Less Reactive is an ablation of our system, which uses less force reactivity throughout the whole feeding process.

blueberry_less_reactive_1.mov

blueberry

strawberry_less_reactive_1.mov

strawberry

pineapple_less_reactive_1.mov

pineapple chunk

20_cheesecake_less_trial_1.MOV

cheesecake

More Reactive is an ablation of our system, which uses more force reactivity throughout the feeding process and is representative to the in-mouth bite transfer work[1].

20_broccoli_more_trial_1.MOV

broccoli piece

carrot_more_reactive_1.mov

carrot

strawberry_more_fail_to_bite.mov

strawberry

tofu_more_reactive_1.mov

tofu chunk

Failure Modes

strawberry_more_fail_to_bite_2.mov

Bite Failure

In Bite Failure, users fail to take a bite of the food. This can occur when either the user retracts early due to mistrust or fails to take a bite before the fork exits.

carrot_less_inaccurate_location.mov

Imprecise

Imprecise denotes the face detector localizing the mouth inaccurately, causing a near-miss or off-center transfer.

carrot_fall_our_short.mov

Drop

In Drop, the food falls off the fork during the transfer process.