Balancing Efficiency and Comfort in Robot-Assisted Bite-Transfer

Suneel Belkhale, Ethan K Gordon, Tracy Chen, Siddhartha Srinivasa, Tapo Bhattacharjee, Dorsa Sadigh

ICRA 2022

bitetransfer_corl21_anon.mp4

Abstract

Robot-assisted feeding in household environments is challenging because it requires robots to generate trajectories that effectively bring food items of varying shapes and sizes into the mouth while making sure the user is comfortable. Our key insight is that in order to solve this challenge, robots must balance the efficiency of feeding a food item with the comfort of each individual bite. We formalize comfort and efficiency as heuristics to incorporate in motion planning. We present an approach based on heuristics-guided bi-directional Rapidly-exploring Random Trees (h-BiRRT) that selects bite transfer trajectories of arbitrary food item geometries and shapes using our developed bite efficiency and comfort heuristics. Real-robot evaluations show our method that optimizes both comfort and efficiency significantly outperforms a fixed-pose based method, and users preferred our method significantly more than that of a method that maximizes only user comfort. Appendices for the main text can be found here: https://drive.google.com/file/d/1562yKZ8DEoxMqGEWJK7wDD57Gu6dD-yP/view?usp=sharing

We present a motion planning based approach for selecting bite transfer trajectories for a continuous space of mouth sizes, food geometries, and poses. Our algorithm, shown in the figure above, takes in a food geometry mesh and acquisition pose on the fork from the real world, and generates an analogous simulation environment for motion planning. We then guide motion planning based on a set of cost functions in order to shape the perceived comfort and bite volume efficiency of the selected trajectories. We gear this approach towards the multi-bite framing of feeding, recognizing that feeding a bite of food should not sacrifice comfort for the sake of bite completion. Our approach balances perceived comfort and eating efficiency through a novel set of heuristics. To our knowledge, our approach is the first to formulate comfort and efficiency for bite transfer, as well as the first to consider non-bite sized food items.

Our Pybullet-based simulation environment. The inputs for the bite transfer algorithm are shown here

Trajectory Planner

Once collision-free goal food poses have been generated and clustered, we must find trajectories to reach these goal poses. We adapt Rapidly-exploring Random Trees (RRT) for our motion planning. We grow one tree from the start food pose, and one tree from each candidate goal food pose (BiRRT). This growth is shaped by a set of heuristics that balance efficiency and comfort, discussed next. We adapt the hRRT algorithm, which selects nodes for growth in the existing tree based on a measure of quality of each node using the cost so far and the cost to come (heuristic).

Modeling Efficiency

We model efficiency as the highest volume of food inside the mouth. This is measured only at the goal food pose of each trajectory. The most efficient goal pose would be one that brings the most food into a person's mouth. Measurements for efficiency are easy to capture in the real world by comparing the food mesh before and after each bite. However, in simulation, it can be challenging to obtain the updated food mesh after a simulated bite without perfect knowledge of the physics of biting for a given user. Instead, we assume a bite is equivalent to a "slice" in the face plane of the initial food mesh. The ratio of final to initial volume represents how efficient we are in eating this food item at a single bite. This is used as part of the heuristic for the start tree for each goal tree.

Modeling Comfort

Efficiency considers only the food object and its positioning within the mouth, and does not take into account the motion of the end-effector and how this changes the perceived comfort by an end user. We introduce a novel comfort score that penalizes trajectories that encroach on a conical region of "personal" space starting at the mouth. The radii of this cone is smaller in the upward than the downward direction, relative to the person's face.

On the right, we show a visualization of the comfort cost, relative to the location of the mouth at the tip of the cone. Here, red is higher cost and green is lower cost. Note the steeper cost gradient in the upward direction compared to the downward direction. This ensures trajectories near the face have a high comfort cost. A sample "hit point" on the surface of the robot is shown as a red dot. This hit point exists some distance along the mouth axis (z) and some 2D offset perpendicular to it (x). This red point in space would evaluate to a low comfort cost, and this cost is accumulated with the other hit points on the robot's surface to define the overall comfort cost.

User Study

We conducted a user study with feeding a carrot with each of our methods at four different orientations. We were limited in the number of participants we could recruit for this study due to Covid-19 restrictions. For simplicity and consistency across tasks, we restricted the set of food geometries to carrots of varying sizes and a fixed set of orientations. We used four sample food geometries and orientations, visualized in the first row of the figure below.

Our user study validates the perceived safety of our real world evaluation pipeline and demonstrates that our method that considers comfort and efficiency jointly provides significantly more preferable trajectories compared to a fixed pose baseline. Furthermore, our method with comfort and efficiency consistently outperforms when considering only comfort or only efficiency.

Sample feeding videos

User studies and videos recorded with consent under IRB

FixedPose

Comfort+Efficiency

vertical_video_fixed_pose_02-22-21--12-12-PM.mp4

vertical_video_comfort_and_efficiency_02-22-21--12-24-PM.mp4

horizontal_video_comfort_and_efficiency_02-21-21--10-03-PM.mp4

horizontal_video_fixed_pose_02-21-21--09-49-PM.mp4

roll_left_up_video_fixed_pose_02-22-21--02-36-PM.mp4

roll_left_up_video_comfort_and_efficiency_02-22-21--02-41-PM.mp4

yaw_left_forward_video_fixed_pose_02-25-21--07-28-PM.mp4

yaw_left_forward_video_comfort_and_efficiency_02-25-21--07-32-PM.mp4