Robot-assisted bed bathing holds the potential to enhance the quality of life for older adults and individuals with mobility impairments. Yet, accurately sensing the human body in such a contact-rich manipulation task remains challenging. To address this challenge, we propose a multimodal sensing approach that perceives the 3D contour of body parts using the visual modality while capturing local contact details using the tactile modality. We employ a Transformer-based imitation learning model to utilize the multimodal information and learn to focus on crucial visuo-tactile task features for action prediction. We demonstrate our approach using a Baxter robot and a medical manikin to simulate the robot-assisted bed bathing scenario with bedridden individuals. The robot adeptly follows the contours of the manikin's body parts and cleans the surface based on its curve. Experimental results show that our method can adapt to nonlinear surface curves and generalize across multiple surface geometries, and to human subjects. Overall, our research presents a promising approach for robots to accurately sense the human body through multimodal sensing and perform safe contact-rich interaction during assistive bed bathing.

The robot collects visual, tactile, and proprioception observations while sliding over the body surface. Our approach first processes the collected observations into a sequence of feature vectors. Then we feed the visual and tactile data into two cross-attention transformer encoders to enable modality interactions. The two streams of cross-attention are further concatenated with proprioception embedding and an extra learnable embedding to classify action outputs. We proceed all features to a standard transformer encoder to model the global context and then output a latent representation passed through a GMM-MLP to generate actions.

Our experiments aim to examine the effectiveness of our proposed method in assistive bed bathing where the robot wipes through the body part's surface with a cleaning cloth. The task requires the robot to maintain continuous contact with the body surface while following the surface contour. We conduct three experiments to investigate: (1) Does visuo-tactile sensing help the robot better perceive the contact-rich bathing environment? (2) How well does Transformer-based learning perform against other baselines in this task? (3) Does the learned policy generalize over body parts variations?