Independence in the Home: A Wearable Interface for a Person with Quadriplegia to Teleoperate a Mobile Manipulator

ACM/IEEE International Conference on Human Robot Interaction (HRI) 2024

Best Paper Award in Systems

[arXiv][Video] [Build Instructions and Code]

This paper builds on work previously presented at the International Conference on Robotics and Automation 2023. More information is located on the HAT1 Website.

Project Video

Abstract

Teleoperation of mobile manipulators within a home environment can significantly enhance the independence of individuals with severe motor impairments, allowing them to regain the ability to perform self-care and household tasks. There is a critical need for novel teleoperation interfaces to offer effective alternatives for individuals with impairments who may encounter challenges in using existing interfaces due to physical limitations. In this work, we iterate on one such interface, HAT (Head-Worn Assistive Teleoperation), an inertial-based wearable integrated into any head-worn garment. We evaluate HAT through a 7-day in-home study with Henry Evans, a non-speaking individual with quadriplegia who has participated extensively in assistive robotics studies. We additionally evaluate HAT with a proposed shared control method for mobile manipulators termed Driver Assistance and demonstrate how the interface generalizes to other physical devices and contexts. Our results show that HAT is a strong teleoperation interface across key metrics including efficiency, errors, learning curve, and workload.

Henry, who is a non-speaking individual with quadriplegia, uses HAT, a head-worn assistive interface for mobile manipulators. The first image provides a system overview. The interface records and sends Henry's head orientation angles over Bluetooth to a companion laptop, which computes and transmits actuator velocities to the robot over WiFi. A clicker, plugged into the companion laptop, is used by Henry to switch between modes, distinct operational states of the HAT system. The second and third images show snippets from a task conducted during the study. In the second image, Henry uses the interface to move a blanket below his knees. In the third image, Henry uses HAT to grasp a tissue to wipe his face and throw the tissue in a trash can

Left: Henry wearing the chin strap with the HAT interface embedded. Right: The HAT Bluetooth communication dongle and foam insert with all electronics labelled.

Human Study Task Videos

Blanket+Tissue+Trash

blankettissuetrash_day1_edited_5.5x.mp4

Day 1

Total task time: 16 mins and 30 seconds

blankettissuetrash_day7_edited_5.5x.mp4

Day 7

Total task time: 8 mins and 44 seconds

For the blanket+tissue+trash task, which was conducted every day of the study, there is a noticeable downward trend in task completion times over the week. From day 1 to day 7, there was a 47% (7m 46s) decrease in task time.

Scratching Itch

scratching_itch_day1_edited_4x.mp4

Day 1

Total task time: 8 mins and 49 seconds

scratching_itch_edited_day7_4x.mp4

Day 7

Total task time: 5 mins and 17 seconds

For the scratching itch task, which was conducted on the first day and repeated on the seventh day of the study, there is a clear improvement with a 40% (3m 32s) reduction in task completion time.

Feeding

feeding_edited_day2.mp4

Day 2

Total task time: 9 mins and 8 seconds

feeding_day7_edited_4x.mp4

Day 7

Total task time: 7 mins and 18 seconds

For the feeding task, which was conducted on the first day and repeated on the seventh day of the study, there is a clear improvement with a 20% (1m 50s) reduction in task completion time.

Removing Soiled Towel

removingsoiledtowel_day2_edited_3x.mp4

Day 2

Total task time: 8 mins and 1 second

removingsoiledtowel_day7_edited_3x.mp4

Day 7

Total task time: 4 mins and 26 seconds

For the removing soiled towel task, which was conducted on the first day and repeated on the seventh day of the study, there is a clear improvement with a 45% (3m 35s) reduction in task completion time.

Driver Assistance: Fetching Red Bull

redbull_edited_da_4x.mp4

Driver Assistance

Total task time: 3 mins and 18 seconds

redbull_teleop_edited_4x.mp4

Full Teleoperation

Total task time: 11 mins and 3 seconds

For the fetching Red Bull task, Driver Assistance, our proposed shared control method, reduced task completion time by 70% (7m 45s) and was rated much lower (better) on the NASA TLX scale. Additionally, our participant reported still feeling in control. The median response for the Likert Item "I felt like I was in control while using driver assistance" was a 5.  

Open Tasks 

wipingtable_day2_edited_2x.mp4

Grasping Towel+Wiping Table

Total task time: 5 mins and 46 seconds

openingfridge_edited_4x.mp4

Opening Fridge

Total task time: 9 mins and 18 seconds

Cursor Control: Searching for a Video

cursorcontrol_day3_edited_realtime.mp4

We evaluated the generalizability of HAT to other physical devices such as a computer cursor. Our participant noted that it was "pretty close" in comparison to his current head tracker and advantageous in not requiring frequent calibration.