Using Fitts' Law to Benchmark Assisted Human-Robot Performance

Jiahe Pan, Jonathan Eden, Denny Oetomo, Wafa Johal

Faculty of Engineering and IT, The University of Melbourne

Abstract

Shared control systems aim to combine human and robot abilities to improve task performance. However, achieving optimal performance requires that the robot's level of assistance adjusts the operator's cognitive workload in response to the task difficulty. Understanding and dynamically adjusting this balance is crucial to maximizing efficiency and user satisfaction. In this paper, we propose a novel benchmarking method for shared control systems based on Fitts' Law to formally parameterize the difficulty level of a target-reaching task. With this we systematically quantify and model the effect of task difficulty (i.e. size and distance of target) and robot autonomy on task performance and operators' cognitive load and trust levels. Our empirical results (N=24) not only show that both task difficulty and robot autonomy influence task performance, but also that the performance can be modelled using these parameters, which may allow for the generalization of this relationship across more diverse setups. We also found that the users' perceived cognitive load and trust were influenced by these factors. Given the challenges in directly measuring cognitive load in real-time, our adapted Fitts' model presents a potential alternative approach to estimate cognitive load through determining the difficulty level of the task, with the assumption that greater task difficulty results in higher cognitive load levels. We hope that these insights and our proposed framework inspire future works to further investigate the generalizability of the method, ultimately enabling the benchmarking and systematic assessment of shared control quality and user impact, which will aid in the development of more effective and adaptable systems.

Experimental Setup

Novint Falcon haptic controller (3-DOF)
Franka Emika Research 3 Robotic Arm (7-DOF)
Virtual Rendering + Point Cloud Visualization (RViz)
Tobii Pro Spark Screen-Based Eye-Tracker

Fitts' Law

Fitts' Law is a widely used human performance model that has been applied to evaluate control interface design in HCI studies. The original formulation predicts the movement time (MT) to reach a target using an index of difficulty (ID) computed from the movement amplitude (A) and target width (W).

The target ring geometry

R = Ring radius
W = Target width
A = Movement amplitude
Theta = Angle between consecutive targets around the ring

A total of 4 rings were generated, corresponding to all combinations of 2 different amplitudes (A) and 2 widths (W).

Reaching sequence around the ring

The task entails sequentially reaching a ring of 9 circular targets (in the sequence shown above). A total of 8 linear movements between consecutive targets are parameterized using Fitts' Law and their completion times recorded.

Participants were instructed to "as quickly as possible" complete the ring without making sudden jerky movements during teleoperation which may damage the robot.

Cognitive Load

The NASA-TLX questionnaire has been widely used to capture cognitive load information, including in HRI and robot teleoperation settings
Pupil Dilation has been associated with more intense cognitive processing

Trust

The MDMT questionnaire is composed of 8 sub-scales in each of the Capacity Trust and Moral Trust categories, and has been widely employed in HRI studies to capture trust information.
Single-Scale Trust question: "How much do you trust the robot?" reported using a 10-point discrete scale.

Experimental Procedure

Each session had two phases: Training and Main.
Within each phase, participants completed multiple trials. Within each phase, there was a total of 4 rings * 3 autonomy levels = 12 trials.
During each trials, the task performance and pupil diameter data were continuously recorded, while participants' self-reported their cognitive load and trust after each trial via a questionnaire.

Results

We performed two-way mixed ANOVAs with autonomy and ring number as the within-subject variables for each measure. Results were considered significant at the threshold α < 0.05. The plots below show the distribution of each measure against ring number, grouped by autonomy. The results from post-hoc pairwise comparisons with Holm-Bonferroni adjustment are also illustrated and their significance levels labelled.

Movement Time

The interaction suggests that while Fitts' Law holds under the no autonomy condition, higher levels of autonomy impacts the effect of Fitts' Index of Difficulty on movement time. This motivates an adapted Fitts' Law which accounts for the level of robot autonomy.

Averaged NASA-TLX

In general, the perceived cognitive load was higher with greater Fitts' Difficulty and lower under higher autonomy levels. However, from the results it is worth noting that under medium autonomy level participants perceived the same level of workload across all difficulty levels.

While a preliminary result that require further examination, this finding may suggest that when there is a change in task difficulty during shared control, a medium level of robot autonomy may be employed to minimize its impact on the user's perceived workload.

Averaged MDMT

In general, subjective trust was higher under higher robot autonomy, and clear differences were not observed across different Fitts' index of difficulty levels.

Demo Video

Page updated

Google Sites

Report abuse