Evaluation Results
Metrics
CRA:
99.13% LLMNode prediction accuracy for several natural language commands and tasks, < 1% confusion.
NSR:
97.96% accuracy in the REM’s ability to abstract the high-level understanding from the LLMNode to the actual robot’s navigation actions.
OIA:
55.20% CLIPNode accuracy in identifying and localizing objects within the robot’s task environment.
Participants:
21 with an average age of 23 (±5).
gender distribution, 61.9% male, 28.6% female, and 9.5% others.
Ratings:
scale of 1 (lowest) - 5 (highest) with 4 and 5 as favourable benchmarks.
80.9% and 76.2% respectively rated the ease of interaction and the intuitiveness of our framework as favourable.
85.7% are satisfied with the response of the robot to their natural language commands.
ART:
0.45s from receiving the user’s chat command to initiating the robot’s movement.