The IDIAL evaluation procedure is defined to address the following three characteristics of a task oriented-conversational agent:
A. Task completion: this is the capacity of the system to achieve the goals of the task for which the system has been designed, in a reasonable amount of time.
B. Effectiveness of the dialogue: this is the capacity of the system to interact with the user in order to accomplish its task. It includes, among the others, the capacity to interpret commands accurately, the robustness of the system to unexpected input, the ease of use of the system, and the fluency of the dialogue.
C. User satisfaction: this is the reaction of the user after having used the system. It includes aspects like the degree of empathy of the system, the ability to read and respond to moods of human participant, the capacity of the system to give conversational cues, and the use appropriate degrees of formality.
The three characteristics (A-C) mentioned above are assessed in IDIAL by means of two evaluation methods, a questionnaire, and a set of linguistic stress tests.
Evaluation Protocol: Download PDF