Emotion Detection & Electrodermal Activity (EDA)

Effect of Time Delay on Human Emotions

Objective and Hypothesis

The objective of this study was to establish the effects of time delay on human emotion, physiological signals, and user satisfaction. Time delay of system is expected to result in increased negative emotions, increased arousal, and decreased user satisfaction for tasks of both low and high difficulty.

Method

Participants were asked to remote-control a robot vehicle to navigate different mazes in a remote location. Time delay was manipulated by introducing lags into system feedback. Subjective and objective measures included emotion tracking through face recognition, and electrodermal activity (EDA).

Result

User frustration, anger, and arousal increased while user satisfaction decreased. A better understanding of how time delay influences user’s emotion and how change in emotion is expressed in physiological signals would be of crucial importance to designing an affect-aware robotic systems that have the ability to appropriately respond to user emotional state.

Publication

• Yang, E. & Dorneich, M. C. (2017) The Emotional, Cognitive, Physiological, and Performance Effects of Variable Time Delay in Robotic Teleoperation. International Journal of Social Robotics. 9(4), 491–508.
• Yang, E. (2016). Mitigating user frustration through adaptive feedback based on human automation etiquette strategies, Doctoral dissertation, Iowa State University.
• Yang, E. & Dorneich, M. C. (2015) The Effect of Time Delay on Emotion, Arousal, and Satisfaction in Human-Robot Interaction. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 59, No. 1, pp. 443-447). SAGE Publications.

Emotion Detection & Noldus FaceReader

Students' Negative Emotions

Human Frustration in Human-Computer Interaction (HCI)

User frustration in HCI indicates a negative feeling that occurs when efforts to achieve a goal are impeded. User frustration impacts not only the communication with the computer itself, but also productivity, learning, and cognitive workload. Affect-aware systems have been studied to recognize user emotions and respond in different ways. Affect-aware systems need to be adaptive systems that change their behavior depending on users’ emotions. Adaptive systems have four categories of adaptations. Previous research has focused on primarily function allocation and to a lesser extent information content and task scheduling. However, the fourth approach, changing the interaction styles is the least explored because of the interplay of human factors considerations.

Emotion Detection by Using Noldus FaceReader

FaceReader is the complete facial expression analysis software. More than 10,000 manually annotated images used for training the software. Objectivity in observations and accurate modeling of the face by describing 500 key points. FaceReader automatically analyzes 6 basic facial expressions, neutral, contempt, and boredom, interest, and confusion. It also calculates gaze direction, head orientation, and person characteristics. Plus 20 Action Units. The Project Analysis Module is ideal for advanced analysis and reporting: you quickly gain insight into the effects of different stimuli.

Research Question

• How does feedback influence user emotions, cognitive workload, performance, and physiological response?
• Does changing the interaction style of feedback by using etiquette strategies mitigate user frustration?
• Does an adaptive system that employs etiquette strategies mitigate user frustration and improve motivation, confidence, satisfaction, and performance?

Conclusion

This work establishes that changing the interaction style alone of a computer tutor can affect a user’s motivation, confidence, satisfaction, and performance. Furthermore, the beneficial effect of changing etiquette strategies is greater when users are frustrated. This work provides a basis for future work to develop affect-aware adaptive systems to mitigate user frustration.

Publication

• Yang, E. & Dorneich, M. C. (2018) Evaluating Human-­Automation Etiquette Strategies to Enhance Learning. Applied Sciences. 8(6), 895.
• Yang, E. & Dorneich, M. C. (2018) Affect-Aware Adaptive Tutoring Based on Human-Automation Etiquette Strategies. Human Factors. 60(4), 510-526.
• Yang, E. & Dorneich, M. C. (2017) The Emotional, Cognitive, Physiological, and Performance Effects of Variable Time Delay in Robotic Teleoperation. International Journal of Social Robotics. 9(4), 491–508.
• Yang, E. (2016). Mitigating user frustration through adaptive feedback based on human automation etiquette strategies, Doctoral dissertation, Iowa State University.
• Yang, E. & Dorneich, M. C. (2016) Evaluation of Etiquette Strategies to Adapt Feedback in Affect-Aware Tutoring. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 60, No. 1, pp. 393-397). SAGE Publications.
• Yang, E. & Dorneich, M. C. (2015) The Effect of Time Delay on Emotion, Arousal, and Satisfaction in Human-Robot Interaction. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 59, No. 1, pp. 443-447). SAGE Publications.
• Iglesias, M., Montealegre, D. A., Yang, E., Zonner, J., & Dorneich, M. (2015) Measuring the Effectiveness of Etiquette Strategies to Mitigate Negative Emotions. Society for Advancement of Chicanos/Hispanics and Native Americans in Science (SACNAS) Conference. Washington D.C., October 29-31.