Yen Na Yum1, Jeff Cheng2, Rosanna Yuen-Yan Chan3, Kai Yuen Tsang4, Sze Wing Adeline Kuo4

1Department of Special Education and Counseling, The Education University of Hong Kong; 2Popsand Technologies Co. Ltd.; 3Department of Information Engineering, Chinese University of Hong Kong; 4Caritas Resurrection School

Introduction

Effective language learning is essential for students with intellectual disabilities because language skills directly impact learning motivation and social interactions. The applications of robotics in the education of students with special needs is growing in recent years [1], and it brings new opportunities in language teaching and learning by incorporating elements of interactivity and adaptive coaching. Using a robot in the classroom may enhance the novelty and multisensory nature of the materials, while providing consistent practice and and feedback to students on an individual basis. However, it is often difficult to know the engagement and acceptance level of new technology for students with more severe intellectual disabilities. Observational data from videos and physiological data from wearable devices may be more reliable and accurate than verbal self-reports.

In this pilot study, a within-subject experimental design was used to evaluate the learning engagement of students with moderate intellectual disabilities and limited verbal abilities when learning Chinese characters under two conditions. Participants learned the same set of characters on Talkbo, a robot designed for language learning, and on tablet computer.

Study objectives

1. To quantify learning engagement in students with moderate intellectual disabilities by using observer coding of eye gaze and arm/hand movements and physiological signals;

2. To explore the inter-relationships of these metrics;

3. To compare the effects of robot compared to more familiar digital devices on the learning engagement measures.

Methods

Participants were 7 male students (15 years old) in a local special school for students with moderate intellectual disabilities. In the testing session, each student individually completed 6 to 7 minutes of assessment tasks on Talkbo and on a tablet computer, with the order counter-balanced between participants.

Students behaviors were video-taped and coded offline in 1 second interval by two raters. The coding scheme consisted of gaze and arm/hand behavior (towards the device; towards the experimenter; non-directed; and not visible) [2]. Participants wore an Empatica E4 wristband to measure their heartrates at 1Hz sampling rate.

Results and Discussions

Statistical results on the Wilcoxon rank tests showed no difference in behavioral engagement with Talkbo or the tablet (gaze or arm/hand movements towards the device, p = .469 and p = .235), nor in heartrates under the two conditions (p = .688). Students moved around significantly more when interacting with Talkbo than the tablet (both on-task and off=task), leading to more time where gaze was not visible (p = .015). Since Talkbo was unfamiliar to the students, they might be reacting to unexpected sounds or movements. An interesting correlation between arm/hand movement towards the tablet and gaze towards Talkbo (r = .79, p =. 036) suggested skill transfer, in that students who were better at using a familiar technology had better visual attention towards a novel technology.

This is a preliminary report with a small sample size and a single time point. Students may show different behaviors with more extended use of Talkbo as a learning tool and more experience with wearable sensors. The testing sessions were kept short so that students were not over-aroused, and thus some physiological measures that required longer coupling time could not be reliably measured. Nevertheless, we believe that robotics is a potential avenue to improve student engagement in the classroom and that further studies may guide its design and implementation.

References

1. Conti, D., Di Nuovo, S., Buono, S., Trubia, G., & Di Nuovo, A. (2015, August ). Use of robotics to stimulate imitation in children with autism spectrum disorder: A pilot study in a clinical setting. In 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN) (pp. 1-6). IEEE.

2. van Otterdijk, M. T., de Korte, M. W., Berk-Smeekens, I. V. D., Hendrix, J., Dongen-Boomsma, M. V., Boer, J. C. D.,.....& Barakova, E. I. (2020). The effects of long-term child-robot interaction on the attention and the engagement of children with autism. Robotics, 9(4), 79.