Motion-Based Game for Children with Autism

Design, development, and in-the-wild deployment study of a Kinect game for scaffolding social skills and movement for children with autism in a classroom setting.

Published in the conference of Interaction Design & Children (IDC) 2015: Full paper PDF

Research team

Arpita Bhattacharya, Mirko Gelsomini, Patricia PérezFuster, Gregory D. Abowd, Agata Rozga

My role

I collaboratively worked on prototyping the Kinect game with lead developer Mirko, worked with teachers to understand requirements of the game, designed and set up the field deployment study in classrooms, led and conducted video data analysis with my team.

Motivation

Games to support social and motor skills have been found to be helpful for children with autism in therapeutic settings and in lab studies. We designed, developed, and studied the impact of using embodied motion-based games in real world classroom settings. We created both one and two player modes to facilitate peer interactions and iterated on customizing the game depending on diverse needs of the children.

Design

We iteratively prototyped the design of Kinect games by using User Centered Design Processes for 9 months with teachers and occupational therapists at a private school in Atlanta region for children with autism. There are two modes of representation of the children in the screen -- 2D skeleton and live green screen representation of the child.

Image on the left shows the 2D skeleton representation and image on right shows live screen representation of players in two player mode. The goal of the players is to work together to "capture" the same object on the screen to score points. We included six different themes.

Storytelling application: When the player flaps their hands, the owl flies into the mitten. (Adapted for educational purposes from The Mitten by Jan Brett)

Based on the curriculum of the class, we also built a Kinect-based digital story telling application. We adapted stories that the teachers were teaching in class which required the children to do simple and repetitive movements such as flying, jumping, and clapping.

Technology

The front-end application was a web application designed using HTML, CSS, Javascript, and JQuery. The Kinect detected joints of the player using a C# application which sent the joint coordinates to the respective 2D skeleton on the web application in real-time using socket communication.

Field deployment study

In months 8 and 9, we let the school teachers run these game sessions in their classrooms as they deemed fit their schedule in the classroom, either individually or in collaborative mode. We asked the teachers to video record the students' game playing session. Video recordings of game play sessions included eighteen students from multiple classes between the ages of 8 and 19 years. We obtained 108 sessions of video recording which we sampled and analyzed using qualitative coding.

Classroom set up: Red circle shows placement of the Kinect to detect the playing children, orange circle shows the placement of the video camera in the classroom.

Working closely with the classrooms helped us to customize the design. The students found the games engaging and it motivated them to use wider range of movements than they usually did without the games. The teachers' role in facilitating the process was crucial. We saw that while only up to two players could play the game with the Kinect, the teachers let the students in the back of the class also participate as audience which led to engaging interactions.