Kinect with signs
This research is exploring how to teach physical skills using the Microsoft Xbox and Kinect focusing specifically on teaching sign language to parents, carers and teachers working with deaf and autistic children. My Interactive Auslan Coach (MIAC) is a system developed help people to learn Australian Sign Language (Auslan) using mainstream gaming technology. It is impossible to learn sign language from a book because of the kinetic nature of the language. There have been a few resources developed to teach sign language using computers but they cannot provide feedback on if the learner is making the sign correctly. This can lead to significant problems as a user may be learning a sign incorrectly and reinforcing the error. MIAC shows the user the correct way to make a sign from several viewpoints, which is a unique aspect of the design. MIAC then uses a Kinect sensor to track a user’s movements and compares this with the ideal model of the sign. The system can provide feedback to people on their errors making it possible for learners to fix incorrect aspects of the sign such as hand shapes and movements. The system use Dynamic time warping to solve the movement over time issues and swarming to identify hand shapes.
Collaborators
Project Lead
- Dr Kirsten Ellis
Project Team
- Dr Louisa Willoughby
- Professor Julie Fisher
- Professor Nicole Rinehart
- Mr Ray Neil
Project Manager
- Dr Jan Carlo Barca
Programmers
- Mr Daniel Waghorn
- Mr Samuel James Trolland
- Ms Shuang Yu
- Mr Whye Leon Seng
- Ms Mudiyanselage Dh Senanayake
- Mr Han Duy Phan
Graphic Design
- Ms Carol Merlo
Auslan Research
- Ms Stephanie Linder
Quality Assurance
- Mr Bob Jacobs
Funding
ARC Linkage Acquiring Physical Skills: Exploiting Games Technology to Teach Sign Language.Investigators Ellis, K, Willoughby, L, Fisher, J, Rinehart, N, Ray, N (2011 - 2014). Australian Research Council, Deaf Children Australia, Victorian Deaf Society
Links
Publications
Ellis, K.A., Fisher, J.L., Willoughby, L.J.V., Barca, J.C., 2015, A design science exploration of a visual-spatial learning system with feedback, Proceedings of the 26th Australasian Conference on Information Systems (ACIS 2015), 30 November 2015 to 04 December 2015, Australasian Conference on Information Systems, Adelaide SA Australia, pp. 1-13. View Publication
Fisher, J.L., Ellis, K.A., Willoughby, L.J.V., Barca, J.C., 2014, Taking a user centred design approach for designing a system to teach sign language, Proceedings of the 25th Australasian Conference on Information Systems (ACIS 2014), 08 December 2014 to 10 December 2014, Australasian Conference on Information Systems, Auckland New Zealand, pp. 1-10. View Publication
Willoughby, L.J.V., Linder, S., Ellis, K.A., Fisher, J.L., 2015, Errors and feedback in the beginner Auslan classroom, Sign Language Studies [P], vol 15, issue 3, Gallaudet University Press, Washington DC USA, pp. 322-347. View Publication