Inclusive Sign Langauge Two Way Communication Platform for Nepali Sign Language
Inclusive Sign Langauge Two Way Communication Platform for Nepali Sign Language
Problem
More than 100K people in Nepal have hearing and speaking disability (NDFN, 2022), while there are nearly 400M+ people globally with the same challenge (WFD, 2018). Their primary source of communication is sign language gestures. There lies a huge communication gap as most of us do not understand sign language, and it isn't feasible to have a translator everywhere. Because of this, people with such disability are held back from many opportunities. Also, any AI research and models developed to this day focus on some basic gestures, lack diversity in datasets, and very little to no work has been done for Nepali Sign Language.
Research Aim
Our research aims to develop a two-way communication platform between sign language users and the audible community. This AI-based system will be able to translate the gestures to respective audio and transform the audio to mapped gestures in real time, thus establishing a smooth communication channel. We aim to study all the natures of sign language. static/dynamic, and gradually implement the model to multiple sign language variations globally.
Current Stage of the research:
Currently, we have trained a hybrid classical-quantum deep learning model for 6 Nepali sign language gestures, "आज", "धन्यबाद", "घर", "जान्छु", "म", "नमस्कार", manually collecting datasets from 3 deaf schools in Kathmandu, Bhaktapur and Banepa. We used Pennylane to simulate the quantum environment implementing "Quantum Variationa; Circuit" inside VGG-16 architecture using Pytorch. We have developed a web-based interface using Flask framework for real-time conversion of these 6 gestures into their respective Nepali audio.
Outcomes so far:
We published a paper titled "Finger Spelling Gesture Recognition for Nepali Sign Language Using Hybrid Classical Quantum Deep Learning Model" at ICE-CEEL 2023.
Top 12 finalists at ICT Awards 2023, product category.
Relevant Links