Real-time hand shape estimation for gesture recognition systems - An HCI device for assisting the differently abled

The project was done during my internship with the MindTree Foundation, a not-for-profit organisation striving for the betterment of the lives of differently-abled people by developing appropriate assistive technologies. The goal of the project was to develop a versatile Human Computer Interfrace (HCI) device for assisting the differently-abled in interacting with a computer. The particular disability targeted was people affected with cerebral palsy. Cerebral palsy is a disease affecting, to varying degrees, the motor conditions most commonly the hand and leg movements.

Alternate Human Computer Interfrace (HCI) devices are a very popular research area. One of the many approaches is that of an instrumented hand glove; in which a glove to be worn on the hand is fitted with a variety of data capturing devices such accelerometers, gyroscopes and bend sensors. The hardware used was very similar to the acceleglove developed by AnthroTronix Inc. A glove was fitted with six tri-axial accelerometers, one behind each finger and one on the back of the palm.

The novelty of the project was in the way the accelerometer signals were processed for recognising gestures. The device was required to be very versatile, in terms of the variety of gestures it could identify, so that the device could be used by people affected with various motor neuron ailments and for a variety of applications.

All gesture recognition systems consist of feature extraction from data followed by machine recognition. Endemic features corresponding to the recognition task are identified and extracted from the data which is fed into a machine recognition engine. Most approaches in gesture recognition have concentrated on identifying a set of features specific to the particular recognition task (for example alphabets of American Sign Language). For developing a versatile device which could be able to recognise and differentiate between a large variety of gestures (and not any specific set of pre-defined gestures for eg. the alphabets of American Sign Language) it was necessary to extract features from the data (in our case accelerations from the fingers and the palm) which would be endemic to all the human gestures and not just for a specific subset of it.

We used finger joint angles and orientation of the palm as endemic features of human gestures. The kinematic relationships of the various finger joints were studied and modelled approximately (eg. revolute joints at the Distal InterPhalangeal and Proximal InterPhalangeal joints). Inverse kinematic analysis was used to calculate a significant subset of the finger joint angles from the accelerometer signals. Only a subset of the finger joint angles and not all of the them could be calculated due to constraints in the number of accelerometers that could be attached on each finger which in turn was chosen so as to make the device affordable. But the method was easily extendable to calculate all the finger joint angles if sufficient number of accelerometers were present.

The video below shows a simple gesture being recognised as a "click" (seen on the laptop screen).