Hand Gesture Key Emulation Toolkit (HandGKET)

Summary

This toolkit facilitates integration of hand gesture control with games and PC applications. This middleware recognizes user's hand gestures and generates keyboard or mouse events to control applications in your computer. This toolkit operates on 3D cameras which support the OpenNI framework. This tool is free to use and distribute for noncommercial purposes (for commercial uses, please contact us : smkzzang@gmail.com). This work was built under inspiration of FAAST.
 
Demo

PC control with HandGKET

                                      

Need for Speed - Hot pursuit game

Hardware & Software

 - MS Kinect or ASUS Xtion pro
 - OpenNI & NiTE, Sensor driver, Windows
 
Installation

  - To use HandGKET, you will need to download and install the following software:
    1. OpenNI 2.1 for Windows
    2. PrimeSense NiTE 2.0 for Windows
    3. 
MS Kinect for Windows SDK v1.7 or MS Kinect for Windows Runtime v1.7 

 - Settings
    Uncomment the following file to enable two hands detection (delete the first character, ";" in each line) : 
    C:\Program Files (x86)\PrimeSense\NiTE2\Redist\NiTE2\HandAlgorithms.ini
 (default installation directory, it can be different according to your system or installation)

 - Make sure that examples of NiTE2 run well after the installation.

Gesture Basics

 - HandGKET has six basic gestures for one-hand mode, but you can extend the number of gesture by setting different threshold for the same gesture. Every gesture must turn back to the starting position of the gesture. 


Usage

 - Make sure HandGKET.exe is running and your application that you want to control is topmost position. Then, follow the steps below
    1. Wave or push your hand 
    2. Wait until system ready 
    3. Make a gesture 

 - Information window shows the status of your hand


Configuration file

 - The gesture_event.txt file defines mapping of gesture to key event. Following is an example of the configuration file
 
# GESTURE  TRHESHOLD  EVENT_TYPE  KEY_TO_GENERATE  KEY_TO_RELEASE  PARAM 
TWO_HANDS_PUSH      20    KEY_HOLD    VK_LSHIFT              0       0
ONE_HAND_UP                5    KEY_CLICK    VK_LWIN,VK_UP    0       0
HAND_RIGHT                  25    KEY_CLICK    VK_BACK,X             0  0

   1. when user pushes two hands toward (20cm from the initial pose), the shift key is pressed and hold. The key is released when the gesture is finished.
   2. when user makes an up-gesture, the window key and up key are clicked in mouse mode(press and release).
   3. when user makes an right-gesture, the backspace key and ‘x’ key are clicked in keyboard mode(press and release).

Syntax

GESTURE  TRHESHOLD  EVENT_TYPE  KEY_TO_GENERATE  KEY_TO_RELEASE  PARAM
    GESTURE          : the name of hand gesture
    THRESHOLD          : the minimum threshold for activation (real distance in centimeter) 
    EVENT_TYPE  : the type of event (KEY_CLICK, KEY_TOGGLE etc.)
    KEY_TO_GENERATE  : the specific key to be generated
    KEY_TO_RELEASE  : the specific key to be released (experimental)
    PARAM          : this varies by KEY

GESTURE
- Two hand gestures
TWO_HANDS_PUSH
TWO_HANDS_BACK
TWO_HANDS_UP
TWO_HANDS_DOWN
TWO_HANDS_OPEN
TWO_HANDS_CLOSE
LEFT_HAND_PUSH
RIGHT_HAND_PUSH
TURN_LEFT
TURN_RIGHT

- One hand gestures for keyboard mode
HAND_PUSH
HAND_BACK
HAND_LEFT
HAND_RIGHT
HAND_UP
HAND_DOWN

- One hand gestures for mouse mode
HAND_MOVE : It is used for system. Don’t change the parameters of this line
HAND_HOLD : It is used for system. Don’t change the parameters of this line except THRESHOLD. The threshold in millisecond means the waiting time to start gesture for mouse button.
ONE_HAND_PUSH
ONE_HAND_BACK
ONE_HAND_LEFT
ONE_HAND_RIGHT
ONE_HAND_UP
ONE_HAND_DOWN

EVENT_TYPE

- keyboard event type
KEY_CLICK : A key is pressed and released when a gesture is performed
KEY_TOGGLE : A key is pressed when a gesture is performed first and the key is release when the gesture is performed again
KEY_MULTI_CLICK : A key is clicked continuously while a gesture is on state, which means that your hand is away from the reference position. (PARAM : delay time(ms) of each click) (only supported in two-hand mode)
KEY_HOLD : A key is pressed when a gesture is on state. The key is released when the gesture is finished, which means that all your hands returned back at the reference position. (only supported in two-hand mode)

- mouse event type 
MOUSE_CLICK : A mouse click event is generated when a gesture is performed
MOUSE_DOUBLE_CLICK : A mouse double click event is generated when a gesture is performed
MOUSE_TOGGLE : A mouse button is pressed when a gesture is performed and the mouse button is release when the gesture is performed again. 
MOUSE_MOVE : A mouse point is moving according to your hand movement (used for system)

KEY_TO_GENERATE, KEY_TO_RELEASE   
- keyboard keys
VK_UP, VK_DOWN, VK_LEFT, VK_RIGHT, VK_CONTROL, VK_LCONTROL, VK_RCONTROL, 
VK_RETURN, VK_ESCAPE, VK_SPACE, VK_SHIFT, VK_LSHIFT, VK_RSHIFT, VK_BACK
VK_TAB, VK_END, VK_HOME, VK_INSERT, VK_DELETE, VK_ADD, VK_SUBTRACK
VK_F1 ~ VKF12
a ~ z, A ~ Z, 0 ~ 1, etc.

- mouse keys
M_MOVE, M_NOMOVE (used for system)
M_LBUTTON, M_RBUTTON, M_MBUTTON

- system keys
CHANGE_MODE :This change event mode. If you are in keyboard mode, it will change the mode to mouse mode and if you are in mouse mode, it will change the mode to keyboard mode.

Known issues

 - Keyboard event could not be generated on some game using DirectX.

Release Notes

 - HandGKET v0.9.2 (2013.4, latest version)
  . working on OpenNI v2.1 and NiTE v2.0
  . wave or push for hand detection

 - HandGKET v0.9 (2013.2)
  . working on OpenNI v1.5 and NITE v1.5
  . supports mouse event
  . raise hand for hand detection

 - HandGKET v0.1 (2011.4)
  . working on OpenNI v1.0 and NITE v1.3
  . supports keyboard event

Contacts
 - For questions, comments and any kind of feedback please send an e-mail to smkzzang@gmail.com
 - This toolkit was developed by Myoung-Kyu Sohn, Sang-Heon Lee, Dong-Ju Kim and Hyunduk Kim at DGIST
Ċ
Myoung-Kyu Sohn,
Apr 4, 2013, 10:50 PM
ċ
HandGKET_v0.9.2(Distribution).zip
(5316k)
Myoung-Kyu Sohn,
Apr 4, 2013, 10:51 PM