Hand Interfaces (CHI'22)

With the power of hand tracking, many modern applications operate under the paradigm of grasping and manipulating an object virtually as if it were physically in your hand. In the case of this project, we explored the idea of allowing our hands to become the object. 

For example, the joystick is a well-known controller type, composed of a rod protruding from a rotating base. We noticed that the physical and kinematic characteristics of a joystick very closely match that of the human thumb. Both have a rod of sorts connected to a ball-socket joint, and both can rotate within a range of motion using that joint. This would form the basis of our project. 

Our project allowed users to use a hand pose to retrieve the desired controller, and once retrieved, they would be able to interact using a specific gesture. For example, creating two O's using both hands and pressing them together would retrieve a binocular. Once retrieved, users could life their hands towards their eyes, as if using real binoculars, to engage the telescoping effect. 

As our design process matured, we created 28 unique poses and gestures, collectively called Hand Interfaces. You can see some of our designs below. 

I specifically helped with hand rigging and analyzing the kinematic behaviors of the human hand, applying posing constraints to the hand model to match that of real hands to avoid discomfort. I was also in charge of creating the website for the project. 

Once we completed implementation of the project, we ran a user study consisting of 17 participants in order to gain some feedback on these techniques. To further facilitate feedback, we created a website in hopes of keeping the project Open Source. Thankfully, after six months of hard work, this paper was accepted for publication and was presented at CHI'22. 

Acknowledgements:

This project was my first successful publication and I give thanks to my colleagues Siyou Pei and Jaewook Lee, as well as my advisor Professor Yang Zhang.