AirMouse
If you are (or had been) visiting this website on a computer, you are using some sort of device to move your mouse around (trackpad, mouse, etc.). In this project, we created a novel way of interacting with your computer's mouse cursor.
Using an Android smartphone and a webcam we made an intuitive motion input device and also a nice arcade-ish game to show off some of the capabilities.
Check out the video at the bottom of the page for an idea of what I am talking about!
AirMouse
The idea behind AirMouse is to use the IMU in the phone along with the webcam to localize the phone in the space in front of the screen. Two added degrees of freedom in the form of yaw and tilt (rotations) of the phone also move the the on screen cursor.
The app running on the phone gives controls to click, scroll and navigate on the page. You can switch to a keyboard to type as well. There is even a version where the phone can act as a trackpad as well.
The phone-side UDP Client with the mouse controls
Kalman Filter
As almost anyone who has worked with IMUs and tried spatial localization, after failing to spatially localize the phone by integrating the acceleration from the IMU twice, I used a Kalman Filter for localization.
The Kalman Filter fused the position measurements from the webcam with the acceleration from the IMU as the input to the standard constant acceleration point kinematic model of the system. The desktop client ran OpenCV to track the center of the phone and obtained the IMU readings over UDP.
The figure on the left shows the filtered and raw (OpenCV only) cursor positions on a screen. As the IMU uptades are much faster than those from the webcam, the filtered signal also has more points. The figure on the right shows the evolution of the state variable (horizontal position) with the measurement with time.
Savior
To show off the added degrees of freedom the user has for input to the computer we built an arcade-ish game called Savior, it is best you have a look at the gameplay in the video below!
It is a fun game where you need to fight off the orange aliens that appear on the screen. Your phone is the controller and gives 2D mobility as well as full rotation capability to the player.
In case you are wondering, no, we are not going to release a skin pack for this game in the future :)
Short Video Demonstration of AirMouse
If you have a problem with the funky EDM music, please take it up with Karthikeyan. I can put you in touch with him!
AirMouse_demo_video.mp4All figures are mine.