3DMouse - Introduction

Abstract

Gesture based operation has fast gained grounds in recent times. Being one of the fastest growing technologies today, there is strong potential for gesture based operations to become a complementary, if not the preferred mode of human-computer interactions in the near future, as compared to the conventional 2D computer mouse used today. 3D gestures today are being developed primarily for 2D desktop and gaming console operations. A majority of gaming consoles, smart televisions and systems rely on handheld remote tracking devices, whereas desktop applications depend on finger markers, gloves or thimbles, sometimes combined with special glasses to visualize tracking points.

With the increase in the use of gesture based operations, it is important to study gesture based operations. We wanted to examine the possibility of developing a system that would allow for operations in the 3D domain using freehand gestures. In August 2013, a new device called Leap Motion [3] which works using freehand gestures was released commercially. This is a novel concept that will allow for gesture based operations on desktops and personal laptops through a USB connection.

This paper presents our examination of gesture based operations by building a framework using the Leap Motion device and API. We built a Tic-Tac-Toe application with a simple graphical interface using JOGL – the Java for OpenGL library. We defined new and intuitive 3D gestures that help demonstrate this new framework and system. We determined that it is fairly simple to mathematically define simple gestures based on the real-time hand and finger sensing data returned by the Leap Motion tracking device. We determined that there is scope to extend this framework to increase the number of gestures, as well as to make our gestures more intuitive to the user.

Introduction

The journey of the computer mouse began in a post-World War II-era in the form of a trackball. The earliest known publication of the colloquial term “mouse” for the computer aided display control device was in Bill English’s publication in 1965. The device has since been a ubiquitous part of computer systems all over the world. The mouse has seen a number of aesthetic and technological changes, such as the use of infrared signal instead of trackballs, added scroll wheels and buttons, and USB and wireless connectivity. Even so the basic means of operation has remained unchanged in the fact that the mouse represents a cursor on the computer screen which moves in relation to the physical handheld device with interaction with the computer system performed by clicking the buttons and using the scroll wheel on the mouse. The physical handheld device is moved in a 2 dimensional place which corresponds to the 2 dimensional movement of the cursor on the computer screen

In recent years, human-computer-interaction (HCI) is rapidly moving away from the paradigm of using pointing devices such as the conventional mouse. Operating systems are now being built to operate using freehand or stylus oriented touch based operations. Multi-touch interactive products have found commercial success and have been widely adopted in many fields, businesses and day to day activities. We encounter these devices everyday - right from simple commercial credit card payment machines at grocery stores, ATM machines to consoles, phones, tablets and now even wristwatches. Numerous gaming consoles such as the Wii and the PlayStation Move have delved into motion tracking with the use of handheld remote trackers, while the Xbox Kinect utilizes freehand gestures for gaming and menu selection on 2D television screens. Numerous smart televisions are also being built with similar features that would free the end user from being tied to bulky remote devices and to allow for quicker and an increasingly natural way of interaction using freehand gestures.

As can be observed, the conventional 2D mouse is now falling short of satisfying the modern day needs. As Ren and O’Neill [1] state, the most compelling features of touch based systems are their “directness, fluidity and immediacy”. 3D gestures allow for 6 degrees of freedom of operation as opposed to 4 degrees of freedom with the conventional 2D mouse. The user is able to interact with the device quickly and accurately using only direct freehand interaction and without needing to rely on any kind of attached or handheld pointing devices. Gesture based operations are now also finding feet outside of home entertainment systems. The use of this technology has great potential to grow and to be adopted into numerous industries, such as mechanical, medical and manufacturing.

It is important that this technology be extensively studied and understood to make it robust, scalar, intuitive and possibly even as ubiquitous as the 2D conventional mouse has been for the last six decades. In this study we examine how 3 dimensional gestures and a framework that supports and uses these gestures can be built.

Page updated

Google Sites

Report abuse