Research

Multimodal Interaction

We are exploring multimodal interaction techniques suitable for collaborative environments, combining modalities such as gaze, speech, and touch. Our research focuses on real-time multi-user collaboration scenarios, particularly in simulated manufacturing environments, to better understand how multimodal input combinations can enhance communication, coordination, and task performance.

A novel technique that enables a tabletop to function as a touchpad, allowing users to interact using their index finger to overcome the limitations of conventional pinch gestures. By combining this method with gaze input, a multimodal interaction is achieved. To address the inaccuracy of gaze tracking, a manual mode is introduced, allowing users to adjust the cursor position manually. The proposed technique is experimentally compared with the traditional pinch gesture.

We process 2D images into a cubemap format and stitch them together to create a fully immersive 3D road view experience in VR. Unlike conventional 3D road view applications that require complex controller-based operations often difficult for digitally marginalized users our system allows intuitive control through gesture-based interactions modeled on everyday movements. This makes the experience more accessible and user-friendly for all, including those with limited digital proficiency.

We are developing interaction techniques that enable users to collaborate easily and intuitively in real-time multi-user virtual environments for assembly simulations. The goal is to enhance the collaborative experience by applying user-friendly interaction methods. In particular, we focus on designing and implementing effective combinations of modalities such as gesture, gaze, and speech that are well-suited for collaborative tasks in shared virtual spaces.

We estimate upper and lower body motion in VR using only head movement data, eliminating the need for external sensors or trackers. By analyzing the head’s orientation and vertical displacement, our algorithm naturally reproduces body posture in real time. This lightweight, hardware-free approach enables more immersive and accessible avatar interaction, especially in multi-user VR environments and training simulations.

Page updated

Google Sites

Report abuse