Abstract : This paper presents MobileBiHap, a real-time interaction framework that transforms commodity smartphones into practical bidirectional haptic controllers for Unity3D-based virtual environments without requiring dedicated hardware. The proposed system captures multi-touch gestures, including pinch, drag, and rotation, and maps them to translation, rotation, and grasp/release operations of a skinned 3D hand model through a lightweight User Datagram Protocol (UDP)-based communication pipeline. To support interaction with reduced object jitter and penetration in the tested cases, convex colliders generated from the hand mesh are used for collision-aware manipulation, while contact events are returned to the smartphone as synchronized vibration and audio feedback. Unlike conventional single-device interaction schemes, MobileBiHap supports coordinated bi-manual manipulation using two smartphones, enabling simultaneous grasping, object transfer, and two-handed manipulation. The main contribution of this work lies in integrating smartphone-based gesture input, Unity-side virtual-hand control, collision-aware physics interaction, bidirectional haptic/audio feedback, and bi-manual coordination into a unified real-time closed-loop framework. We evaluate the proposed framework using both engineering metrics and a structured participant study, including end-to-end latency, frame rate stability, packet rate, grasp success rate, manipulation stability, System Usability Scale (SUS), National Aeronautics and Space Administration Task Load Index (NASA-TLX), task completion time, interaction errors, perceived control, and user preference. The results show an end-to-end latency of 28–45 ms, a frame rate of 58–72 frames per second (FPS), and a packet rate of 55–60 Hz under a local wireless environment. A structured participant study further yielded a SUS score of 78.6, a NASA-TLX score of 38.4, a perceived-control score of 4.1/5, and a user-preference score of 4.2/5, suggesting usable and controllable smartphone-based interaction for representative manipulation tasks. We further demonstrate the extensibility of the framework through integration with an ink-painting simulator, showing its applicability to XR interaction, digital content creation, and rapid prototyping of low-cost haptic interfaces.

[paper]