Our paper has been accepted to Internet of Things  Journal

Posted Date: Dec 12, 2025

Jihwan's work titled "TouchAI: Towards Waterproof Mobile Touchscreen Interface via Acoustic-Inertial Sensing Modalities" to apper in IEEE Internet of Things Journal, 2025.

Congratulations!

TouchAI: Towards Waterproof Mobile Touchscreen Interface via Acoustic-Inertial Sensing Modalities

Jihwan Lee, Youngwook Son, Tiago Koketsu Rodrigue, Yishi Zhu, Chulyoung Kwak, and Saewoong Bahk

Abstract

Conventional capacitive touchscreens in mobile devices often malfunction in the presence of moisture, dust, or gloved hands, leading to unreliable user interactions. To overcome these limitations, we present TouchAI, a multi-modal touch detection and localization system that leverages built-in inertial sensors and audio interfaces. Running as a background process, TouchAI constantly monitors inertial measurements for potential touch input and then activates acoustic sensing only upon detected events, thus reducing power consumption and privacy concerns associated with continuous audio recording. We introduce a novel event identification method that distinguishes touch-start and touch-end instances, providing precise timing for data sampling. For fine-grained touch localization, TouchAI employs a lightweight Transformer-based classification model to capture spatiotemporal features from combined acoustic-inertial signals. Experiments on commercial smartphones demonstrate true positive rates of 95.3% and 99.1% for touch detection and event identification, respectively, at false positive rates only less than 3%. Ultimately, TouchAI achieves up to 97.0% and 86.5% localization accuracy on 4×2 and 6×3 touch input grids, respectively, confirming its practicality and effectiveness as an alternative interface under touchscreen malfunction scenarios.