Abstract : This paper presents a learning-based hand–wind interaction framework that enables real-time airflow generation and cloth simulation in a web-based environment using only a standard webcam. Conventional gesture-driven wind interfaces typically map hand motion to a single force vector, which limits their ability to represent the spatial structure and temporal continuity of airflow induced by natural hand gestures. As a result, abrupt wind transitions and visually implausible cloth responses often occur, especially during gesture changes. To address these limitations, we propose a data-driven airflow representation based on learned airflow templates. Instead of directly solving physically accurate fluid dynamics, our approach focuses on visual plausibility by learning representative airflow patterns from paired hand-motion and material-response data. We construct a dedicated dataset by simultaneously recording hand gestures and airflow-induced paper motion using a dual-camera setup. The resulting airflow vector fields are clustered via unsupervised learning to extract a small set of representative airflow templates. At runtime, a temporal gesture classification model based on Long Short-Term Memory (LSTM) networks analyzes recent hand-motion sequences and predicts a probability distribution over the airflow templates. The inferred airflow is obtained through probability-weighted template blending, which ensures smooth transitions between different wind patterns. This airflow field is then injected as an external force into a lightweight fluid solver to generate divergence-free velocity fields, which are subsequently coupled with a Position-Based Dynamics (PBD) cloth simulation. Experimental results demonstrate that the proposed framework produces stable, continuous, and visually coherent hand–wind interactions in real time within a web browser. The system successfully translates natural hand gestures into spatial airflow patterns that interact intuitively with deformable cloth, without requiring specialized sensors or plugins. Our work highlights the potential of learning-based airflow modeling as a practical alternative to heuristic wind control for immersive, web-based interactive simulations.

[paper]