Project

Research Interests

Projects

Co-speech gestures are crucial for enriching both human-human and human-agent communications. Yet, the specific impacts of gesture size—especially when generated by advanced data-driven techniques—remain underexplored. This study investigates how varying gesture sizes affect human-agent interactions across two distinct persuasive contexts (informational and emotional), with a focus on social outcomes such as persuasion and empathy. We conducted two controlled experiments, each involving 36 participants, comparing three gesture conditions: Minimal gesture, Small gesture, and Large gesture conditions.

Managing negative emotions through emotion regulation (ER) is key to mental well-being. While virtual reality (VR) shows promise for supporting ER, prior work has primarily focused on self-regulation. This paper introduces a virtual agent that helps users manage emotions through conversation-based ER strategies. We compared three conditions: no agent, an agent with non-supportive responses, and an agent with ER-supportive responses. Results showed that the ER-supportive agent significantly improved users' emotional states and overall experience. Building on this, we conducted a second experiment to examine how the agent's appearance (realistic vs. cartoon) and voice tone (emotional vs. neutral) affect ER.

This study introduces three techniques in VR that enable users to manually adjust emotional facial expression while still reflecting real-time facial tracking results. The Ekman (Ek) technique allows users to select six discrete emotions via button interaction, while the Scrollable-Ekman (SEk) technique extends this by allowing users to scale the intensity of the selected emotion. The Arousal-Valence (AV) technique offers nuanced control within a two-dimensional arousal-valence space. We evaluated these techniques against a baseline condition that synchronizes users’ natural facial expressions, focusing on the expression of happiness, sadness, and anger.

Many researchers have studied three common visual communication cues (pointer, sketching, and hand gesture) in remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored We studied the effect of four cue combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, with three problem tasks: Lego, Tangram, and Origami. The study results showed that the participants completed the task significantly faster and felt a significantly higher level of usability with the additional sketch cue along with the hand gesture cue, but not with the additional pointer cue. Participants also preferred the combination with the sketch cue over the other combination. However, using the additional pointer and sketch cues increased the perceived mental effort and did not improve the feeling of co-presence. We discuss the implications of these results and future research directions.

SharedSphere is a Mixed Reality based remote collaboration system which not only allows sharing a live captured immersive 360 panorama, but also supports enriched two-way communication and collaboration through sharing non-verbal communication cues, such as view awareness cues, drawn annotation, and hand gestures.