Embedded Files
Takumi Akaishi*, Tamon Miyake*, Yushi Wang, Takuma Tsukakoshi, and Shigeki Sugano
Waseda University
(*equal contribution)
The development of general-purpose humanoid robots capable of performing diverse assistive tasks has attracted increasing attention, as such systems could reduce the burden on human caregivers and address labor shortages in care settings.
Among caregiving activities, sit-to-stand (STS) motion is a fundamental yet complex and physically demanding task. Effective assistance requires not only sufficient actuation capacity but also accurate anticipation of human motion during close physical interaction. STS support involves safe body contact, non-cyclic synchronization with the user, and stable multi-contact control. While previous studies have investigated stable human–robot contact strategies, reliably recognizing human posture during assistance remains a significant challenge.
In this study, we propose a multimodal latent representation framework for reconstructing 3D human poses from limited observations using a variational autoencoder combined with Flow Matching. The model learns a conditional latent transport mechanism that reconstructs full multimodal states from sparse sensory inputs, including two-dimensional skeletal cues and robot joint somatosensory signals. Experimental results demonstrate that the proposed method reconstructs structurally coherent 3D poses across variations in body size and chair height.
Temporal alignment of the estimation further indicates that the robot can synchronize its assistive control with the participant’s movement, enabling timely initiation and sustained STS assistance
Objective and Approach
The objective of this work is to develop a multimodal latent representation framework that reconstructs full three-dimensional human poses from limited onboard observations, without relying on external sensing, for deployment on a humanoid robot during STS assistance.
When the user’s Center of Gravity (CoG) remains above the chair, the robot should apply a forward-directed pulling force to promote trunk flexion and facilitate the transition to a forward-leaning posture. Once the CoG has shifted anteriorly beyond the base of support of the chair, the robot should provide an upward assistive force to support vertical elevation and standing. The timing of force application is critical: synchronizing the robot’s assistance with the user’s voluntary movement enables effective support with minimal external force. By adapting both force direction and timing to the user’s movement phase, the robot can achieve safe, effective, and minimally intrusive STS assistance. Since the STS motion involves dynamic and phase-dependent whole-body transitions, accurate 3D pose estimation of the care recipient is fundamental to ensuring coordinated assistance.
Proposed Model for Estimating Foot and Garment States Using Semantic Visual Attention
We combine 2D human skeletal cues with robot somatosensory information, including joint states and tactile sensing, to reconstruct 3D human pose from limited observations. A multimodal latent representation is learned with a VAE, and Flow Matching reconstructs the latent human state from the observations available onboard the robot.
During training：
Data from the robot’s on-board sensors and the external cameras during the past 5 frames are used.
Fisheye images are processed by YOLO v8.
16-dimensional latent variable 𝑧_1 through the VAE encoder.
Conditional velocity field is trained using paired samples of latent targets
For inference：
Using only the robot’s joint angles, torque, tactile information, and 2D human skeletal coordinates.
The estimated latent variable is generated via Flow Matching.
VAE Decoder estimates the 3D human pose.
Acknowledgments
Acknowledgments
This work was supported by JST Moonshot R&D, Grant No. JPMJMS2031.
Citation
coming soon...
BibTex :
DOI:
Page updated
Google Sites
Report abuse