In humans, vision plays a critical role in adapting motor strategies to changing environmental contexts. Translating this principle to wearable robotics, this talk explores how computer vision, particularly real-time scene understanding and object recognition, can be leveraged to enable context-aware control in soft wearable assistive robots. These systems are designed to move beyond reactive or fixed-pattern assistance by dynamically modulating support based on the user's surroundings, movement intent, and task demands. At the core of this approach is a shared control framework where wearable robots interpret motion data from the user merged with artificial vision information to cooperatively deliver tailored assistance in response to terrain changes or object interactions. The talk will present recent advances in both upper and lower limb exosuits, demonstrating how physics-informed and machine learning-based vision systems enable lightweight, efficient, and embedded control solutions. Experimental results highlight significant improvements in metabolic efficiency, muscle activity reduction, and task adaptability during real-world locomotion and object manipulation tasks. This work points toward a new generation of intelligent wearable robots designed for enhancing human performance in daily life for wellness and industrial applications.

Self-balancing exoskeletons are a key enabling technology for individuals with mobility impairments. While the current challenges focus on human-compliant hardware and control, unlocking their use for daily activities requires scene perception. In this talk I will present Exosense, a vision-centric scene understanding system for self-balancing exoskeletons. We introduce a multi-sensor visual-inertial mapping device as well as a navigation stack for state estimation, terrain mapping, and long-term operation. We tested Exosense - attached to both a human leg and Wandercraft’s Personal Exoskeleton - in real-world indoor scenarios. Experimental results will demonstrate Exosense reconstructing terrain as the Exo explores a building as well as enabling visual localisation in a previous map. 

Individuals with motor impairments need soft wearable robotic hands to assist with daily tasks, and these systems typically operate by interpreting user intentions. This talk explores a new paradigm in intention detection for wearable hand robots, leveraging vision-based approaches to bridge the gap between human intent and robotic assistance. We first introduce VIDEO-Net, a deep learning model that predicts user intention by analyzing spatial and temporal visual cues from a first-person- view camera, enabling soft robotic hands to respond seamlessly and without requiring calibration or additional user input. Expanding on this foundation, our second study extends intention detection to bimanual tasks, allowing individuals with spinal cord injuries to perform complex activities of daily living, such as preparing a meal, using bimanual hands. We hypothesize that the intentions underlying bimanual hand movements can be effectively inferred by classifying a set of fundamental low-level actions —such as grasping, pressing, tightening, squeezing and releasing. The third study targets cluttered environments common in occupational therapy, such as the peg-and-hole and box-and-block tests. We present a vision-based system that uses kinematic features to detect not only grasping intentions but also intermediate pregrasp postures—crucial for stable and precise object manipulation in stroke rehabilitation. Our findings demonstrate its effectiveness in structured therapy scenarios and highlight its applicability for goal-oriented pick-and-place tasks relevant to occupational therapy in daily living scenarios. 

Equipping AI and robotic systems with the ability to understand human behavior in everyday life is essential for enabling them to better assist people across a wide range of applications. However, the availability of high-quality human motion data for learning such knowledge remains extremely limited. In this talk, I will present our lab’s efforts to scale and enrich 3D human motion datasets by capturing everyday human movements and natural human-object interactions. I will first introduce ParaHome, our new multi-camera system designed to capture human-object interactions in natural home environments. Next, I will present MocapEvery, a lightweight and cost-effective motion capture solution that uses two smartwatches and a head-mounted camera to enable full-body 3D motion capture across diverse settings. Finally, I will discuss our recent work that enables machines to model comprehensive affordances for 3D objects by leveraging pre-trained 2D diffusion models, allowing for unbounded object interaction capabilities. 

Consumer wearable devices such as VR headsets, smart glasses, watches, and even AirPods are now packed with sensors and onboard compute, unlocking new possibilities for egocentric full-body motion tracking, synthesis, and understanding. In this talk, I will first present results in real-time full-body motion reconstruction from VR devices, comparing physics-based simulation with data-driven motion synthesis approaches. Next, I will share preliminary findings using only the Aria smart glasses (https://www.projectaria.com/glasses/), highlighting the importance of environmental context in motion understanding under limited sensing. Key challenges, including minimal sensing, real-time constraints, integration of multi-modal data, and dataset availability, will be discussed, along with opportunities for future research and impactful applications. Together, these advances are ushering in a new era of context-aware physical intelligence, where always-on wearables seamlessly integrate into daily life to deliver truly personalized and immersive user experiences.

In this talk I will do a technical deep dive into Mocapade: Meshcapade's markerless motion capture solution from any video. Mocapade is based on our CVPR 2025 paper: PromptHMR Promptable Human Mesh Recovery. PromptHMR reimagines human pose and shape estimation through the integration of spatial and semantic prompts. Unlike existing methods that either sacrifice scene context when working with cropped images or lose accuracy when processing full scenes, PromptHMR achieves both contextual awareness and high precision. I will demonstrate how our system leverages multiple input modalities—from facial bounding boxes in crowded scenes to natural language descriptions of body shape—to achieve state-of-the-art performance across challenging scenarios.

Egocentric vision puts a head‑mounted camera where a wearable robot most needs eyes—next to the person it assists. From this first‑person view, we can predict which object will be touched next, trace the steps of an unfolding task, and detect when an action drifts off script. I will present our lab’s progress toward these abilities, covering next‑active‑object prediction, task‑graph learning, and gaze‑guided error detection. Finally, I’ll discuss how egocentric vision needs to move towards online, streaming, and ultimately real-time processing to support real-world applications in wearable assistive systems and beyond.

Reconstructing 3D humans from single-image observations is critical for many robotics applications. The recent rise of larger datasets and foundation models raises the question of how to best leverage these resources to achieve new heights of reconstruction quality. I present two of our group’s recent works aimed at this challenge on two levels of abstraction. In the first work, Neural Localizer Fields (NLF), we formulate body pose and shape estimation pointwise in correspondence to a continuous canonical template. This allows us to merge 50 heterogeneous large-scale datasets that use different human representations, and to train a state-of-the-art generalist model that can trace out the posed body shape point-by-point in any landmark or mesh format. Next, in Human-3Diffusion, we tackle template-free 3D clothed avatar reconstruction, by combining the realistic appearance priors of 2D image diffusion models with the explicit 3D consistency of Gaussian splatting. Finally, I discuss the pros and cons of these different levels of image-based 3D human reconstruction, including remaining open challenges.