CAST - STEM | CGALAB

DR. YU XIANG | Jikai Wang | Qifan Zhang

Nicole Song, May Espinola, Reanna Jin, Rebecca Bo, Lyndon Yang, Isaac Chen, Ida Zou

Intelligent Robotics and Vision Lab (IRVL) at the University of Texas, Dallas

~ 2024 STEM CAST SUMMER CAMP 2ND PLACE WINNER ~

PRESENTATION

UTD-016_Final_Presentation

POSTER

016 Poster.pptx

PROJECT SYNOPSIS

Our project’s overarching focus was to estimate hand and object poses from data collected in a multi-camera system. We developed an end-to-end pipeline for simultaneously tracking hand and object poses from collected recordings as commonly used in AR/VR (XR), robotic and computer vision, security, etc.

MANO

Implemented a pre-trained model called MANO, to render objects and shapes. The MANO hand model is a trained model which takes shape parameters and pose parameters for the 3D hand shape. MANO is based on SMPL models. SMPL takes the shape of the body position/hand position. SMPL, short for Skinned Multi-Person Linear, uses a pipeline to create a variety of poses and body shapes. We calibrated the hand model to fit the merged point cloud for each person’s hand shape.

HAMER

HaMeR is an approach for Hand Mesh Recovery. Given the bounding box from a single frame, it reconstructs the hand in 3D from monocular input in the form of a MANO mesh.

HaMeR takes multiple datasets and sets them as frames. These frames are used to estimate the hands. These datasets have to include an RGB image.

FOUNDATIONPOSE

FoundationPose is a model used for 6D object pose estimation and tracking. FoundationPose uses RGBD images to collect data. This model uses 6D to estimate the object pose, hence being model-free.

PROCESS

We studied computer vision basics, such as deprojection, triangulation, transformation, MANO hand model and object 6D pose tracking. We collected recordings of hand movements with the various objects and calibrated our MANO hand shape parameters. Then, we split into two groups to focus separately on hand and object pose. Group A used MediaPipe and Hamer to estimate hand poses, while Group B used FondationPose to track object poses. Then, we combined the tracked hand and object poses into the final poses.