3D Human Pose Estimation

Human Motion Pose Estimation (HMPE) aims to localize the joints and estimate the pose when a person makes movements in the video. Ever since the pandemic broke out, telemedicine using webcams or general 2D RGB cameras has become increasingly popular and on the rise. There is a strong need for new technology such as 3D Pose Estimation using deep learning models that can estimate and evaluate posture as well as extract 3D data with a patient's image only, especially in physical rehabilitation treatment. 

 Although there are some useful machine learning models, there are still no clear criteria of directions or angles for images of input data when using those models. Therefore, the goals of our study include the extraction of 3D coordinates and angles of the joints of interest (JOI) from the 2D RGB video and analysis of the output data to find out the quantitative differences of angles among the joints of interest for the videos taken in two different directions for the same motion.

 We first extracted joint data using Google's BlazePose GHUM model and preprocessed the numerical data to compare the angle differences for each video taken from two different directions. The deep learning model we utilized has a total of 33 key points including facial muscles. The model is a well-performing publicly available solution, and robust especially in occlusion, truncation, and velocity (30-45 fps) of the image.

 Input data used for the study are the bicep curls exercise videos filmed in the front/diagonal direction. The 3D coordinate value for each frame was output in .csv format. Considering the visualization of the result data, the video in which the joint skeleton is overlaid was extracted in .mp4 to estimate the angle of the front and diagonal images and analyze the quantitative differences of both using RMSE. In addition, the angles of the centre point (elbow) were estimated based on the shoulders, elbows, and wrists on the left/right side, which are the JOI (joint of interest) of the bicep curls exercise, during exercise.

The estimation of the elbow angle and error (RMSE) was analyzed for both sides of the shoulders, elbows, and wrists, which are the exercise parts of the bicep curls. There was a difference of 10.54 degrees in the right elbow and 9.83 degrees in the left elbow (comparison between the right elbow angle from the frontal/diagonal direction; the left elbow angle from the frontal left/diagonal direction). 

Based on the direction of the videos, RMSE in elbow angles on both sides was 14.67 degrees in front and 9.79 degrees in the diagonal plane (comparison between the left and right elbow angle in the front-view video, that of the diagonal-view video). As shown in the figure4, it was verified that the direction or angle of filming the video acts as an important factor in human pose estimation using a machine learning model by the angle difference of the joints used in the image taken from the front and diagonal. 

In the process of estimating the angle of the joint of interest, we found that there was a subtle angle difference in performing the same operation with both arms, the result of which was confirmed by the instructor later that it is true that the left elbow is often less bent than the right one due to the sports damage.

Throughout the study, we've seen the possibility of developing quantitative metrics to determine not only the error but also the accuracy of performance using more detailed motion pose evaluation metrics in the near future.

Furthermore, the fact that the direction of the video is important when utilizing deep learning models for the pose estimation led us to devise our own regression models for the joint/skeleton alignment in the subsequent studies.

[1] Grishchenko, I., Bazarevsky, V., Zanfir, A., Bazavan, E. G., Zanfir, M., Yee, R., Raveendran, K., Zhdanovich, M., Grundmann, M., & Sminchisescu, C. (2022). BlazePose GHUM Holistic: Real-time 3D Human Landmarks and Pose Estimation. arXiv. https://doi.org/10.48550/arXiv.2206.11678

[2] Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F., & Grundmann, M. (2020). BlazePose: On-device Real-time Body Pose tracking. arXiv. https://doi.org/10.48550/arXiv.2006.10204

[3] Moon, G., & Lee, K. M. (2020). I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image. arXiv. https://doi.org/10.48550/arXiv.2008.03713 

[4] https://www.dreamstime.com/physiotherapist-blue-uniform-demonstrating-female-patient-position-to-exercise-legs-knees-rehabilitation-image214742196 

[5] http://www.hospi.co.kr/user/board/read_view/1315/36?cate=1315&ch=