Skeleton-Based Quality Assessment and Abnormality Detection in Human Action Performance

Elderly people can be provided with safer and more independent living by the early detection of abnormalities in their performing actions and the frequent assessment of the quality of their motion. Low-cost depth sensing is one of the emerging technologies that can be used for unobtrusive and inexpensive motion abnormality detection and quality assessment. In this study, we developed and evaluated vision-based methods to detect and assess neuromusculoskeletal disorders manifested in common daily activities using three-dimensional skeletal data provided by the SDK of a depth camera (e.g., MS Kinect and Asus Xtion PRO). The proposed methods are based on extracting medically-justified features to compose a simple descriptor. Thereafter, a probabilistic normalcy model is trained on normal motion patterns. For abnormality detection, a test sequence is classified as either normal or abnormal based on its likelihood, which is calculated from the trained normalcy model. For motion quality assessment, a linear regression model is built using the proposed descriptor in order to quantitatively assess the motion quality. The proposed methods were evaluated on four common daily actions—sit to stand, stand to sit, flat walk, and gait on stairs—from two datasets, a publicly released dataset and our dataset that was collected in a clinic from 32 patients suffering from different neuromusculoskeletal disorders and 11 healthy individuals. Experimental results demonstrate promising results, which is a step toward having convenient in-home automatic health care services.

The currently used approaches for detecting abnormal action performance and assessing motion quality take place at medical facilities where the patient is required to perform several standardized tests (e.g., walk, turn, sit down, stand up) while a specialist assesses the motion quality using traditional tools (e.g., stopwatch, questionnaire). Therefore, the patient is required to visit the clinic periodically, which could be a potential burden for him/her. Therefore, our research aims to develop an automatic system for in-home motion abnormality detection and quality assessment. This system will allow for the earliest possible detection of abnormality, which will lead to providing the elderly with a safer life and reducing the cost and burden of frequent clinic visits.

While there are numerous works for depth-based and skeleton-based activity recognition, our work focuses on abnormal action performance detection and assessment. A number of methods have been proposed for detecting abnormal events, such as falling, using depth cameras. In contrast, our proposed method for abnormality detection aims at detecting the abnormal performance of normal daily activities.

We have introduced a new dataset collected from patients with different types/degrees of neuromusculoskeletal disorders while performing three types of actions—walking, standing up, and sitting down. This new dataset is used in this work to confirm the efficiency of our proposed features in detecting and assessing motion disorders in realistic scenarios. We have proposed different normal motion models that rely on our simple action-independent descriptor, which encodes both the spatial and the temporal characteristics of the motion. The proposed descriptor consists of three medically justified and highly predictive features: asymmetry, velocity magnitude, and Center of Mass (CoM) trajectory deformation. These features are computed from the 3D skeletal data extracted from the SDK of a depth sensor that can be mounted at home or a clinic. For abnormality detection, we have built a probabilistic normalcy model using the features extracted by our proposed descriptor from normal sequences only. The reason behind building an action normalcy model is that the abnormality in motion is diverse and cannot be specified by certain samples of abnormal action performance. While treating the problem as a classification problem might yield better results, it may not generalize well to abnormalities not seen in the dataset. Instead, we have built a normalcy model for each action using normal sequences and for testing a sequence, we have evaluated the fitness of the sequence to that model. In other words, our method measures the deviation of the action performance from the normal action performance model, and this deviation reflects the likelihood of the action performance being normal/abnormal.

In contrast, for motion quality assessment, we have built a linear regression model using the proposed descriptor. The function of this model is to predict an assessment score for the action performance, which reflects the degree of abnormality in performing the action. The model is trained on the ground truth scores provided by a professional physiatrist. We have evaluated our proposed methods on two different datasets—a publicly released dataset and our collected dataset, E-JUST Motion Quality Assessment (EJMQA). Both datasets include different actions (e.g., walking, sitting, standing) and different types of neuromusculoskeletal disorders (e.g., Parkinson’s disease (PD), stroke, freezing). Results show that our proposed descriptor can capture abnormal action performance for different actions with high accuracy and can be effectively used for assessing the quality of action performance, even in real-life applications.

Conclusion

The clinical assessment of gait and motion and the interpretation of their abnormalities are crucial for the proper diagnosis and management of neuromusculoskeletal disorders. The ability to detect the presence of motion abnormalities and assess the quality of motion is an add-on in the clinical practice. Due to the aforementioned factors, in this paper, we study, compare, and evaluate two methods of motion abnormality detection and quality assessment. The methods depend on constructing a general descriptor comprising the concatenating of three efficient to compute features and then building a probabilistic/statistical model for motion abnormality detection or a regression model for quantitative motion quality assessment. We tested the proposed methods on different datasets. The results indicate that the proposed methods can detect the abnormality in performing different activities for patients with different types of neuromusculoskeletal disorders and assess the quality of the action performance as well.

The proposed methods can be used for in-home monitoring and rehabilitation as well as to assist specialists in performing motion analysis. This work can be extended by considering the limitations of the skeleton data, e.g., limited range (i.e., around 0.5–4 m for MS Kinect V2) and noisy skeletal data in case of occlusion, for more realistic applications where these limitations can prove to be hindrances. One of our extensions in this regard is to use raw depth data to compute the same descriptor in a more robust manner. Furthermore, it would be interesting to evaluate the extent to which the extracted features can be used to classify the type of motion abnormality, e.g., neurological disorders, articular disorders, or orthopedic disorders.