The following work packages (WPs) are planned and carried out within the project

WP1: Equipment setup

As a first step, we will define the detailed specifications of the MOCAP system, the storage system and the computing infrastructure, so that the ordering of the equipment can proceed. As soon as the equipment is made available, it will be installed at our premises at an already selected and allocated space. At the same time, a plan of use will be compiled and disseminated, so that interested academic and industrial organizations in Greece become aware of the terms and conditions of usage of the whole infrastructure. Finally, as part of the "plan of use" the website of the "MOCAP as a service" will be implemented that will enable interested groups and organizations to book time on the system.

WP2: Datasets acquisition

After the acquisition, installation, and setup of the MOCAP system (WP1), it will be used to capture datasets that will be used (a) for training of learning-based methods and (b) for the quantitative evaluation of the performance of the algorithms developed within WPs 3-5. The developed datasets will be annotated, preprocessed and made available to the research community. Datasets are foreseen in all aspects of the technical work, including Head pose, facial landmarks, 3D hand pose, 3D body pose, Facial expressions andHand gestures. To achieve this, we will develop and use algorithms for pose estimation and motion tracking of human bodies and body parts at different spatial scales, as well as for action and gesture recognition. Datasets will consider humans performing in isolation, but also in interaction with objects or other humans in their environment.

WP3: Capturing human motion kinematics

The objective of this work package is to capture and model the kinematics of human motion, an essential part of understanding and replicating complex human motion patterns. Using the datasets procured during WP2, we will engage state-of-the-art neural network models to analyze the motion of body parts across different spatial scales. The focus will be on developing algorithmic solutions that can efficiently handle the complexity and variety of human motion without the need for physical markers.This will facilitate markerless, vision-based motion capture, capable of accurate, real-time representation of individual body parts pose. We aim to create a robust system that can capture, model, and replicate real-time human motion across a broad spectrum of scales and activities.

WP4: Capturing human motion semantics

In this work package, we will leverage the datasets from WP2 along with the results of markerless motion capture methodologies developed in WP3 to tackle the semantic interpretation of human motion. Neural networks will be trained to analyze human motion and extract meaningful information, such as facial expressions, hand gestures, and various activities at different spatial scales. Emphasis will be placed on not just understanding the motion, but the underlying intent and sentiment associated with it. By developing state-of-the-art techniques for gesture and expression recognition, we will be able to interpret and predict the semantic meaning of different body movements. This will pave the way for advanced motion-based communication systems, interactive entertainment, and more efficient human-computer interfaces. Ultimately, this work package will enable us to understand the 'why' behind the 'how' of human movement.

WP5: AR/VR for assistive healthcare

We will capitalize on the research outcomes of WP3 and WP4 to develop applications that perform performance evaluation in the context of medical rehabilitation. Our intelligent agents will provide interpretable corrective feedback which will assist the users for improvement. Our goal is to automate the time-consuming process of “observing and correcting” multiple iterations of an activity. In addition, we aim to provide novel tools that can reveal hidden details related to the effective and proper execution of a human activity. We will use AR/VR systems to provide the feedback in a natural way and online, for instance by overlaying an avatar that adopts the proposed correction while the user is performing the action.

WP6: Dissemination and exploitation

This WP will deal with the dissemination of the developments and the findings of this project both to the scientific community and to the general public. We will also be very active towards increasing the awareness of the Greek scientific community regarding the availability of the MOCAP infrastructure and the opportunities that exist for using it for research and development purposes. In the context of this WP, we will also consider the possibility of fostering innovation and exploiting the research results in relevant application domains and sectors.

WP7: Management

Financial, administrative and ethical management of the whole project. Besides the typical administrative issues that are typically taken care off in any research project, particular emphasis will be given to guarantee that the MOCAP infrastructure will be specified, ordered, installed, set-up, and become operational on time, giving access to it to the local but also to the broader Greek research community. Given that the project involves acquisition of datasets consisting of video recordings of humans in action, we will guarantee that this is achieved in full accordance with the relevant national and EU regulations.

The following Gantt chart summarizes this work