Models

Models will be developed to turn experimentally-collected data into predictors, at the basic level of trajectory continuation and at the level of expressive qualities of perception (RO2). Models exist for the kinematics of trajectories at a given scale of human action. They will be stressed at different scales and across the senses, and their validity assessed. New models will be developed based on the collected experimental data.

Feature-based machine learning techniques (e.g., support vector machines, regression trees, Gaussian processes regression, matrix completion) will be applied to predict trajectories and fill possible gaps (RQ2). The predictive power of features coming from different sensory modalities will be compared, when they will be used either alone or in combination with other features. Feature selection methods based on cooperative game theory (e.g., the Shapley value) will be used to assess the relative importance of features of different nature at different spatial and temporal scales. The capability of each feature to extract information related to other missing features will be investigated, making it possible to map visual to auditory or tactile stimuli, and vice versa. The minimal amount of information needed for the successful prediction of trajectories will be searched for. The derived models will be able to express the uncertainty in trajectory prediction, and to relate it to human characteristics such as expertise or psychomotor impairments. The high-level origin-of-movement feature, extracted using a mixed cooperative game/graph theoretical model, will be extended from individuals to groups of people, to assess how movement originates and propagates in such groups (RQ3). Such a feature will be also exploited in training feature-based machine learning models of trajectories. Additional models based on optimal control, reinforcement learning, and game theory will be used to predict trajectories of individuals or groups of people in the application contexts considered by the project, and they will be assessed in validation sessions. The ability of human beings to learn optimal trajectories derived by such models will be evaluated by making them really act in specific environments modeled by optimal control problems, and play with/against artificial agents implementing collaborative /adversary trajectories coming from the solutions to cooperative/noncooperative game-theoretical models.

Featureless models (such as deep feedforward and recurrent neural networks and convolutional neural networks) will be investigated as alternatives to feature-based models, the former avoiding the preliminary construction of hand-engineered features. Deep learning architectures will be trained on the raw experimental data, and features will be automatically discovered in their training phase, based on embeddings found either by principal component analysis or by nonlinear autoencoders. Width and depth of such neural networks will be optimized by cross-validation, depending on the specific learning task. Exploratory data analysis will be also used to reduce the dimensionality of the data, via an automatic construction of synthetic features. The relevance of features discovered by deep learning architectures (e.g., features associated with different time scales in recurrent neural networks) will be assessed using relevance indices derived by cooperative game theory. For better interpretability, simpler surrogate models will be trained under the supervision of previously trained and more complex deep learning architectures. Feature-based and featureless models will be compared in their predictive performance. The features constructed by featureless models will be compared and possibly integrated, at different scales and across modalities, with the ones proposed by experts in movement analysis and used in kinematic models of trajectories. By applying deep learning models not only to time series associated with trajectories but also to their time-reversed series, their potential ability to discover different highly-predictive features for such two kinds of time series will be assessed.