This session will focus on multimodal data fusion and analysis. After a quick overview of multimodal data fusion and its challenges, participants will dive into hands-on exercises using the PR-HRI dataset [1] (summary is given below) which is a rich multimodal dataset collected in a robot-mediated collaborative educational setting. Exercises will include feature level fusion of sensor data, exploring dependencies between modalities with correlation and statistical tests.

We will explore the statistical tests from two conceptual frameworks shown in the following table based on [2]. 

A brief introduction to the dataset will be provided at the beginning of the hands-on activity, to improve the overall understanding of the exercise. Additionally, the participants are encouraged to bring their own datasets and test different approaches. 

In summary, we will cover the following 4 parts: 

• 1.30 pm - 1 .50 pm: Overview of multimodal data fusion and its challenges

• 1.50 pm - 2.20 pm: Introduction to Frequentist vs Bayesian data analysis

• 2.20 pm - 2.40 pm: Hands on (Frequentist approach for data analysis)

• 2.40 pm - 3.00 pm: Hand on (Bayesian approach for data analysis)

Resources and Tools:

• PR-HRI dataset: A multimodal temporal dataset collected in a robot-mediated collaborative learning setting. It includes team behaviors, speech, gaze, affective states, and learning outcomes for 34 teams of children aged 9–12, enabling research on engagement, motivation, and collaboration (links).

• Hands-on activities with interactive coding exercises, with code snippets (links to be provided)

References

[1] J. Nasir, B. Bruno, M. Chetouani, and P. Dillenbourg, “What if social robots look for productive engagement? automated assessment of goalcentric engagement in learning applications,” International Journal of Social Robotics, vol. 14, no. 1, pp. 55–71, 2022.

[2] J. K. Kruschke and T. M. Liddell, “The bayesian new statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a bayesian perspective,” Psychonomic bulletin & review, vol. 25, pp. 178–206, 2018.

[3] J. Wakefield, Bayesian and Frequentist Regression Methods, vol. 23, New York: Springer, 2013.

[4] T. K. Mohd, N. Nguyen, and A. Y. Javaid, “Multi-modal data fusion in enhancing human-machine interaction for robotic applications: A survey,” arXiv preprint, arXiv:2202.07732, 2022.