research

The main objective of my research is the development of robots capable of interacting and communicating in a natural way with humans [1], expressing, through their shapes and behaviors, a high degree of Social Intelligence [2,3] (Figure 1, left). This intelligence "emerges" through the coherent exploitation of socio-cognitive skills [4] that ensure robots have "the ability to get along with others" [5], establishing long-term personal relationships with humans. The grand challenge of achieving such close interaction between humans and robots translates into two major challenges [6]: the analysis of interpersonal dynamics, capable of revealing the basic socio-cognitive skills that make interaction between humans possible; the development of socio-cognitive skills in robots that explicitly take into account the presence of humans in the perception-cognition-action loop [7]. These challenges enrich each other in order to propose generic models from a multidisciplinary perspective, establishing a convergence between the elaboration of social signals, social robotics, cognitive sciences, psychology, psychiatry, and neuroscience.

Thus, my activities have led me to explore various aspects of social robotics.

Modeling interpersonal interaction and exploiting social signals are among the major scientific problems addressed by affective computing and social robotics [8]. In particular, the analysis of interpersonal interaction dynamics aims to reveal the basic socio-cognitive skills that enable interaction between humans [9].

Consistent with this theoretical framework, during my periods at Osaka University and UPMC, I developed a system aimed at recognizing non-verbal social signals based on real-time RGB-D sensors, focusing specifically on capturing gaze direction, posture, and pointing gestures [10] (Figure 1, right). I then used this system to develop a set of metrics relevant to the human-robot interactive framework as a measure of interaction engagement [11], defining it as the process through which individuals involved in an interaction begin, maintain, and terminate their mutual connection [12]. In particular, I utilized these metrics in various experiments aimed at enhancing the credibility of humanoid robots iCub and Nao [13, 14].

At the same time, I began using this experimental platform to explore social behaviors within the context of developmental cognitive deficits. In this context, social signals are basic components for human development, essential skills that give rise to complex behaviors [9]. In Project Michelangelo 1, a European project aimed at developing computer technologies for the diagnosis and therapy of children with autism spectrum disorders, I had the opportunity to focus on attention [15,16,17] and imitation [18], showing significant differences between children with autism spectrum disorders, dyspraxia, and typical development [19, 20].

Subsequently, I had the opportunity to extend my research activities on these developmental deficits by studying perspective-taking with the same metrics [21, 22], thanks to the implementation through a virtual agent of the tightrope walker paradigm [6323]. These studies (Figure 2) represent a first step towards realizing a motor signature in autism spectrum disorders [24].

Humans are unique individuals, with their unique personalities, personal preferences, social roles, and consequently, they expect to be treated according to this uniqueness [25]. In a context of very close, long-term collaboration with human partners, social robots must be able to adapt their behaviors and interact with them in a "personalized" manner [6, 26]. This challenge translates into the production of metrics, models, techniques, and algorithms capable of capturing and describing the individual differences of human partners in terms of physical characteristics, personality, preferences, and social connections, as well as in the adaptation of robot behaviors in terms of social signals, vocabulary, and social rules that it must respect.

In accordance with this theoretical framework, during my collaborations with Osaka University, I proposed a model for learning and representing knowledge about people through memorization of their physical characteristics, voice, and face, and their preferences, expressed by information from a social network (likes or dislikes, social relationships, such as friendships and groups with common interests). Due to its complexity, the development of this robotic platform is still ongoing. However, the initial results suggest the feasibility and effectiveness of this approach [27, 29].

The dynamics of interpersonal interaction are characterized by continuous and mutual adaptation of the interlocutors in terms of behaviors [28]: analyzing this dynamic and consequent exploitation of social signals can become a useful tool for inferring individual differences among human partners. In the framework of Project EDHHI 2, I showed the possibility of predicting personality, particularly extraversion, during a social interaction with a humanoid robot [11] (Figure 3, left).

During my PhD studies at the University of Palermo and afterwards, in my collaborations with Osaka University, I also tried to exploit speech recognition to improve the mental model of my robots' partners. My work on natural language processing is based on the hypothesis that the performance of existing automatic speech recognition systems may struggle to generate an exact transcription of conversations in natural interaction situations. On the other hand, the robot can ignore the details of sentence structure and focus only on the overall meaning of communication. In line with this idea, I proposed the use of latent semantic analysis of sentences to create a "semantic-emotional" space. The informative content of verbal interaction is augmented by the emerging emotional information from the semantic space, which is then exploited by a Robovie-M robot capable of displaying emotional behaviors [30]. I followed a similar approach for personalizing behaviors according to the current topic of conversation: through the use of a classical text mining method, document classification based on "Term Frequency - Inverse Document Frequency" measurement, hierarchically applied to the entire Wikipedia document graph, I developed a Robovie Synchy robot capable of filtering out conversation details and understanding its main topic [31] (Figure 3, right). Although this system is still under evaluation, the initial results encourage further research on this approach.

One of the major weaknesses of experimental studies is the artificiality of the contexts in which they are conducted: a laboratory setup, a hospital, etc. The activities performed in these environments and their results may imply a lack of generalization. That's why in my research activities, I have always tried to "get out of the lab" and put to the test the models, algorithms, and more generally, the systems designed in real-world environments.

As part of a project at the University of Palermo, I developed a robotic guide capable of giving tours to visitors at the Agrigento museum. From there, I developed a group of heterogeneous robots, an Aldebaran Nao and a Peoplebot and a P3-AT, both from ActiveRobotics, used as robot stewards and receptionists [32] with behaviors inspired by theater theory, giving each robot a unique role and personality. These experiments showed the limited reliability of traditional sensors embedded in autonomous robots (Figure 4, left). At the University of Padua and Osaka, I improved the socio-cognitive and autonomous reasoning capabilities of robots in real-world environments by proposing the use of a multimodal sensor network deployed in the environment [29]. This project materialized in the development of a distributed perception system for the P3-AT robot in apartments for the elderly capable of recognizing and intervening in dangerous situations [33]. In the context of the Pramad project at UPMC, I had the opportunity to further explore this topic by developing an RGB-D sensor network deployed in the environment capable of enhancing engagement and interaction capabilities with elderly people of a PR2 robot from WillowGarage.

Recently, I became interested in schools to develop a school life assistant robot. The scenario of classroom use can be seen as a compromise between the structure of a laboratory and the complexity of real life: in this semi-structured context, the use of the robot is more feasible than in other less structured scenarios, such as children's homes, for example. Therefore, the school represents a sufficiently manageable scenario from a technological point of view and very interesting from an educational point of view. In particular, the robot in school has the potential to become a very powerful tool for adapting intensive therapies to the specific needs of the child.

In this context, and in line with my experiences with EEG data analysis and brain-computer interfaces [34, 35], I developed, in collaboration with the Child and Adolescent Psychiatry Department of the Pitié-Salpêtrière Hospital group, a robotic neurofeedback system for children with attention disorders: a Nao robot is capable of refocusing a child's attention during joint activities through real-time EEG data analysis, particularly exploiting the beta-theta ratio [36, 37] (Figure 4, center). This project will be implemented in a classroom at the Georges Heuyer School and will target a population of children with autism spectrum disorders and attention deficit disorders. The school is an integral part of the Child and Adolescent Psychiatry Department and represents the ideal context in which such a project can be developed, as it is in line with its mission of very close collaboration with the healthcare teams of the department, with a focus on care-study-insertion.

In the same school, I am developing, in collaboration with EPFL Lausanne, a co-writing robot capable of learning handwriting from example writing models provided by children (Figure 4, right). The central idea of this project is to reverse the relationship between teacher and student, making the child become the teacher of the robot. The student will therefore act as a mentor who will help his robot-protégé in classroom activities, practicing writing without realizing it [38]. The study will focus in particular on dysgraphia, the difficulty in handwriting and automating manual writing, an obvious symptom of dyspraxia. 

Bibliography

[1] C. L. Breazeal. Sociable machines : expressive social exchange between humans and robots. PhD thesis, Massachusetts Institute of Technology, 2000.

[2] N. Cantor and J. F. Kihlstrom. Personality and social intelligence. Prentice-Hall Englewood Cliffs, NJ, 1987.

[3] J. F. Kihlstrom and N. Cantor. Social intelligence. Handbook of intelligence, 2 :359–379, 2000.

[4] K. Dautenhahn. Socially intelligent robots : dimensions of human–robot interaction. Philosophical Transactions of the Royal Society B : Biological Sciences, 362(1480) :679–704, 2007.

[5] P. E. Vernon. Some characteristics of the good judge of personality. The Journal of Social Psychology,4(1) :42–57, 1933.

[6] C. L. Breazeal. Designing sociable robots. MIT press, 2004.

[7] A. Vinciarelli, M. Pantic, D. Heylen, C. Pelachaud, I. Poggi, F. D’Errico, and M. Schröder. Bridging the gap between social animal and unsocial machine : A survey of social signal processing. Affective Computing, IEEE Transactions on, 3(1) :69–87, 2012.

[8] K. Dautenhahn. Getting to know each other - artificial social intelligence for autonomous robots.Robotics and autonomous systems, 16(2) :333–356, 1995.

[9] A. Vinciarelli, M. Pantic, and H. Bourlard. Social signal processing : Survey of an emerging domain. Image and Vision Computing, 27(12) :1743–1759, 2009.

[10] S. M. Anzalone and M. Chetouani. Tracking posture and head movements of impaired people during interactions with robots. In International Conference on Image Analysis and Processing, pages 41–49.Springer, Berlin, Heidelberg, 2013.

[11] S. M. Anzalone, S. Boucenna, S. Ivaldi, and M. Chetouani. Evaluating the engagement with social robots. International Journal of Social Robotics, 7(4) :465–478, 2015.

[12] C. L. Sidner, C. Lee, C. D. Kidd, N. Lesh, and C. Rich. Explorations in engagement for humans and robots. Artificial Intelligence, 166(1-2) :140–164, 2005.

[13] S. Ivaldi, S. Anzalone, W. Rousseau, O. Sigaud, and M. Chetouani. Robot initiative increases the rhythm of interaction in a team learning task. In Proceedings of Timing in Human-Robot Interaction, Workshop of the 9th ACM/IEEE International Conference on Human-robot Interaction-HRI, pages1–4, 2014.

[14] S. Ivaldi, S. M. Anzalone, W. Rousseau, O. Sigaud, and M. Chetouani. Robot initiative in a team learning task increases the rhythm of interaction but not the perceived engagement. Frontiers in neurorobotics, 8 :5, 2014.

[15] S. M. Anzalone, E. Tilmont, S. Boucenna, J. Xavier, A.-L. Jouen, N. Bodeau, K. Maharatna, M. Chetouani,D. Cohen, M. S. Group, et al. How children with autism spectrum disorder behave and explore the 4-dimensional (spatial 3d+ time) environment during a joint attention induction task with a robot. Research in Autism Spectrum Disorders, 8(7) :814–826, 2014.

[16] S. M. Anzalone, S. Boucenna, D. Cohen, M. Chetouani, et al. Autism assessment through a small humanoid robot. In Proceedings of the HRI : A Bridge between Robotics and Neuroscience, Workshop of the 9th ACM/IEEE International Conference on Human-Robot Interaction, Bielefeld, Germany, pages 3–6, 2014.

[17] S. M. Anzalone, J. Xavier, S. Boucenna, L. Billeci, A. Narzisi, F. Muratori, D. Cohen, and M. Chetouani. Quantifying patterns of joint attention during human-robot interactions : An application fo rautism spectrum disorder assessment. Pattern Recognition Letters, 2018.

[18] S. Boucenna, S. Anzalone, E. Tilmont, D. Cohen, and M. Chetouani. Learning of social signatures through imitation game between a robot and a human partner. IEEE Transactions on Autonomous Mental Development, 6(3) :213–225, 2014.

[19] D. Cohen, C. Grossard, O. Grynszpan, S. Anzalone, S. Boucenna, J. Xavier, M. Chetouani, and L. Chaby. Autisme, jeux sérieux et robotique : réalité tangible ou abus de langage  In Annales Médico-psychologiques, revue psychiatrique, volume 175, pages 438–445. Elsevier Masson, 2017.

[20] D. Cohen, C. Grossard, O. Grynszpan, S. Anzalone, S. Boucenna, J. Xavier, M. Chetouani, and L. Chaby. Autism, serious games and robotics : Tangible reality or abuse of language  In Annales Médico-psychologiques, revue psychiatrique, vol. 175, no. 5, pp. 438-445. Elsevier Masson, 2017.

[21] S. Gauthier, S. M. Anzalone, D. Cohen, M. Chetouani, F. Villa, A. Berthoz, J. Xavier, et al. Behavioral own-body-transformations in children and adolescents with typical development, autism spectrum disorder and developmental coordination disorder. Frontiers in psychology, 9 :676, 2018.

[22] J. Xavier, S. Gauthier, D. Cohen, M. Zahoui, M. Chetouani, F. Villa, A. Berthoz, and S. Anzalone. Interpersonal synchronization, motor coordination, and control are impaired during a dynamic imitation task in children with autism spectrum disorder. Frontiers in psychology, 9, 2018.

[23] B. Thirioux, G. Jorland, M. Bret, M.-H. Tramus, and A. Berthoz. Walking on a line : a motor paradigm using rotation and reflection symmetry to study mental body transformations. Brain and cognition, 70(2) :191–200, 2009.

[24] J. Xavier, H. Guedjou, S. Anzalone, S. Boucenna, E. Guigon, M. Chetouani, and D. Cohen. Toward a motor signature in autism : Studies from human-machine interaction. L’Encéphale, 2018.

[25] D. C. Funder. The personality puzzle : seventh international student edition. WWNorton & Company,2015.

[26] K. Dautenhahn. Robots we like to live with  !-a developmental perspective on a personalized, lifelong robot companion. In RO-MAN 2004. 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No. 04TH8759), pages 17–22. IEEE, 2004.

[27] S. M. Anzalone, Y. Yoshikawa, H. Ishiguro, E. Menegatti, E. Pagello, and R. Sorbello. Towards partners profiling in human robot interaction contexts. In International Conference on Simulation, Modeling, and Programming for Autonomous Robots, pages 4–15. Springer, Berlin, Heidelberg, 2012.

[28] O. Hargie. Skill in theory : Communication as skilled performance. The handbook of communication skills, 3 :7–36, 2006.

[29] S. M. Anzalone, E. Menegatti, E. Pagello, Y. Yoshikawa, H. Ishiguro, and A. Chella. Audio-video people recognition system for an intelligent environment. In Human System Interactions (HSI), 20114th International Conference on, pages 237–244. IEEE, 2011.

[30] S. Anzalone, G. Balistreri, R. Sorbello, and A. Chella. An emotional robotic partner for entertainment purposes. International Journal of Computational Linguistics Research, 1(3/4) :94–104, 2010.

[31] S. M. Anzalone, Y. Yoshikawa, H. Ishiguro, E. Menegatti, E. Pagello, and R. Sorbello. A topic recognition system for real world human-robot conversations. In Intelligent Autonomous Systems 12,pages 383–391. Springer, Berlin, Heidelberg, 2013.

[32] S. M. Anzalone, A. Nuzzo, N. Patti, R. Sorbello, and A. Chella. Emo-dramatic robotic stewards. In International Conference on Social Robotics, pages 382–391. Springer, Berlin, Heidelberg, 2010.

[33] S. Ghidoni, S. M. Anzalone, M. Munaro, S. Michieletto, and E. Menegatti. A distributed perception infrastructure for robot assisted living. Robotics and Autonomous Systems, 62(9) :1316–1328, 2014.

[34] A. Chella, E. Pagello, E. Menegatti, R. Sorbello, S. M. Anzalone, F. Cinquegrani, L. Tonin, F. Piccione, K. Prifitis, C. Blanda, et al. A bci teleoperated museum robotic guide. In Complex, Intelligent and Software Intensive Systems, 2009. CISIS’09. International Conference on, pages 783–788. IEEE, 2009.

[35] L. Billeci, A. Tonacci, G. Tartarisco, A. Narzisi, S. Di Palma, D. Corda, G. Baldus, F. Cruciani,S. M. Anzalone, S. Calderoni, et al. An integrated approach for the monitoring of brain and autonomic response of children with autism spectrum disorders during treatment by wearable technologies. Frontiers in neuroscience, 10 :276, 2016.

[36] S. M. Anzalone, A. Tanet, O. Pallanca, D. Cohen, and C. Mohamed. A humanoid robot controlled by neurofeedback to reinforce attention in autism spectrum disorder. In AIRO@AI*IA, 2016.

[37] P. Nahaltahmasebi, C. Mohamed, D. Cohen, and S. M. Anzalone. Detecting attention breakdowns in robotic neurofeedback systems. In AIRO@AI*IA, 2017.

[38] P. Le Denmat, T. Gargot, C. Mohamed, D. Archambault, D. Cohen, and S. M. Anzalone. Cowriter robot : improving attention in a learning-by-teaching setup. In AIRO@AI*IA, 2018.