The BabyAffect Project

Despite the recognized importance of affect and non-verbal communication in early language development there has been surprisingly little research attempting to experimentally and formally identify the connection between the lexical acquisition process and the non-lexical content of child-adult interaction. This is due to a variety of reasons most notably: the lack of annotated data and the truly interdisciplinary nature of such an undertaking. The proposed line of research is especially relevant for furthering our basic understanding of the language and knowledge acquisition process, especially the role of multimodal and extra-linguistic cues in this process. Demonstrating the connection between affect, intent and language learning is also relevant for the autistic spectrum disorders and language-delayed toddler population that is often lacking in non- verbal communication skills.

The main scientific preposition behind BabyAffect is that the extra-lexical and extra-linguistic stream in child-caregiver communication, e.g., affect, communicative intent, is an important source of (often complementary) information that enhances significantly the lexical acquisition process in early childhood both in terms of quality (e.g., semantic categorization ability) and quantity (rate of learning, vocabulary spurt). We intend to demonstrate this both experimentally using statistical information extracted from audio- visual recordings of infants (and their caregivers) and formally using cognitive models of the lexical acquisition process using parallel distributed models and semantic networks.

In the past ten years, we have witnessed a flurry of research activity in the multimedia signal processing and machine learning communities in the areas of affective analysis, emotion recognition, behavior tracking, personality modeling and social signal processing for human-human and human-machine communication. Motivated by the fact that a significant portion of the information conveyed when communicating with other humans or machines is extra-linguistic, researchers have built models to recognize, classify and (even) synthesize moods, emotions, communicative intent, behaviors etc from features extracted from audio-visual recordings. Despite the applicability and relevance of (the majority of) these techniques to younger populations (infants, toddlers, schoolchildren) research has almost exclusively targeted adults. Recent results on children and preschoolers (and more recently infants by members of the BabyAffect consortium) have demonstrated for example, that emotion recognition works at least as well for pre-schoolers (interacting with a computer game) as for adults and that machine learning algorithms that utilize prosodic features can accurately classify the communicative state of infants and toddlers. Thus, although automatic speech recognition technology remains fragile for young populations, it is possible to automatically analyze the affective content, communicative functions and behavior of infants and toddlers from audio-visual recordings with very good accuracy. This realization opens new avenues for the automatic annotation and exploitation of longitudinal audio and/or video recordings of children.

BabyAffect Main Goals

  1. To develop a computational model for early vocabulary development using multimodal data conveying emotions and communicative functions from typical and atypical populations.
  2. To collect and make available to different disciplines (AI, Psycholinguists, Developmental Psychologists, Human Language Technology) a large amount of multimodal data from Greek speaking children of the one-word stage in natural environments.
  3. To investigate the ability of typical and atypical children to express emotions and communicative functions through distinct acoustic patterns, in order to develop an automatic screening tool for detecting children with autism and language delay (on the basis of their ability to use distinct acoustic patterns to express different emotions and communicative functions).

The main outputs of BabyAffect are the annotated audio and audio-visual data, as well as the software used for the automatic annotation of these data.

Recent Announcements

  • BabyAffect Workshop The project BabyAffect under the 'Excellence II " program of GSRT, the School of Electrical and Computer Engineering, NTUA, and the Department of Philosophy and History of Science  invite you to a single-day workshop  which will be held on Friday, November 13, 2015, 11:00-16:30 in the Multimedia Auditorium, Central Library, National Technical University of Athens, Zografou Campus.
    Posted Nov 9, 2015, 7:45 AM by Elias Iosif
  • Timothy Rogers Lecture Invitation The project BabyAffect under the 'Excellence II "and the School of Electrical and Computer Engineering, NTUA, and the Department of Philosophy and History of Science in the Contexts of the Graduate Program "Basic and Applied Cognitive Science", invite you to a lecture by Professor Timothy Rogers from the University of Wisconsin at Madison on: Searching for meaning in the human brain which will be held on Monday, September 29, 2014, 19:00 in the Auditorium "Alkis Argyriadis" Main Building, University 30.The attendance of the lecture is essential for all students of M.Sc. "Basic and Applied Cognitive Science"
    Posted Nov 10, 2014, 10:14 AM by Michail Toutoudakis
  • Project Starts BabyAffect Project starts officially on February 1st, 2014!
    Posted Apr 2, 2014, 8:45 AM by Michail Toutoudakis
Showing posts 1 - 3 of 3. View more »