Mobile Meets AI: Unveiling Opportunities and Use Cases

Abstract: In this workshop presentation, I will share our group's recent work at the intersection of mobile technology and artificial intelligence. I will specifically focus on our endeavors in developing experimental human-centered software systems, highlighting the technologies we have developed to capture and interpret human behavior, emotions, and the surrounding context. Additionally, I will introduce various applications spanning domains such as healthcare, childcare, and education that harness these technologies.

An automated classifier for periods of sleep and target-child-directed speech from LENA recordings

Abstract: Some theories of language development propose that children learn more effectively when exposed to speech that is directed to them (target child directed speech, tCDS) than when exposed to speech that is directed to others (other-directed speech, ODS). During naturalistic daylong recordings, it is useful to identify periods of tCDS and ODS, as well as periods when the child is awake and able to make use of that speech. In this talk we will present our efforts to develop free publicly-available classifiers (https://kachergis.shinyapps.io/classify_cds_ods/) that can support automated processes in the coding workflow for 1) periods of sleep vs. awake (sleep classifier), as well as 2) target-child-directed versus other-directed speech (tCDS/ODS classifier). We include over 1,000 hours of audio from daylong recordings including 153 English- and Spanish-speaking families in the U.S. with 17- to 28-month-old children. Results reveal high sensitivity and specificity for our sleep classifier, and moderate sensitivity and specificity of our tCDS/ODS classifier. Suggested workflows and limitations will be discussed. 

Refining Infant Word Recognition Analysis through Optimised Gaze Feature Selection based on Random Forest

Abstract: Infant eye-tracking studies are crucial for understanding early cognitive and linguistic development, by analysing gaze patterns to infer infants' processing of stimuli. Challenges like fluctuating attention spans, biases and experimental noise in gaze data persist. Our study uses the Random Forest (RF) algorithm, ideal for the complex, non-linear nature of eye-tracking data. We trained RF using gaze features, including raw and adjusted gaze proportions towards the target, and differences in gaze proportions post-stimulus compared to the baseline. Validated through a Leave-One-Out process on data from 25 Korean infants, the model achieved 88.45% accuracy, surpassing traditional models. Notably, it improved correlation with parental reports on the MacArthur-Bates Communicative Development Inventories from r’s between .47 and .52 to .65, indicating enhanced convergence of infant word recognition with parental reported measures. Future research will expand gaze data quantification methods and dataset diversity, and develop advanced AI models like neural networks. This approach will enhance feature selection, with subsequent efforts focused on evaluating the tool's reliability and validity in assessing cognitive and linguistic development.

Factors Influencing Webcam-Based Automated Gaze Coding: Implications for Infant Online Testing

Abstract: Online experiments have enabled developmental scientists to increase sample sizes, access diverse populations, lower the costs of data collection, and promote reproducibility. When researchers conduct online experiments on infants, a main outcome measure is infants’ gaze behavior as collected via a webcam. However, webcam data collected at home may vary in terms of lighting conditions, the field of view, or infants’ face position, leading to difficulties in classifying gaze directions automatically. We created an adult webcam dataset (N = 60) that systematically reproduced noise factors from infant webcam studies which may affect the accuracy of automated infant gaze coding. We varied participants’ left-right offset, distance to the camera, facial rotation, and the direction of the lighting source. By running state-of-the-art classification algorithms on the dataset, we found that facial detection performance was affected by distance to the camera, while gaze coding accuracy was affected by distance to the camera, facial rotation, and lighting source. Left-right offset had less influence. This will guide the design of better instructions for participants during online experiments. Moreover, training algorithms using the dataset will allow researchers to improve robustness and efficient data collection.

Linguistic predictors of Korean infants’ early word learning 

Abstract: Which words are easier for infants to learn? Previous research has shown that concreteness and the frequency of each word play important roles. Given that the patterns of language development can vary across different languages, it is important to examine whether factors previously identified as influential in word learning apply to other language communities. In this study, we investigate the effects of various linguistic factors on Korean children’s word learning. Two forms--Words & Gestures and Words & Sentences--of Korean MacArthur-Bates Communicative Development Inventories (K-CDI) were used to estimate children’s comprehension and production of each word. Comprehension and production were predicted by the following linguistic factors: total frequency, solo frequency, the number of synonyms, the number of inflected forms for each word appeared in the Korean corpus, and concreteness for each word. Preliminary analyses revealed that the effect of solo frequency was weaker in older children compared to younger children in both comprehension and production. The effect of solo frequency varied across word categories; it was stronger for predicates in comprehension, but in production, it was stronger for words in the other category. In production, as children’s age increased, the effect of concreteness also increased, but the effect of the number of inflected forms decreased. The effect of total frequency was stronger for nouns. Overall, the current results extend the previous findings to Korean and provide evidence that the significance of linguistic factors may vary depending on children’s age and the word category. 

Shape bias in vocabulary development of young Korean children

Abstract: We examined the role of shape-biased vocabulary, an underexplored predictor in Korean vocabulary development. As children acquire around 50 object nouns, they may utilize the shape of referents to generalize labels. Using the Korean version of the MacArthur-Bates Communicative Development Inventory, we assessed expressive vocabulary in 1,575 children (803 boys, aged 18-36 months). Employing Samuelson and Smith's (1999) approach, we identified 146 shape-biased items. The study revealed that the proportion of shape-biased words to object nouns significantly predicted children's percentile scores. Furthermore, there was a noteworthy interaction between object noun size and the proportion of shape-biased words. The results suggest that the shape of referents plays a crucial role in the early vocabulary development of Korean children. Importantly, the impact of shape-biased word proportion is moderated by object noun size, highlighting the nuanced relationship between these factors. Overall, this study contributes valuable insights into the significance of shape-biased words as predictors of children's vocabulary outcomes.

Research Opportunities for Early Language Development in Human-Computer Interaction Domain

Abstract: In this presentation, I would introduce our study "Lookee: Gaze-based Infant Vocabulary Comprehension Assessment and Analysis" and discuss opportunities for future applications for early childhood language development and education. Lookee is a lightweight solution that leverages the IPLP method to assess and analyze infants' vocabulary understanding. Based on the findings from Lookee, I would like to discuss future directions to improve the accuracy of the analysis. This involves not only collecting extensive data and developing diverse analytical models but also extends to utilizing various modalities beyond gaze (e.g., voice or facial expressions). Also, I would like to further discuss opportunities in developing educational applications for young children with state-of-the-art technologies including Large Language Models, Generative AI, and Mixed Reality.

An automated classifier for periods of sleep and target-child-directed speech from LENA recordings

Abstract: Some theories of language development propose that children learn more effectively when exposed to speech that is directed to them (target child directed speech, tCDS) than when exposed to speech that is directed to others (other-directed speech, ODS). During naturalistic daylong recordings, it is useful to identify periods of tCDS and ODS, as well as periods when the child is awake and able to make use of that speech. In this talk we will present our efforts to develop free publicly-available classifiers (https://kachergis.shinyapps.io/classify_cds_ods/) that can support automated processes in the coding workflow for 1) periods of sleep vs. awake (sleep classifier), as well as 2) target-child-directed versus other-directed speech (tCDS/ODS classifier). We include over 1,000 hours of audio from daylong recordings including 153 English- and Spanish-speaking families in the U.S. with 17- to 28-month-old children. Results reveal high sensitivity and specificity for our sleep classifier, and moderate sensitivity and specificity of our tCDS/ODS classifier. Suggested workflows and limitations will be discussed. 

Decoding the efficacy of child-directed speech as input for learning

Abstract: Caregivers often modify their communication style when interacting with children. This particular speech register, called Child-Directed Speech (CDS), is recognized for its role in providing tailored input to facilitate infants' language acquisition. Drawing insights from analyses of interactions between Korean mothers and their children, I explore various features associated with CDS and address the question of how these features contribute to infants' learning. CDS contains "islands of clarity", characterized by small segments of enriched input. I posit that infants' inclination toward novelty-driven learning and their attention to "oddball" elements could offer valuable insights into the underlying mechanisms at play. Furthermore, I examine broader characteristics of CDS influenced by statistical regularities, which may contribute to an overall enhanced language learning experience. In essence, CDS serves as a dynamic and multifaceted communication strategy, incorporating both localized features that capture infants' attention and global features influenced by statistical patterns, collectively fostering an enriched environment for language learning.

Invited discussant

Margarethe is the director of the Speech in Little Bilinguals (SLBL) lab at the University of Kansas. She brings a wealth of knowledge and experience in bilingual speech production and perception in children, with a specific focus on the effects of exposure to accented speech on phonetic and phonological development. Her work aims to create tools for assessing speech and language abilities in children who speak languages for which few or no normed assessments are available. Margarethe will offer valuable insights and perspectives, contributing to the depth and richness of our discourse during the session.

Cognitive Neuroscience-inspired Next-generation AI

Abstract: Mobile and ubiquitous AI mandates resource-efficient AI algorithms since centralized AI suffers from several issues, e.g., privacy, personalization, and networking. However, state-of-the-art AI algorithms consume huge computational resources including time and energy.  Interestingly, the human mind spends orders-of-magnitude less energy than recent AI algorithms for intelligent behaviors. It motivates me to advance AI techniques by applying the structure and mechanism of the biological mind. First, I present VECA(Virtual Environment for Cognitive Assessment), a first benchmark to assess the overall cognitive development of an AI agent by virtually implementing the Bayley-Scales test for human infants and toddlers. Second, I introduce Papez, which applies the biological mechanism of cognitive auditory processing to a representative DNN model, Transformer, on a speech separation task to substantially reduce its model size and inference latency with minimal performance loss. Finally, I introduce the spiking neural network, the neuro-biologically inspired next-generation neural network architecture that consumes orders-of-magnitude smaller energy per inference compared to traditional DNN.

Machine Learning versus Statistics: Extracting Arguments from ML Results

Abstract: In the era of machine learning (ML), often interchangeably referred to as statistical learning, vast opportunities unfold before us. While ML has become a powerful tool, its potential is sometimes hindered by misconceptions, particularly the perception of ML models as inscrutable black boxes. Even seasoned researchers can falter in extracting meaningful insights, leading to the risk of making errors and asserting inaccurate claims due to the opaque nature of ML models. In this talk, I will explore the distinctions between traditional statistics and ML-based analysis, with a particular focus on elucidating the intricacies of what and how ML models learn. This will involve a specific examination of the numbers undergoing updates and a clarification of the mechanisms governing these changes. By delving into these intricacies, our goal is to offer a comprehensive understanding of the processes inherent in ML models and facilitate your interpretation of the results.

Realizing accessibility and improving reliability in developmental science: Remote, online testing with ManyBabies-AtHome

Abstract: Laboratory studies of infant development frequently rely on the measurement of infant looking behavior, using a highly controlled environment, but often only testing a small section of the world’s population. Online testing holds great promise towards more ecologically valid, more highly powered, and more representative experiments. Despite tremendous advances in at-home testing, there are significant obstacles, which the ManyBabies-AtHome project aims to address collaboratively, focusing on the challenges of accessibility and reliability of infant online testing across the world while remaining resource-friendly. To ensure that infant online testing is accessible, we provide researchers with an open framework to meet ethical and data protection regulations as well as an approach that aims to reduce language and cultural barriers. To achieve greater reliability, we aim to provide generally applicable solutions in procedure, documentation, standardization and analysis for infant online testing. This presentation will introduce the ManyBabies-AtHome Framework and present preliminary results from two studies within the project, which focus on visual stimuli (Study 1) and synchronized audio and visual stimuli (Study 2). Study 1 examines infants’ visual preference, focusing on the reliability of coding infants’ looks to the left and right of the screen, as well as measuring attentional differences between static and moving stimuli. Study 2 studies the developmental trajectory of word recognition in infants learning different languages.