My research stands at the intersection of natural language processing, automatic speech recognition and language acquisition.

This current project aims to model infants' acquisition of vacabulary knowledge from raw speech signals. In the proposed computational model, the memory components are incoporated in order to simulate the top-down influence in langauge acquisition.  [Slides; Codes]

[Acknowledgement]  Special thanks to Prof. dr. Emmanuel Dupoux, whose expertise was invaluable in formulating the research questions and methodology. 

One of the most important milestones in children’s social and cognitive development is the ability to hold conversations with people around them in a contingent manner. This study how this skill emerges in preschool (2 to 5 years of age) in the context of child-caregiver naturalistic interactions, using question-initiated sequences as units of analysis. [Paper; Codes]

[Acknowledgement]  Special thanks to Dr. Fourtassi (Abdellah), whose expertise was invaluable in formulating the research questions and methodology. 

Conversation is a coordination activity, which generally involves collaborative multimodal signaling (e.g. gesture, eye gaze and intonation change) to achieve shared understanding. The current project aims to predict children’s encoding (producing cues to exhibit engagement or request feedback) and decoding (detecting backchannel or backchannel-inviting cues) abilities by leveraging recent advances in machine learning. The conversation come from the ChiCo Corpus. This project is published on ICMI Conference 2022. [Paper; Thesis; Video; Codes]

[Acknowledgement]  Special thanks to Dr. Fourtassi (Abdellah), Dr. Özyürek(Asli) Mitja and Kübra whose expertise was invaluable in formulating the research questions and methodology. 

This ongoing project aims to construct a feasible ANN model with optimal prediction accuracy of human speech adaptation by selecting modeling architecture (e.g. the number and size of neural layers) and combining adaptation techniques in automatic speech recognition (ASR). This project is presented on InterSpeech 2022. [Codes][Poster]

[Acknowledgement]  Special thanks to Dr. T. F Jaeger (Florian) whose expertise was invaluable in formulating the research questions and methodology. 

Idiomatic expressions are assumed to be represented holistically in mental lexicon (Cutting&Bock, 1997). Such hypotheses need to be complemented with data that show the pathway of activation during normal speech production and whether the word-like representation is affected by language proficiency. This project aims to explore the difference of Dutch idiom speech production of native and L2 German speakers using eGeMAPS feature set and RFE. The speech data come from ISLA project. This project is presented on LREC 2022.        [Codes][Slide]

[Acknowledgement]  Special thanks to dr. W.A.J. Strik (Helmer) for his support of research design and methodology.

This project aims to construct a plausible computational method of Chinese words’ orthographic differences for the simulation of Multilink model (Dijkstra et al., 2018). Unlike alphabetic system for which Levenshtein distance is used as the measure of orthographic similarity (see Schepens, Dijkstra & Grootjen, 2012), Chinese characters’ visual similarity is more complex due to its ideographical feature. Therefore, we compared three computational methods(dHash code, Siamese CNN and Unicode) with lexical decision RT benchmarking. [Paper; Codes]

[Acknowledgement]  Special thanks to Prof. A.F.J. Dijkstra (Ton) whose expertise was invaluable in refining methodology. 

This ongoing project aims to explore the effect of ‘wakeful rest’ on memory consolidation during L2 incidental word learning. Wakeful rest (WR) is a short period of time in which people receive minimal stimulation while still staying awake, the mnemonic effect of which has been proved especially for intentional learning. However, much less is known about such influence of wakeful rest on incidental learning. This project uses a price-comparison paradigm that approximates naturalistic L2 learning in which Dutch native speakers learn English words incidentally. After a pretest to exclude known words, a pricing-comparison dialogue task is conducted to incidentally teach participants the words. Then a short period of wakeful rest or a visual task is performed for experimental or control group. Finally, an immediate post-test are conducted to test the word retention.

[Acknowledgement]  Special thanks to Dr K.M. Lemhöfer (Kristin) whose expertise was invaluable in formulating the research questions and methodology. 

English phraseology abounds with rhymed idioms (e.g. walk the plank), which suggests their comparative advantage to become stock phrases. One plausible explanation for this advantage is that rhymed idioms are relatively memorable although there is little directly pertinent empirical evidence. Therefore, in the this project, I investigate whether rhyme has a potential to facilitate memorization with or without awareness raising. Then I further explore different impacts of encoding types (meaning-focused v.s. form-focused) on idiom memory.                                                                      The experiment results indicate that the presence of rhyme makes the form of English idioms relatively easy to recall, the effect of which can be synergistically promoted by explicit instruction in the learning process. Also, while the form-focused encoding may enhance implicit memory, the meaning-focused encoding can promote semantic memory.    [Paper; POSTER; Codes]

[Acknowledgement]  Sincerest thanks are owed to Dr. Yang (Yanning) for his support of research design.

This project aims to minimize the amount of labeled data required while maximizing the effectiveness (increase per iteration) of the text classification model. This project has been selected in the final list of the Hackthon 2023 Data science Contest.

This project aims to evaluate and optimize the input features for tone classification , which can inspire the improvement of current automatic classification of Mandarin tones and more importantly, lower computation cost by selecting the optimal input feature vectors. The results suggest the following indications:   

(1) Conventional lower-order MFCCs are sufficient for tone classification unlike conventional ASR studies. Rather, higher-order cepstral coefficients (37-48) carry more tone-related information.                                          

(2) The MFCCs are highly influenced by gender difference. So it is wise to incorporate other features in training.    

(3) Compared with MFCCs, normalized pitch feature is more stable in terms of gender generalizability, especially for log transformed z-score.   [Paper; Poster; Codes]

[Acknowledgement]  Special thanks to Dr L.F.M. ten Bosch (Louis) whose expertise was invaluable in formulating the research questions and methodology. 

Among the tasks in an automated grading system, the detection of off-topic essays is challenging because previous systems tended to be “cheated” by well-organized but content-unrelated text, i.e. off-topic essays. This project proposes an innovative approach for off topic essay detection by converting the similarity between the on topic sample essay and the target essay into similarity grids on word level, and then classifying the extent of topic relevance with residual neural networks (ResNet). This novel method achieved F1 scores of 92% and 95.8% based on different semantic features, which was substantially higher than the F1 score of a baseline system Random Forest Classifier (83.7%). [Paper; Codes]

This project focuses on the search strategies and decision trees. [Codes]