Overview
I started my new adventure in educational measurement pretty since the summer of 2013 after I joined in ETS. My current research centers on collaborative problem solving, game and simulation-based assessment, educational data mining & analytics, natural language processing and automated scoring. I am currently leading the computational psychometrics subinitiative under the FASP initiative at ETS, and has been co-leading the infrastructure subinitiative of the game, simulation and collaboration initiative from 2014 to 2016. I am leading several research projects at ETS for designing simulation-based assessments, web-based platform for collaborative assessments and data analytics packages for game-based assessments.
Educational data mining and analytics
game/simulation provides a complex digital environment that allows us to measure some skills that cannot be measured via traditional approach. On the other hand, the increased space will also lead to divergent responses, which poses a challenge for scoring the performance fairly. Some of the scoring rules can be designed during the designing of the tasks, but there are additional scoring elements that cannot be foreseen during the designing phase and can only be obtained via data mining after getting the actual data. By combining the two, we can improve the reliability and validity of the assessment instrument. Here are some papers I author/co-authored so far on this topic. I also included links to my presentations on data mining and big data.
Collaborative problem solving
CPS is one of the critical skills of the 21st century but is very difficult to measure. I am leading a series of projects at ETS to develop tasks and platforms to make such assessments possible. In the following, I listed one presentation and two papers from our current project. More are coming.
Data model and analytics for virtual performace assessment (e.g., games, simulations, etc)
the new item types, such as games and simulations, generate a lot of process data during the assessment. These time-stamped data need to be properly recorded to facilitate the evidence identification later on. However, there is no well-established data model for this purpose so far and there is a lack of proper analysis tools for handling these types of data for assessment purpose. I am leading a project to develop a generic data model for the log files and also to develop a suite of functionalities to analyze the data. Here are some publications in this regard.
NLP and automated annotation/scoring:
Natural language processing plays an important role in automating the process of scoring/tagging some essays or verbal responses for OE items. Generally, we want to create a mapping between text representations (e.g., n-gram, or word/paragraph vector) and the labels/scores created by human raters based on certain rubrics.
Keystroke mining:
Recording the writing process may reveal many things that cannot be obtained from the end-product essay. I proposed a hierarchical vectorization method to quantify the keystroke information and obtained very interesting results concerning the writing styles. This is work in-progress and some publications are appearing