Educational Measurement

Overview

I started my new adventure in educational measurement pretty since the summer of 2013 after I joined in ETS. My current research centers on collaborative problem solving, game and simulation-based assessment, educational data mining & analytics, natural language processing and automated scoring. I am currently leading the computational psychometrics subinitiative under the FASP initiative at ETS, and has been co-leading the infrastructure subinitiative of the game, simulation and collaboration initiative from 2014 to 2016. I am leading several research projects at ETS for designing simulation-based assessments, web-based platform for collaborative assessments and data analytics packages for game-based assessments.

Educational data mining and analytics

game/simulation provides a complex digital environment that allows us to measure some skills that cannot be measured via traditional approach. On the other hand, the increased space will also lead to divergent responses, which poses a challenge for scoring the performance fairly. Some of the scoring rules can be designed during the designing of the tasks, but there are additional scoring elements that cannot be foreseen during the designing phase and can only be obtained via data mining after getting the actual data. By combining the two, we can improve the reliability and validity of the assessment instrument. Here are some papers I author/co-authored so far on this topic. I also included links to my presentations on data mining and big data.

    • Zapata, D., Liu, L., Chen, L., Hao, J., & von Davier, A (2017), Assessing Science Inquiry Skills in Immersive, Conversation-based Systems. Big Data and Learning Analytics: Current Theory and Practice in Higher Education, Daniel, B. & Butson, R. (Eds). Springer
    • Hao, J., Smith, L., Mislevy, R., Davier, A., & Bauer, M. (2016). Taming Log Files From Game/Simulation‐Based Assessments: Data Models and Data Analysis Tools. ETS Research Report Series.
    • Hao, J., Shu, Z., & von Davier, A. (2015). Analyzing Process Data from Game/Scenario-Based Tasks: An Edit Distance Approach. JEDM-Journal of Educational Data Mining, 7(1), 33-50.[PDF]
    • Mislevy, R. J., Oranje, A., Bauer, M. I., von Davier, A., Hao, J., Corrigan, S., Hoffman, E. DiCerbo, K., & John, M. (2014). Psychometric considerations in game-based assessment. GlassLab, Redwood City CA. [PDF]
    • Hao, J., (2014, June). Data mining in a nutshell, Presentation given to ETS summer interns. [PDF]
    • Hao, J., (2014, September), Big data vs. right data: implications for educational assessment, Presentation given to the ETS big data series, [PDF]

Collaborative problem solving

CPS is one of the critical skills of the 21st century but is very difficult to measure. I am leading a series of projects at ETS to develop tasks and platforms to make such assessments possible. In the following, I listed one presentation and two papers from our current project. More are coming.

    • Liu, L., Hao, J., Andrews, J., Zhu, M., Mislevy, R., Kyllonen, P., et al, (2017), Collaborative Problem Solving: Innovating Standardized Assessment, Proceedings of CSCL 2017, Philadelphia, PA.
    • Hao, J., Liu, L., von Davier, A. A., & Kyllonen, P. C. (2017). Initial steps towards a standardized assessment for CPS: Practical challenges and strategies. In A. A. von Davier, M. Zhu, & P. C. Kyllonen (Eds.), Innovative Assessment of Collaboration. New York: Springer
    • Andrews, J. J., Kerr, D., Mislevy, R. J., Davier, A., Hao, J., & Liu, L. (2017). Modeling Collaborative Interaction Patterns in a Simulation-Based Task. Journal of Educational Measurement, 54(1), 54-69.
    • Halpin, P. F., von Davier, A. A., Hao, J., & Liu, L. (2017). Measuring Student Engagement During Collaboration. Journal of Educational Measurement, 54(1), 70-84.
    • Hao, J., Liu, L., von Davier, A., Kyllonen, P., & Kitchen, C., (2016). Collaborative problem-solving skills versus collaboration outcomes: findings from statistical analysis and data mining, Proceedings of the 9the International Conference on Educational Data Mining, Durham, NC.
    • Hao, J., Liu, L., von Davier, A. & Lederer, N., (2016), EPCAL: ETS Platform Form Collaborative Assessment and Learning, Extended abstract and poster presentation at the Collective Intelligence 2016
    • Hao, J., Liu, L., von Davier, A., & Kyllonen, P. (2015), Assessing collaborative problem solving with simulation based tasks, proceeding of 11th international conference on computer supported collaborative learning, Gothenburg, Sweden [PDF]
    • Liu, L., Hao, J., von Davier, A., Kyllonen, P., & Zapata-Rivera, D. (2015). A tough nut to crack: Measuring collaborative problem-solving. To appear in Y. Rosen, S. Ferrara, & M. Mosharraf (Eds). Handbook of Research on Computational Tools for Real-World Skill Development. Hershey, PA: IGI-Global.
    • Luna Bazaldua, D. A., Khan, S., von Davier, A. A., Hao, J., Liu, L.,& Wang, Z. (2015) On Convergence of Cognitive and Noncognitive Behavior in Collaborative Activity. Proceedings of the 8th International conference on educational data mining, Madrid, Spain.
    • Hao, J., 2015, Assessing collaborative problem-solving skills: challenges and findings, presentation was given to 2015 interns [PDF]

Data model and analytics for virtual performace assessment (e.g., games, simulations, etc)

the new item types, such as games and simulations, generate a lot of process data during the assessment. These time-stamped data need to be properly recorded to facilitate the evidence identification later on. However, there is no well-established data model for this purpose so far and there is a lack of proper analysis tools for handling these types of data for assessment purpose. I am leading a project to develop a generic data model for the log files and also to develop a suite of functionalities to analyze the data. Here are some publications in this regard.

    • Hao, J., Smith L., Mislevy, R., von Davier, A., & Bauer, M., (2016). Taming log files from the game and simulation-based assessment: Data model and data analysis tool. ETS Research Report RR-16-11. Princeton, NJ: Educational Testing Service.
    • Hao, J., Smith, L., Mislevy, R., & von Davier, A. (2014). Systems and methods for designing, parsing, and mining of game log files, U.S. patent application # 14/527,591

NLP and automated annotation/scoring:

Natural language processing plays an important role in automating the process of scoring/tagging some essays or verbal responses for OE items. Generally, we want to create a mapping between text representations (e.g., n-gram, or word/paragraph vector) and the labels/scores created by human raters based on certain rubrics.

    • Flor, M., Yoon, S. Y., Hao, J., Liu, L., & von Davier, A. A. (2016). Automated classification of collaborative problem-solving interactions in simulated science tasks. In 11th Workshop on Innovative Use of NLP for Building Educational Applications, San Diego, California.
    • Hao, J., Chen, L., Flor, M., Liu, L., & von Davier, A.A.(in press) CPS-rater: automated sequential annotation for conversations in collaborative activities, ETS Research Report

Keystroke mining:

Recording the writing process may reveal many things that cannot be obtained from the end-product essay. I proposed a hierarchical vectorization method to quantify the keystroke information and obtained very interesting results concerning the writing styles. This is work in-progress and some publications are appearing

    • Zhang, M., Hao, J., Li, C., & Deane, P. (2016). Classification of Writing Patterns Using Keystroke Logs. In Quantitative Psychology Research (pp. 299-314). Springer International Publishing.