Kadriye Ercikan

Vice President of Research, Educational Testing Service

President, CEO, ETS Canada Inc.

Professor Emerita, Faculty of Education, University of British Columbia

Kadriye Ercikan is responsible for ETS’s foundational and applied research and ETS’s contract for the National Assessment of Educational Progress, PISA, PIAAC and PARAKH contracts. Her research focuses on validity and fairness issues and sociocultural context of assessment. Her recent research includes validity and fairness issues in innovative digital assessments including using response process data, AI applications, and adaptivity.

Ercikan is a Fellow of the International Academy of Education, President Elect of the International Academy of Education, and President Elect of the International Test Commission. Her research has resulted in six books, four special issues of refereed journals and over 100 publications. She was awarded the AERA Division D Significant Contributions to Educational Measurement and Research Methodology recognition for another co-edited volume, Generalizing from Educational Research: Beyond Qualitative and Quantitative Polarization, and received an Early Career Award from the University of British Columbia. Ercikan is currently serving as the NCME Book Series Editor (2021-2026).

Plenary Speaker Saturday 10:30 AM

Optimizing Validity and Fairness of Artificial-Intelligence-Based Automated Scoring

Artificial intelligence (AI) based automated scoring is now widely used in language testing for scoring written text as well as speech. My presentation will focus on validity and fairness of interpretation and uses of scores from such assessments. I will highlight task design, construct representation, human involvement and quality evaluations as critical to validity and fairness. The starting place for validity and fairness of interpretation of scores is the design of tasks and the degree to which performance on tasks can provide relevant and adequate evidence of the targeted construct. The first part of my presentation will focus on discussing an evidence-centered design (ECD) framework for designing assessments for targeted populations, claims, inferences, interpretations and with limitations of AI-based scoring in mind. Central to the validity of interpretation of AI generated scores as indicators of the targeted constructs is the evaluation of quality of scores including determining the degree to which AI generated scores are consistent with those from human raters and for different gender, cultural and language groups. The second part of my presentation will provide a description and discussion of the methodologies used at ETS in the evaluations of the quality of scores generated by AI algorithms. This will be followed by a discussion of future directions including the role of AI based automated scoring in personalization, simulation based, and “test free” assessments and the impact on validity and fairness of score interpretations.