Embedded Files
The Human in the Loop:
Computer Laboratory (Cambridge) & Hybrid | May 21 (Half-day)
Program
Program
Workshop Schedule
Workshop Schedule
Where? Lecture Theatre 2, Computer Laboratory (William Gates Building, 15 JJ Thomson Ave, Cambridge CB3 0FD, United Kingdom). See how to get there.
Schedule:
13:00 – 13:30: Registration (welcome drinks)
13:30 – 13:40: Introduction
13:40 – 16:00: Invited Speaker Presentations (4 speakers)
16:00 – 17:00: Poster Session & Networking (coffee break)
17:00 – 17:50: Moderated Panel Discussion ("The future of AI in assessment") + Q&A
17:50 – 18:00: Closing Remarks
____________________________________________________________________________________________________________________________________________________________________
19:30 - 21:00: Evening social dinner at Browns Restaurant (limited capacity, get in touch ASAP to confirm your spot).
Invited Speakers
Invited Speakers
Speaker order:
13:40 – 14:10: Dr. Mark Brenchley
14:15 – 14:45: Dr. Sha Liu
14:50 – 15:20: Dr. Chanjin Zheng
15:25 – 15:55: Dr. Carla Pastorino
Dr. Mark Brenchley, Cambridge University Press & Assessment
Dr. Mark Brenchley, Cambridge University Press & Assessment
Principal AI Evaluation Manager
Principal AI Evaluation Manager
Mark Brenchley is Principal AI Evaluation Manager at Cambridge University Press & Assessment. A member of the Applied AI Capabilities team, he helps oversee the evaluation of AI-enabled automarking of Second Language Speaking and Writing. He obtained his PhD in Education from the University of Exeter, where he specialised in the linguistic development of children’s speaking and writing in the UK education system.
Talk: "Maximising the Humans in the Loop: Managing the Interactions between Human and Automated Scores"
Abstract: Properly viewed, humans are inextricably part of the AI assessment loop. After all, quite apart from any decision to involve humans operationally, human scores are what we both train on and evaluate against. Seen in this light, the question is less a normative one of whether humans should be in the loop, so much as identifying where and how it’s possible to maximise the value of human scoring to help ensure high quality automarking. In this talk, I will discuss three examples of how we have been looking to do just that in our automarking work, targeting three different interactional contexts: training, evaluating, and live marking.
Webpage: https://www.linkedin.com/in/mark-brenchley-943830280/
Dr. Sha Liu, British Council
Dr. Sha Liu, British Council
Test Development Researcher
Test Development Researcher
Dr. Sha Liu is a Test Development Researcher at the British Council. She holds a PhD in Language Assessment from the University of Bristol, and her research focuses on AI-mediated speaking and writing assessment, with particular emphasis on multimodal speaking assessment using generative AI, automated diagnostic feedback for language learners, and eye-tracking methodologies for understanding learner engagement. Her commitment to inclusive assessment practices centres on developing AI-based systems that provide equitable feedback across diverse learner backgrounds.
Dr. Liu is Co-Convenor of the EALTA AI for Language Assessment Special Interest Group, and serves on editorial boards for Language Assessment Quarterly, Assessing Writing, Research Synthesis in Applied Linguistics and Artificial Intelligence in Language Education.
Talk: "What Are We Assessing When AI Plays Multiple Roles in the Loop?"
Abstract: What roles can AI play in a language assessment? How does each role reshape what is being assessed? And what is required of the human in the loop to keep the construct intact?
Bridging research and practice, this talk explores the answers to these three questions. Research in the past few years has shown that AI in current assessment practice operates in at least three roles: as task and content generator, as interlocutor, and as evaluator (encompassing scoring and feedback provision). Across these three established roles, however, AI-mediated assessment has so far focused predominantly on the product of performance rather than the process leading to it. This product-oriented use of AI risks validity through both construct underrepresentation and construct-irrelevant variance, and increasingly through diminished ecological validity, as learners now routinely co-produce language with AI in real-world communication. Researchers have therefore begun to call for the use of AI as a scaffolder or mediator during the task itself, a fourth role that remains relatively underexamined. Each of the four roles reshapes what is assessed in different ways; each demands a different kind of expert judgement from the assessment professional working alongside it.
To illustrate this evolving landscape, the talk walks through two paired practical cases: AI as interlocutor in speaking, and AI as evaluator in writing. In each, the features an assessment is meant to capture can be sharpened or distorted by prior design choices (e.g., model selection, configuration, and prompting) in relation to the intended learners. Locating both cases within the key layers of construct alteration in the assessment development process, the talk argues that closing the gap between what each AI role can deliver and what the construct requires calls for the AI-literate domain expert. It closes by discussing the kind of AI literacy that assessment professionals need to build to keep the construct intact when AI is in the loop across assessment stages.
Webpage: https://www.britishcouncil.org/english-assessment/english-language-research/meet-researchers/sha-liu
Dr. Chanjin Zheng, East China Normal University
Dr. Chanjin Zheng, East China Normal University
Associate Professor of Education, Psychology & Computer Science
Dr. Chanjin Zheng is an Associate Professor of Education, Psychology and Computer Science at East China Normal University, where he serves as Assistant to the Dean of the Shanghai Institute of Intelligent Education. He earned his Ph.D. in Educational Psychology from UIUC in 2015. His research focuses on the intersection of AI, learning, and educational assessment, particularly LLM-driven assessment systems and "Psychometrics for AI". His work is widely published in top-tier venues, including CHI, ACL, SIGKDD, Behavior Research Methods, and Information Processing and Management.
Talk: "Resilient System Design Approach to AI-driven Assessment and Learning Systems"
Abstract: The traditional paradigm of standardized assessment is facing a foundational crisis in the age of generative AI. The challenge extends far beyond the automation of essay writing; it represents a profound shift toward "cognitive outsourcing," where the delegation of critical thinking to AI tools risks bypassing the human learning process and compromising systemic integrity. This environment of radical uncertainty necessitates a "resilient" approach to system design, ensuring that assessment remains a meaningful driver of cognitive engagement rather than a mere target for optimization.
This presentation addresses these challenges by redefining "Assessment as a Design Science." Drawing on the foundational work of Herbert Simon, we view assessment not as a post-hoc measurement tool, but as a generative engineering process tailored for solving complex educational problems.
To establish a cohesive logical framework, the talk synthesizes Simon’s design science with Niklas Luhmann’s Social Systems Theory and the principles of second-order cybernetics. This intersection gives rise to "Resilient Design" (second-order design) specifically applied to assessment, marking a logical leap from first-order control to systemic evolution. Within this framework, the focus of assessment shifts from the direct verification of static outputs to architecting interaction rules and recursive feedback loops where teachers, students, and AI agents co-evolve. While the primary emphasis is on restructuring assessment to maintain its systemic integrity, this logic naturally extends into a learning paradigm.
Webpage: https://scholar.google.com/citations?user=OXSQU8EAAAAJ&hl=en
Dr. Carla Pastorino-Campos, Cambridge University Press & Assessment
Dr. Carla Pastorino-Campos, Cambridge University Press & Assessment
Principle Research Manager
Carla Pastorino-Campos is a Principal Researcher at Cambridge University Press & Assessment, where she leads research on topics related to justice, equity, diversity and inclusion and the validity of emerging technologies in language assessment. Her recent work explores how AI systems used for language assessment purposes can be designed and validated to support human expertise while maintaining ethical standards and advancing fairness and inclusion. Carla obtained her PhD in Theoretical and Applied Linguistics from the University of Cambridge, where she specialised on the psychological aspects of language learning and processing.
Talk: "A framework for ethical AI in language assessment"
Abstract: The rapid integration of artificial intelligence (AI) into language assessment is transforming how language tests are designed, delivered, and scored, and raising critical questions about how the technology can be integrated while safeguarding assessment validity and fairness. In this talk, I will present a comprehensive, domain-specific framework for the ethical use of AI in language assessment, developed through a systematic analysis of global AI policy documents and educational guidelines and extended to address the particular demands of assessment contexts. The framework consists of ten interrelated principles - including human control of technology, human-centricity, fairness and justice and assessment standards – supported by a set of detailed considerations that guide the design, development, implementation and evaluation of AI systems.
In the paper and presentation, we will argue for an ethical-by-design approach, which embeds ethical values across the AI system lifecycle and highlights the need for human-AI collaboration. In this perspective, AI is not positioned as a substitute for human expertise but as a complementary tool that enhances decision-making, where human oversight, review, and accountability remain essential, particularly in high-stakes contexts. We will further examine emerging tensions inherent in deploying AI in assessment, which often arise when attempting to implement ethical frameworks in highly applied environments.
Webpage: https://www.cambridgeenglish.org/english-research-group/meet-the-team/carla-pastorino/
Page updated
Report abuse