Title: "Human-AI Interaction in Analytic Essay Scoring: Investigating Rater Processes through Eye-Tracking"

Authors: Gabrielle Gaudeau, Hongyi Yang, Andreas Säuberli, Andrew Caines

Abstract: As large language models (LLMs) increasingly support human scoring in writing assessment, key questions emerge: How do raters engage with AI-generated input? Do they defer to these suggestions, critically evaluate them, or disregard them altogether? Which analytic traits are more susceptible to AI influence, and how might this reshape scoring cognition? This study investigates how experienced raters respond to LLM-suggested analytic scores during essay evaluation. Using a mixed-methods design, we combine eye-tracking with interviews to explore rater cognition in real time. Thirty experienced raters participated in two 2h-long sessions, evaluating a total of 30 essays drawn from the ELLIPSE corpus. A split-screen interface displayed the analytic rubric (left) and the essay plus score entry (right). In the AI-assisted phase, rubric bands aligned with AI-suggested scores were visually highlighted. This study contributes to ongoing conversations on scoring consistency and fairness in human–AI collaboration. It offers timely insights for designing rater training and developing responsible assessment policy as co-scoring becomes more prevalent.