National Tutoring Observatory’s Interactive Workshop on AI Annotation and Teacher Analytics 

As generative AI tutors become increasingly common in educational settings, understanding how their emotional communication patterns compare to human tutors is critical. This study applies sentiment analysis to tutoring transcripts from UPchieve, a free online tutoring platform, comparing sessions with an AI tutor (N=787) to matched human tutor sessions. We find that AI tutors display significantly higher positive sentiment (M=0.601) than human tutors (M=0.197, d=2.13), creating large sentiment gaps with students. While human tutor sessions show converging sentiment trajectories (54\%), AI sessions more often diverge (54\%). Critically, large sentiment gaps in AI sessions predicted negative student sentiment change, whereas human sessions showed no such relationship. These findings suggest that AI tutors may benefit from calibrating emotional tone to better match student affect rather than maintaining consistently high positivity.

This workshop examines how Third Space Learning is approaching the transition from human tutoring to AI tutoring by identifying the interaction patterns that support student thinking. The focus is on a central question: which sequences of tutor moves are most effective at eliciting evidence of student reasoning and construction of understanding? Using a set of human and AI tutoring sessions, participants will apply a tutor-move taxonomy and develop a complementary prompt to classify the level of reasoning visible in student responses. This will allow the group to compare whether human tutoring currently elicits richer reasoning than AI tutoring and to identify tutor-move sequences that may be important to preserve or adapt as tutoring systems become more automated. The workshop is intended to inform how AI tutors can move beyond isolated responses toward more effective sequences of pedagogical support.

Timely and accurate identification of student misconceptions is key to improving learning outcomes and pre-empting the compounding of student errors. However, this task is highly dependent on the effort and intuition of the teacher. In this work, we present a novel approach for detecting misconceptions from student-tutor dialogues using large language models (LLMs). First, we use a fine-tuned LLM to generate plausible misconceptions, and then retrieve the most promising candidates among these using embedding similarity with the input dialogue. These candidates are then assessed and re-ranked by another fine-tuned LLM to improve misconception relevance. Empirically, we evaluate our system on real dialogues from an educational tutoring platform. We consider multiple base LLM models including LLaMA, Qwen and Claude on zero-shot and fine-tuned settings. We find that our approach improves predictive performance over baseline models and that fine-tuning improves both generated misconception quality and can outperform larger closed source models. Finally, we conduct ablation studies to both validate the importance of our generation and reranking steps on misconception generation quality.

Wide-scale analyses of utterance-based conversational data require scalable methods for annotating behaviors of interest. With gold standard annotations on a subset of the full sample, language models can be used to extend human expertise and annotate behaviors of interest across a full dataset. However, development of gold standard samples typically involves an iterative process of developing a schema or codebook for annotation, having humans annotate conversations, reviewing disagreements, and updating the schema. We introduce a framework and software that leverages language models to automate the annotation step in this process, allowing researchers to focus on defining and clarifying behaviors of interest. In particular, we propose a session-by-session procedure where researchers propose a measure of interest within a set of tutoring transcripts, language models annotate these within a single conversation, the reviewer approves annotations and reviews disagreements between language models, and then make changes to the schema. In addition, by use of text embeddings, we are able to conduct a guided search over the space of schema changes and automatically generate proposals for schema changes, which the researcher may approve or deny. Our framework and software provide a tool for education researchers conducting analyses of conversational data to rapidly develop schemas describing any behaviors of interest within conversational data. Furthermore, by deriving our classifier from the current state of the schema in real time, our method narrows the gap between schema and classifier—once a researcher is satisfied by the semantic and predictive accuracy of a schema of their behavior of interest, they immediately have a classifier that can be extended to the entire dataset. Our work enables education researchers to shape analyses around their own behaviors of interest rather than deferring to which behaviors have gold-standard datasets.

Human tutoring can meaningfully increase student engagement and persistence, yet it remains unclear which specific tutor moves are most effective during brief conversational interventions. Prior work combining dialogue transcripts with learning analytics has identified correlations between certain forms of tutor support, such as stepwise scaffolding, and subsequent gains in student engagement. However, much of this analysis relies on manual coding and relatively coarse categorizations of dialogue. At the same time, online tutoring environments generate rich multimodal data, including dialogue, screen activity, and system log traces, that capture finer-grained aspects of tutor-student interactions. These data open new opportunities to move beyond predefined coding schemes and instead discover patterns of effective tutor moves in a more data driven way, leveraging complementary signals across modalities. This brief input introduces embeddings as a tool for discovering and interpreting patterns in tutor-student interactions. Using a dataset of de-identified tutoring transcripts and multimodal features from fall 2025 (including dialogue and behavioral traces from systems such as MATHia), participants will explore how embedding-based clustering can surface meaningful groupings of tutor moves. The activity centers on human interpretation: participants will examine cluster structures, compare embedding configurations, and reflect on how different modalities and hyperparameters shape the resulting groupings. Through hands-on activities, attendees will (1) experiment with clustering tutor dialogue and multimodal actions, (2) interpret emergent cluster “types” of tutor moves, and (3) consider how these groupings relate to downstream engagement outcomes. We will also discuss how embedding-based approaches can complement traditional qualitative analysis by helping researchers and practitioners rapidly identify promising interaction patterns for further study.

Large language models (LLMs) offer a scalable alternative to human coding for data annotation tasks, enabling the scale-up of research across data-intensive domains such as learning analytics. While LLMs are already achieving near-human accuracy on objective annotation tasks, their performance on subjective annotation tasks, such as those involving psychological constructs, is less consistent and more prone to errors. Standard evaluation practices typically collapse all annotation errors into a single alignment metric, but this simplified approach may obscure different kinds of errors that affect final analytical conclusions in different ways. Here, we propose a diagnostic evaluation paradigm that incorporates a human-in-the-loop step to separate task-inherent ambiguity from model-driven inaccuracies and assess annotation quality in terms of their potential downstream impacts. We refine this paradigm on ordinal annotation tasks, which are common in subjective annotation. The refined paradigm includes: (1) a diagnostic taxonomy that categorizes LLM annotation errors along two dimensions: source (model-specific vs. task-inherent) and type (boundary ambiguity vs. conceptual misidentification); (2) a lightweight human annotation test to estimate task-inherent ambiguity from LLM annotations; and (3) a computational method to decompose observed LLM annotation errors following our taxonomy. We validate this paradigm on four educational annotation tasks, demonstrating both its conceptual validity and practical utility. Theoretically, our work provides empirical evidence for why excessively high alignment is unrealistic in specific annotation tasks and why single alignment metrics inadequately reflect the quality of LLM annotations. In practice, our paradigm can be a low-cost diagnostic tool that assesses the suitability of a given task for LLM annotation and provides actionable insights for further technical optimization.

Paper available