Workshop

Advancing the Science of Teaching with Tutoring Data: A Collaborative Workshop with the National Tutoring Observatory

Full-day Workshop as part of the 2025 ACM Learning @ Scale Conference

July 21, 2025, 9am-4pm CET at the University of Palermo, Palermo, Italy

This full-day and in-person workshop aims to bring together members of the ACM Learning @ Scale, Educational Data Mining, and AI in Education communities to share progress, identify common challenges, and explore collaborative solutions to understanding effective teaching and tutoring moves. The structure of this workshop will include presentations of accepted papers (see our Call for Papers below), a demonstration by the National Tutoring Observatory (NTO), and a panel discussion featuring key researchers in tutoring and teaching.

Date and time:

July 21, 2025, 9 am - 4 pm (Central European Time)

Location:

Room TBD, University of Palermo, Piazza Marina, 61, 90133, Palermo, Sicily

National Tutoring Observatory

The National Tutoring Observatory (NTO) is leading the creation of the Million Tutor Moves dataset—the largest open-access collection of tutoring interactions—using generative AI to accelerate the science of teaching at scale. This workshop will unite the Learning at Scale community to share progress, tackle shared challenges, and explore joint solutions. The agenda includes paper presentations, interactive demos, and a panel with researchers, developers, and tutoring providers, all aimed at advancing a shared vision for impactful, data-driven tutoring through collaborative research.

Workshop Theme

This workshop aims to facilitate discussion and engagement among the Learning at Scale community. In particular, the workshop hosts updates on progress, findings, and challenges to collecting, pre-processing, annotation, and modeling interactional datasets. The goal of this workshop is to foster discourse, exchange valuable insights, make connections with the community, and develop a potential user base of contributors and users of the Million Tutor Moves repository. We invite empirical and theoretical papers aligned with the listed themes, particularly (but not exclusively) within the following areas of research and application:

Understanding tutoring and teaching moves: Investigating in-the-moment instructional strategies, tutor-student interactions, and contextual factors that enhance learning outcomes.
Data pre-processing, standardization, and deidentification: Developing scalable methods for cleaning, structuring, and anonymizing large-scale tutoring data, ensuring interoperability, privacy compliance, and reproducibility in educational research.
Multimodal data collection and annotation: Addressing challenges in capturing, segmenting, and annotating large-scale tutoring interactions across text, audio, and video.
Predictive modeling of interactional data to learning outcomes: Exploring how human and AI-driven tutoring models influence student performance, engagement, and short- and long-term learning outcomes using machine learning and learning analytics
Data sharing, privacy, and legal frameworks: Examining ethical, legal, and policy considerations for securely sharing and protecting tutoring data while enabling open research.
Fairness, equity, and inclusion: Developing inclusive approaches, mitigating bias, and ensuring equitable access within large-scale tutoring data to promote fairness in educational AI applications.
Key Challenges: Identifying barriers, considerations, and challenges to creating a collaborative infrastructure to share data analysis workflows and analysis routines
Interoperability and scalable data management: Designing collaborative, shareable data management systems that integrate with existing educational technologies and platforms.

Target Audience & Participation

The target audience for this workshop includes researchers, educators, EdTech developers, policy makers, and leaders of tutoring organizations interested in tutoring and teaching research and AI-assisted learning. This workshop will be particularly relevant to members of the Learning at Scale community who are working at the intersection of AI and human tutoring to create individualized and effective learning experiences. Participation requirements include a device to access accepted papers, demonstrations, and materials, and an interest in advancing research on the science of learning and teaching.

Workshop Format

This will be a full-day, in-person workshop with the following:

presentations of accepted papers with Q&A
a demonstration by the NTO
a moderated panel with audience participation focused on upcoming research in this area
a closing summary and discussion session

These activities will be oriented toward the workshop goals to develop a shared understanding of the current state of tutoring and teaching data and to pose key research questions and challenges for future research. The length of presentations will be determined by the organizing committee based on the maturity of the work, level of interest among L@S, and significance to the themes. A demonstration will be presented by the NTO. A subsequent whole-group discussion with moderated panel will contain a question-and-answer period with in-person panelists. A summary of the key issues and responses from the panel discussion, along with commonalities among accepted papers, will be published in Volume 2 of the conference proceedings.

Call for Papers

The workshop will include presentations of accepted (non-archival) papers and facilitated discussion sessions. We will solicit papers relevant to the themes using the research paper format described in the conference proceedings guidelines. Papers will go through a single-blind review process, with reviewers anonymous and authors known. Reviewers will be required to make a recommendation of either acceptance or rejection of the paper and explain their reasoning behind their decision. They will assess the paper based on three criteria, using a scoring system of -1, 0, or 1; alignment with the workshop’s theme, level of interest to L@S, and overall quality. Authors of accepted papers will give presentations at the conference. Important dates for the call for papers are shown below.

Important Dates

Call for papers: April 11, 2025

Paper submission deadline: June 1, 2025 June 9, 2025 (extended)

Paper review period: June 2 - June 13, 2025

Final paper decisions: June 14, 2025

Notification of acceptance: June 15, 2025

Camera-ready deadline: June 27, 2025

Workshop day: July 21, 2025

Submission Instructions

Submit your papers on EasyChair here: https://easychair.org/conferences/?conf=lsnto25

We will solicit papers relevant to the themes using the research paper format described in the conference call for papers guidelines. Papers will go through a single-blind review process, with authors known and reviewers unknown.

Reviewers will be required to make a recommendation of either acceptance or rejection of the paper and explain their reasoning behind their decision. They will assess the paper based on three criteria, using a scoring system of -1, 0, or 1; alignment with the workshop’s theme, level of interest to L@S, and overall quality. Authors of accepted papers will provide presentations at the conference (more details to be announced).

Accepted papers are non-archival to encourage submissions of exciting and important work without concerns.

Paper reviewing and notification to authors will be handled using the workshop EasyChair account (link above). Submissions must be in PDF format. All papers must follow the ACM 2-column proceddings template (Word or Latex), written in English, contain original work, and not be under review for any other venue while under review for this conference.

Submitted papers should be between 3 - 10 pages in length, excluding references.

If you have any questions regarding the submission process, reach out to

Danielle Thomas (drthomas@cmu.edu)
Rene Kizilcec (kizilcec@cornell.edu)

Organizing Committee

(alphabetical order)

dora demszky, PhD. Stanford university

Dora is an Assistant Professor in Education Data Science at Stanford University. Her research focuses on measuring equity, representation and student-centeredness in educational texts, with the goal of providing insights to educators to improve instruction. She develops measures based on natural language processing that work well for high-dimensional, unstructured data, and she applies these measures to provide feedback to educators. Dr Demszky has received her PhD in Linguistics at Stanford.

Email: ddemszky@stanford.edu

www.dorademszky.com

Rene f. Kizilcec, PhD. Cornell university

Kizilcec is an Associate Professor in the Bowers College of Computing and Information Science at Cornell University, where he directs the Cornell Future of Learning Lab. He is a PI of the National Tutoring Observatory. Kizilcec studies behavioral, psychological, and computational aspects of technology in education to inform practices and policies that promote learning, equity, and academic and career success. Kizilcec has authored over 100 research papers, won numerous Best Paper awards, and received funding from the NSF, Schmidt Futures Foundation, Gates Foundation, Jacobs Foundation, Chan Zuckerberg Initiative, and Google.

Email: kizilcec@cornell.edu

Kenneth R. Koedinger, PhD. Carnegie Mellon University

Ken is the Hillman professor of Computer Science and Psychology at Carnegie Mellon University and founder of PLUS tutoring. He is a co-founder of CarnegieLearning, Inc. that has brought Cognitive Tutor based courses to millions of students since it was formed in 1998, and leads LearnLab, the scientific arm of CMU's Simon Initiative. Through extensive research and development in human-AI tutoring, Ken has demonstrated a doubling of math learning among middle school students and aims to bring similar high-quality tutoring that is cost-effective to scale. He has authored over 300 research papers and over 60 grant proposals.

Email: koedinger@cmu.edu

PLUS: http://tutors.plus

http://pact.cs.cmu.edu/koedinger.html.org

Josh Marland, PHD NTO

Josh is the Managing Director for the NTO. He has worked in a variety of education data and technology roles for almost 20 years. Josh is driven by finding new ways to create opportunities for those not served by traditional systems and has focused his research at the intersection of equitable measurement and policy. He earned a Master's degree from Brown University in Urban Education Policy and a Doctorate in Research, Educational Measurement, and Psychometrics from the University of Massachusetts Amherst. Josh is the first in his familty to go to college and a proud community college graduate.

Email: jm2945@cornell.edu

Doug pietrzak Freshcognate

Doug is the Technology Director of the NTO and the CEO of Freshcognate, an instructional design firm tackling the most challenging educational situations at scale. Doug is an educator at heart, as he was a former CPS English teacher focused on equity who combines his passion for technology and education to drive impactful teaching and learning experiences and products at scale. He is one of the original members of Lynda.com and EdX and was a major contributor to education at Google.

Email: doug@freshcognate.com

Justin Reich, EDD. MIT

Justin is an associate professor of digital media in the Comparative Media Studies/Writing department at MIT and the director of the Teaching Systems Lab. He is the author of Iterate: The Secret to Innovation in Schools and Failure to Disrupt: Why Technology Alone Can’t Transform Education, and he is the host of the TeachLab Podcast. He earned his doctorate from the Harvard Graduate School of Education is a past Fellow at the Berkman-Klein Center for Internet and Society. His writings have been published in Science, Proceedings of the National Academy of Sciences, Washington Post, The Atlantic, and other scholarly venues. He started his career as a high school history teacher and wrestling coach.

Email: jreich@mit.edu

Teaching Systems Lab

Rachel Slama, EDD. RAND

Rachel is the director of the labor and workforce development program and a senior policy researcher at RAND. Her work focuses on strengthening domestic STEM and emerging tech talent pipelines to promote national security and economic opportunity through the development of technology and workforce training infrastructure. Rachel previously served as co-director of the Teaching Systems Lab at MIT, a research and design lab comprised of learning scientists, engineers, and STEM education and simulation experts. A former teacher in New York City, her current focus is on promoting innovation and learning through human and AI tutoring.

Email: rslama@rand.org

Danielle R. Thomas, EdD. Carnegie Mellon University

Danielle is a Systems Scientist at Carnegie Mellon University, the Research Lead on the PLUS tutoring project, and the Interim Research Director at the NTO. She is a former middle school teacher, instructional coach, and school administrator. Danielle wishes every child had their own human math tutor. Given that's not feasbile, she is striving to improve human-AI tutoring so that its not only impactful but cost-effective and scalable. In recent years, she has first-authored over a dozen papers in conferences, such as Artificial Intelligence in Education and International Learning Analytics and Knowledge.

Email: drthomas@cmu.edu

PLUS: http://tutors.plus

LinkedIn daniellethomas.org

Amalia Toutziardi MIT

Amalia is a undergraduate researcher at the NTO and working towards her Bachelor's in Computer Science and Engineering at MIT. She is an incoming SWE intern at Microsoft and current AI Education research fellow at ReadyAI. Amalia is a student athlete in women's track and field specializing in high jump.

Email: amaliat@mit.edu

Page updated

Report abuse