eRevise+RF Project

eRevise+RF: An automated writing evaluation system to support text-based argumentation and revision

Using Natural Language Processing for Scoring Writing and Providing Feedback At-Scale

Our project studies the use of Natural Language Processing (NLP) techniques to score students' argument writing and provide automated feedback to students to strengthen their essays. We have recently expanded our project to focus on automated assessment of students' revision efforts with feedback to develop their revision skills. Our work has been funded by the National Science Foundations, the Department of Education's Institute of Education Sciences, and the Learning Research & Development Center at University of Pittsburgh. A summary of some of our research contributions is provided in the research brief published by the RAND Corporation. Further detail about our research is located below in the Publications section.

eRevise+RF project has been launched here https://erevise.lrdc.pitt.edu/.

NAACL Poster V5.pdf

People

We are an interdisciplinary team of researchers from computer science and education, with expertise in natural language processing, literacy instruction, quantitative and qualitative methods and policy.

Publications

Our publications span investigations of the technical quality of our scores, fairness evaluations, connections of our system to learning theory, and validity investigations evaluating the relationship of our scores to multiple measures of classroom instruction.

2025

Zhexiong Liu and Diane Litman. Efficient Layer-wise LLM Fine-tuning for Revision Intention Prediction. Findings of The Empirical Methods in Natural Language Processing (EMNLP), Suzhou, China, November 2025.
Zhexiong Liu, Diane Litman, Elaine Wang, Tianwen Li, Mason Gobat, Lindsay Clare Matsumura, Richard Correnti. eRevise+RF: A Writing Evaluation System for Assessing Student Essay Revisions and Providing Formative Feedback. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), System Demonstrations, pages 173–190, Albuquerque, New Mexico, 2025
Tianwen Li, Lindsay Clare Matsumura, Elaine L Wang, Zhexiong Liu, Diane Litman, Richard Correnti. Designing Writing Feedback with Large Language Models: Developing Young Adolescents' Conceptual Understanding of Effective Revision. Proceedings of The International Society of the Learning Sciences (ISLS) Annual Meeting, June 2025 (Poster).
Tianwen Li, Lindsay Clare Matsumura, Elaine L Wang, Diane Litman, Richard Correnti, Zhexiong Liu. Leveraging large language models to fostering young students' metacognition of revision. In American Educational Research Association (AERA) Annual Conference, April 2025 (Abstract).
Elaine L Wang, Lindsay Clare Matsumura, Zhexiong Liu, Tianwen Li, Richard Correnti, Diane Litman. Elementary Students’ Revision Moves in Text-Based Argument Essay Writing: A Corpus Study. In American Educational Research Association (AERA) Annual Conference, April 2025 (Abstract).

2024

Tianwen Li, Zhexiong Liu, Lindsay Clare Matsumura, Elaine Wang, Diane Litman, Richard Correnti. Using Large Language Models to Assess Young Students' Writing Revisions. Proceedings of the 19th Workshop on Innovative Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 365–380, Mexico City, Mexico, 2024
Richard Correnti, Elaine Lin Wang, Lindsay Claire Matsumura, Diane Litman, Zhexiong Liu, Tianwen Li. Supporting Students’ Text-based Evidence Use via Formative Automated Writing and Revision Assessment. The Routledge International Handbook of Automated Essay Evaluation, 2024.

2023

Zhexiong Liu, Diane Litman, Elaine Wang, Lindsay Matsumura, and Richard Correnti. Predicting the Quality of Revisions in Argumentative Writing. Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 275–287, Toronto, Canada, 2023.

2022

Matsumura, L.C., Wang, E.L., Correnti, R., & Litman, D. (2023). Tasks and feedback: An exploration of students’ opportunity to develop adaptive expertise for analytic text-based writing. Assessing Writing, 55. doi: https://doi.org/10.1016/j.asw.2022.100689
Correnti, R., Matsumura, L.C., Wang, E.L., Litman, D., & Zhang, H. (2022). Building a validity argument for an automated writing evaluation system (eRevise) as a formative assessment. Computers & Education Open. doi: https://doi.org/10.1016/j.caeo.2022.100084
Wang, E.L., Matsumura, L.C., Litman, D., & Correnti, R. (2022). Contributions to research on automated writing scoring and feedback systems (RB-A1062-1). Santa Monica, CA: RAND Corporation. doi: https://doi.org/10.7249/RBA1062-1

2021

Diane Litman, Haoran Zhang, Richard Correnti, Lindsay Matsumura and Elaine Wang, A Fairness Evaluation of Automated Methods for Scoring Text Evidence Usage in Writing, Proceedings of 22nd International Conference on Artificial Intelligence in Education (AIED), June, 2021.
Haoran Zhang and Diane Litman, Essay Quality Signals as Weak Supervision for Source-based Essay Scoring, 16th Workshop on Innovative Use of NLP for Building Educational Applications (BEA), pp. 85-96, April, 2021.

2020

Tazin Afrin, Elaine Lin Wang, Diane Litman, Lindsay Clare Matsumara, and Richard Correnti, Annotation and Classification of Evidence and Reasoning Revisions in Argumentative Writing, Proceedings of the 15th Workshop on Innovative Use of NLP for Building Educational Applications, July, 2020.
Correnti, R., Matsumura, L.C., & Wang, E.L., Litman, D., Rahimi, Z., & Kisa, Z. "Automated scoring of students’ use of text evidence in writing". Reading Research Quarterly, 55(3), 2020.
Elaine Lin Wang, Lindsay Clare Matsumura, Richard Correnti, Diane Litman, Haoran Zhang, Emily Howe, Ahmed Magooda, Rafael Quintana, eRevis(ing): Students' Revision of Text Evidence Use in an Automated Writing Evaluation System, Assessing Writing 44, 2020.
Haoran Zhang and Diane Litman, Automated Topical Component Extraction Using Neural Network Attention Scores from Source-based Essay Scoring, Proceedings of the 58th Annual meeting of the Association for Computational Linguistics (ACL), July, 2020.

2019

Zhang, H., Magooda, A., Litman, D., Correnti, R., Wang, E., Matsmura, L. C., ... & Quintana, R. "eRevise: Using Natural Language Processing to Provide Formative Feedback on Text Evidence Usage in Student Writing". Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, pp. 9619-9625), at IAAI Honolulu, HI, February, 2019.

2018

Haoran Zhang and Diane Litman, "Co-Attention Based Neural Network for Source-Dependent Essay Scoring," Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, at NAACL New Orleans, LA, June, 2018.

2017

Haoran Zhang and Diane Litman, "Word Embedding for Response-To-Text Assessment of Evidence", Proceedings Student Research Workshop of the Annual Meeting of the Association for Computational Linguistics, pp. 75-81, Vancouver, Canada, July, 2017.
Rahimi, Z., Litman, D., Correnti, R., Wang, E. and Matsumura, L.C., "Assessing students’ use of evidence and organization in response-to-text writing: Using natural language processing for rubric-based automated scoring". International Journal of Artificial Intelligence in Education, 27(4), pp.694-728. 2017

2016

Zahra Rahimi and Diane Litman, "Automatically Extracting Topical Components for a Response-to-Text Writing Assessment", Proceedings 11th Workshop on Innovative Use of NLP for Building Educational Applications (NAACL Workshop), pp. 277-282, San Diego, CA, June, 2016. (short paper)

2015

Zahra Rahimi, Diane Litman, Elaine Wang and Richard Correnti, "Incorporating Coherence of Topics as a Criterion in Automatic Response-to-Text Assessment of the Organization of Writing", Proceedings 10th Workshop on Innovative Use of NLP for Building Educational Applications (NAACL Workshop), pp 20-30, Denver, Colorado, June, 2015.

2014

Zahra Rahimi, Diane Litman, Richard Correnti, Lindsay Clare Matsumura, Elaine Wang and Zahid Kisa, "Automatic Scoring of an Analytical Response-To-Text Assessment", Proceedings 12th International Conference on Intelligent Tutoring Systems (ITS), pp. 601-610, Honolulu, HI, June, 2014.
Correnti, R., Matsumura, L. C., Hamilton, L., & Wang, E. "Assessing students' skills at writing analytically in response to texts", The Elementary School Journal, 114(2), pp.142-177. 2014.

Screenshots of eRevise(+RF) Systems

The Architecture of eRevise+RF (New)

The eRevise+RF system is built on the prior eRevise system. It focuses on analyzing revisions in text-based argument essays. After students submit their first and second drafts, we score the quality of students' revisions with eRevise+RF's AES system and provide formative feedback on their use of text evidence. Students could improve their third draft based on the automated feedback messages.

The architecture of eRevise (Prior System)

After students submit their first drafts, eRevise's AES component extracts features representing the quality of text-based evidence usage in terms of constructs in the RTA Evidence rubric. Some of these features are then passed as input to the AWE system's feedback selection algorithm, which will in turn output a subset of predefined feedback messages that are believed to best address the problems of the first draft based on the features.

The first phase of eRevise

Students write their first drafts via Qualtrics Survey system.

The second phase of eRevise

Students revise their drafts via eRevise system with helpful feedback selected by the automatic system.

Software & Code

Leveraging ChatGPT to Predict Revision Quality

We study the relationship between Argument Contexts (ACs) and Argument Revisions (ARs) in argumentative writing. We use Chain-of-Thought prompts to facilitate ChatGPT to generate ACs for identifying successful vs. unsuccessful ARs. We show ChatGPT does help predict revision quality.

View on Github

Co-Attention Based Source-Dependent Essay Grading

We use a co-attention mechanism to help the model learn the importance of each part of the essay more accurately. Also, this paper shows that the co-attention based neural network model provides reliable score prediction of source-dependent responses.