Beata Beigman Klebanov
Greetings!
I am a Principal Research Scientist at Educational Testing Service, Princeton, NJ. Here is my CV.
I engage primarily in two lines of work.
ONE is building language technology to encourage developing and sometimes reluctant readers and writers to actually read and write.
I am the principal investigator behind Relay Reader, a reading and listening app. You can learn more in our R&D News piece.
Our most exciting findings using Relay Reader are that (a) children like it; (b) it is possible to detect learning while kids are reading and having fun; and (c) we were able to advance the understanding of the nature of oral reading using our data.
Relay Reader got some shoutouts: It is listed first in "the best free reading apps for kids" by DigitalTrends and FreeKidsBooks and made the list on ScreenRant and Troomi , too!
It is also featured on BilingualBabies and is now recommended by the Israeli Ministry of Education for teachers of English in Israel.
Relay Reader has been /is being used for summer programming at the Camden Dream Center, NJ , RISE summer enrichment program , and NJ Tutoring Corps.
I am also one of the researchers behind Writing Mentor, a tool to help writers with academic writing. Writing Mentor has been taken down for now; you can still see the demo.
TWO is automatically extracting evidence of skill from language performance and helping people improve those skills.
The bulk of my work in this direction has been on writing (effective argumentation, use of figurative language, development, adherence to genre conventions, topicality, reflection, sentiment, sourcing, and use of facts). More recently I have also worked on oral reading and on leading a classroom discussion.
In a related line of work, we modeled the noisy human annotation process in demanding and subtle tasks, such as detection of semantic links and metaphoricity.
Before joining ETS, I was a post-doctoral fellow at the Kellogg's School of Management and Northwestern's Institute for Complex Systems where I used computational approaches to analyze political rhetoric. Check out some surprising findings about fish from an automated theme analysis of a speech by Margaret Thatcher, and see what Barack Obama did with his use of antithesis!
I hold a PhD in Computer Science from the Hebrew University of Jerusalem, Israel.
For my dissertation, I built a computational model to replicate some meaning-management patterns I observed in a close reading experiment.
To send me an email, put periods between the words in my tripartite name, at gmail.
Tools / Apps
I am the principal investigator behind Relay Reader, relevant research can be found here.
Try it out. Let us know!
Data Releases
Metaphor data and benchmarks have been released here.
Sentiment data for noun-noun compounds has been released here.
Academic service
2025 ARR Senior Area Chair for ACL 2025, Resources and Evaluation
2024-present Associate Editor, International Journal of AI in Education
2024 ARR Area Chair for ACL 2024, NLP Applications
2023 ACL 2023 Industry Track Chair
2022-2023 Guest Editor, a special issue of the International Journal of AI in Education on applications in languages other than English.
2020-present Action Editor, Transactions of the ACL
2022 ACL Senior Area Chair, NLP Applications
2021 ACL Area Chair, NLP Applications
2020 ACL Area Chair, Lexical Semantics
2019 NAACL Area Chair, Style
2019 *SEM Area Chair, Resources and Evaluation
Organizer/PC Chair, Workshops on Figurative Language in NLP (@EMNLP 2022, @ACL 2020, 2018) and Metaphor in NLP (@NAACL 2013-2016)
2013-present Associate Editor, ETS Technical Review, NLP area
I have been reviewing for *ACL and related venues for the last 15 years if not more; occasionally for JASIST, JAIR, IJAIED, as well as book and grant proposals.
Publications
Books
Beigman Klebanov, B. and Madnani, N. (2021). Automated Essay Scoring. Synthesis Lectures in Human Language Technologies. Hirst, G. (Ed.). Morgan & Claypool Publishers. [link]
NEW A review of the book by A. Tack -- online first in Computational Linguistics: link
Veale, T., Shutova, E., Beigman Klebanov, B., (2016) Metaphor: A Computational Perspective. Synthesis Lectures in Human Language Technologies. Hirst, G. (Ed.). Morgan & Claypool Publishers. [link]
A review of the book by C. Strapparava published in Computational Linguistics 44(1):191-192 link
A review of the book by P. Sommerauer published on Linguist List link
Book Chapter
Beigman Klebanov, B. (2023) Narrative writing form users-in-the-wild: A computational rhetorical analysis. Corpora and Rhetorically Informed Text Analysis. Brown, D.W. and Zawodny Wetzel, D. (Eds.). John Benjamins. [link]
Burstein, J., McCaffrey, D., Holtzman, S., & Beigman Klebanov, B. (2023) Making sense of college students' writing achievement and retention with automated writing evaluation. Advancing Natural Language Processing in Educational Assessment. Yaneva, V. and von Davier, M. (Eds.). Routledge. [link]
Beigman Klebanov, B., Kaufer, D., Yeo, P., Ishizaki, S., & Holtzman, S. (2016) Argumentative writing in assessment and instruction: A comparative perspective. Genre in Language, Discourse and Cognition. Stukker, N., Spooren, W., & Steen, G. (Eds.). Berlin: Mouton De Gruyter. [authors' final draft] .
Burstein, B., Beigman Klebanov, B., Madnani, N., Faulkner, A. (2013) Sentiment Analysis for Essay Evaluation. Handbook of Automated Essay Scoring: Current Applications and Future Directions. Shermis, M.D., Burstein, J. (Eds.). New York: Routledge.
Edited volumes
Sitaram, S., Beigman Klebanov, B., & Williams, J. (eds.), 2023. Proceedings of ACL 2023 Industry Track. Toronto, Canada. [pdf] [bib]
Ghosh, D., Beigman Klebanov, B., Muresan, S., Feldman, A., Poria, S., Chakrabarty, T. (eds.), 2022. Proceedings of EMNLP 2022 Workshop on Figurative Language Processing. Abu Dhabi, UAE & Cyberspace. [pdf] [bib]
Beigman Klebanov, B., Shutova, E., Lichtenstein, P., Muresan, S., Leong, C., Feldman, A., Ghosh, D. (eds.), 2020. Proceedings of ACL 2020 Workshop on Figurative Language Processing. Cyberspace. [pdf] [bib]
Beigman Klebanov, B., Shutova, E., Lichtenstein, P., Muresan, S., Leong, C. (eds.), 2018. Proceedings of NAACL 2018 Workshop on Figurative Language Processing. New Orleans, LA. [pdf] [bib]
Beigman Klebanov, B., Shutova, E., Lichtenstein, P. (eds.), 2016. Proceedings of NAACL 2016 Workshop on Metaphor in NLP. San Diego, CA. [pdf] [bib]
Shutova, E., Beigman Klebanov, B., Lichtenstein, P. (eds.), 2015. Proceedings of NAACL 2015 Workshop on Metaphor in NLP. Denver, CO. [pdf] [bib]
Beigman Klebanov, B., Shutova, E., Lichtenstein, P. (eds.), 2014. Proceedings of ACL 2014 Workshop on Metaphor in NLP. Baltimore, MD. [pdf] [bib]
Shutova, E., Beigman Klebanov, B., Tetreault, J., Kozareva, Z. (eds.), 2013. Proceedings of NAACL 2013 Workshop on Metaphor in NLP. Atlanta, GA. [pdf] [bib]
Technical reports
NEW Timpe-Laughlin, V., Beigman Klebanov, B., Loukina, A. (2024). Exploring the use of Relay Reader app with young English as a foreign language learners: A case study. ETS Research Memorandum, RM-24-01. Educational Testing Service. [pdf]
Song, Y., Deane, P., Beigman Klebanov, B. (2017) Toward the Automated Scoring of Written Arguments: Developing an Innovative Approach to Annotation. ETS Research Report Series. DOI:10.1002/ets2.12138 [pdf]
Journals
Liceralde, V., Loukina, A., Beigman Klebanov, B., Lockwood, J. R. (2022) Beyond text complexity: Production-related sources of variability in oral reading fluency. Journal of Educational Psychology, 114(1):16-36. [author's draft] [publisher version]
Beigman Klebanov, B., Ramineni, C., Kaufer, D., Yeoh, P., Ishizaki, S. (2019) Advancing the validity argument for standardized writing tests using quantitative rhetorical analysis. Language Testing, 36(1): 125-144. [authors' final draft] [publisher version]
Ringler, H., Beigman Klebanov, B., Kaufer, D. (2018) Placing writing tasks in local and global contexts: The case of argumentative writing. Journal of Writing Analytics, 2: 34-77. [pdf]
Beigman Klebanov, B., Priniski, S., Burstein, J., Gyawali, B., Harackiewicz, J., Thoman, D. (2018) Utility-value score: A case study in system generalization for writing analytics. Journal of Writing Analytics, 2: 285-313. [pdf]
Burstein, J., Elliot, N., Beigman Klebanov, B., Madnani, N., Napolitano, D., Schwartz, M., Houghton, P., Molloy, H. (2018) Writing Mentor: Writing progress using self-regulated writing support. Journal of Writing Analytics, 2: 314-328. [pdf]
Beigman Klebanov, B., Burstein, J., Harackiewicz, J., Priniski, S., Mulholland, M. (2017) Reflective Writing About the Utility Value of Science as a Tool for Increasing STEM Motivation and Retention – Can AI Help Scale Up? International Journal of Artificial Intelligence in Education, 27(4): 791-818. [publisher version]
Flor, M., Beigman Klebanov, B. (2014) Associative lexical cohesion as a factor in text complexity. International Journal of Applied Linguistics, 165(2): 223-256. [abstract] [authors' final draft]
Beigman Klebanov, B., Burstein, J., Madnani, N.(2013) Sentiment Profiles of Multi-Word Expressions in Test-Taker Essays: The Case of Noun-Noun Compounds. ACM Transactions on Speech and Language Processing, 10(3):12 [pdf] [data]
Beigman Klebanov, B., Madnani, N., Burstein, J. (2013) Using Pivot-Based Paraphrasing and Sentiment Profiles to Improve a Subjectivity Lexicon for Essay Data. Transactions of the Association for Computational Linguistics (TACL), 1: 99-110 [pdf]
Kaufer, D., Perry-Giles, S., Beigman Klebanov, B., (2012) The "image-bite" and political language: NBC news coverage of Hillary Clinton from scorned wife to senate candidate. Journal of Language and Politics, 11(3): 336-356 [pdf]
Beigman Klebanov, B., Kaufer, D., Franklin, H. (2010). A figure in a field: Semantic fields based analysis of antithesis in political speech. Cognitive Semiotics, 6: 121-154 [pdf]
Beigman Klebanov, B., Beigman, E. (2009). From annotator agreement to noise models. Computational Linguistics, 35(4): 495-503 [pdf]
Beigman Klebanov, B., Diermeier, D., Beigman, E. (2008). Lexical cohesion analysis of political speech. Political Analysis, 16(4): 447-463 [pdf]
Beigman Klebanov, B., Diermeier, D., Beigman, E. (2008). Automatic annotation of semantic fields for political science research. Journal of Information Technology and Politics, 5(1): 95-120 [pdf]
Beigman Klebanov, B., Shamir, E. (2006). Reader-based exploration of lexical cohesion. Language Resources and Evaluation, 40(2): 109-126 [pdf] [annotation protocol]
Proceedings
NEW Beigman Klebanov, B., Weeks, J., & Sinharay, S. (2024). To read or not to read: Predicting student engagement in interactive reading. AIED 2024 [html] [citation]
NEW Mikeska, J., Beigman Klebanov, B., Marigo, A., Tierney, J., Maxwell, T., & Nazaretsky, T. (2024). Exploring the potential of autaomtd and personalized feedback to suppot science teacher learning. AIED 2024 [html] [citation]
NEW Beigman Klebanov, B., Suhan, M., O'Reilly, T., & Wang, Z. (2024) From miscue to evidence of difficulty: Analysis of automatically detected miscues in oral reading for feedback potential. NLP for Building Educational Applications 2024 [pdf] [bib]
NEW Gurin Schleifer, A., Beigman Klebanov, B., Ariely, M., & Alexandron, G. (2024) Anna Karenina strikes again: Pre-trained LLM embeddings may favor high-performing learners. NLP for Building Educational Applications 2024 [pdf] [bib]
NEW Ilagan, M., Beigman Klebanov, B., & Mikeska, J. (2024) Automated evaluation of teacher encouragement of student-to-student interactions in a simulated classroom discussion.NLP for Building Educational Applications 2024 [pdf] [bib]
Beigman Klebanov, B., Suhan, M., Wang, Z., & O'Reilly, T. A dynamic model of lexical experience for tracking of oral reading fluency. (2023) NLP for Building Educational Applications 2023 [pdf] [bib]
Gurin Schleifer, A., Beigman Klebanov, B., Ariely, M., & Alexandron, G. (2023) Transformer-based Hebrew NLP models for short answer scoring in biology. NLP for Building Educational Applications 2023 [pdf] [bib]
Jiang, Y., Beigman Klebanov, B., Livne, O., & Hao, J. (2023) Analyzing users' interactions with writing feedback and their effects on writing performance. AIED 2023 [html]
Nazaretsky, T., Mikeska, J. and Beigman Klebanov, B. (2023) Empowering teacher learning with AI: Automated evaluation of teacher attention to student Ideas during argumentation-focused discussions. LAK 2023 [pdf]
Beigman Klebanov, B., and Loukina, A. (2021) Exploring structured error to improve automated scoring of oral reading fluency. AIED 2021 [pdf]
Misra, A., Loukina, A.. Beigman Klebanov, B., Gyawali, B., & Zechner, K. (2021) A good start is half the battle won: Unsupervised pre-training for low resource children's speech recognition for an interactive reading companion. AIED 2021 [pdf]
McCaffrey, D., Burstein, J., Holtzman, S., & Beigman Klebanov, B. (2021) What can we learn about college retention from student writing? Learning Analytics & Knowledge Conference LAK 2021 [pdf]
Beigman Klebanov, B., and Madnani, N. (2020) Automated Evaluation of Writing -- 50 years and counting. ACL 2020 [pdf] [bib]
Ghosh, D., Beigman Klebanov, B., and Song, Y. (2020) An exploratory study of argumentative writing by young students: A transformer-based approach. NLP for Building Educational Applications 2020 [pdf] [bib]
Chen, X., Leong, C., Flor, M., Beigman Klebanov, B. (2020) Go Figure! Multi-task transformer-based architecture for metaphor detection using idioms: ETS team in 2020 metaphor shared task. ACL Workshop on Processing Figurative Language 2020 [pdf] [bib]
Leong, C., Beigman Klebanov, B., Hamill, C., Stemle, E., Ubale, R., Chen, X. (2020) A report on the 2020 VUA and TOEFL metaphor detection shared task. ACL Workshop on Processing Figurative Language 2020 [pdf] [bib]
Beigman Klebanov, B., Loukina, A., Lockwood, J., Liceralde, V., Sabatini, J., Madnani, N., Gyawali, B., Wang, Z., Lentini, J. (2020) Detecting learning in noisy data: The case of oral reading fluency. Learning Analytics & Knowledge Conference LAK 2020 [pdf]
Shao, S., Beigman Klebanov, B., Loukina, A., Kannan, P., Heincke, P. (2020) Where AI meets the learner: Classroom as a mediator. AAAI workshop on AI for Education 2020 [pdf]
Loukina, A., Beigman Klebanov, B., Lange, P., Qian, Y., Gyawali, B., Madnani, N., Misra, A., Zechner, K., Wang, Z., Sabatini, J. (2019) Automated estimation of oral reading fluency during summer camp e-book reading with My Turn To Read. INTERSPEECH 2019 [link]
Madnani, N., Beigman Klebanov, B., Loukina, A., Gyawali, B., Sabatini, J., Lange, P., Flor, M. (2019) My Turn To Read: An Interleaved E-book Reading Tool for Developing and Struggling Readers. ACL 2019 [pdf] [bib]
Beigman Klebanov, B., Loukina, A., Madnani, N., Sabatini, J., Lentini, J. (2019) Would you? Could you? On a tablet? Analytics of children's e-book reading. Learning Analytics & Knowledge Conference LAK 2019 [pdf]
Loukina, A., Madnani, N., Beigman Klebanov, B., Misra, A., Angelov, G., Todic, O. Evaluating on-device ASR on field recordings from an interactive reading companion. IEEE Workshop on Spoken Language Technology (SLT) 2018 [pdf]
Madnani, N., Burstein, J., Elliot, N., Beigman Klebanov, B., Napolitano, D., Andreyev, S., and Schwartz, M. Writing Mentor: Self-Regulated Writing Feedback for Struggling Writers. COLING 2018 [pdf] [bib]
Loukina, A., Liceralde, V., Beigman Klebanov, B. Towards understanding text factors in oral reading. NAACL 2018 [pdf] [bib]
Beigman Klebanov, B., Leong, C., Flor, M. A corpus of non-native written English annotated for metaphor. NAACL 2018 [pdf] [bib] [data]
Leong, C., Beigman Klebanov, B., Shutova, E. A report on the 2018 VUA metaphor detection shared task. ACL Workshop on Processing Figurative Language 2018 [pdf] [bib]
Flor, M., Beigman Klebanov, B. Catching idiomatic expressions in EFL essays. ACL Workshop on Processing Figurative Language 2018 [pdf] [bib]
Loukina, A., Zechner, K., Bruno, J., Beigman Klebanov, B. Using exemplar responses for training and evaluating automated speech scoring systems. NLP for Building Educational Applications 2018 [pdf] [bib]
Loukina, A., Beigman Klebanov, B., Lange, P., Gyawali, B., & Qian, Y. Developing speech processing technologies for shared book reading with a computer. WOCCI 2017 [pdf] [bib]
Beigman Klebanov, B., Loukina, A., Sabatini, J., O'Reilly, T. Continuous fluency tracking and the challenges of varying text complexity. NLP for Building Educational Applications 2017 [pdf] [bib]
Burstein, J., McCaffrey, D., Beigman Klebanov, B., Ling., G. Exploring relationships between writing and broader outcomes with automated writing evaluation. NLP for Building Educational Applications 2017 [pdf] [bib]
Beigman Klebanov, B., Gyawali, B., Song, Y. Detecting good arguments in a non-topic-specific way: an oxymoron? ACL 2017 [pdf] [bib]
Beigman Klebanov, B., Leong, C., Gutierrez, D., Shutova, E., Flor, M. Semantic classifications for detection of verb metaphors. ACL 2016 [pdf] [bib] [data]
Beigman Klebanov, B., Stab, C., Burstein, J., Song, Y., Gyawali, B., Gurevych, I. Argumentation: Content, Structure, and Relationship with Essay Quality. Argument Mining 2016 [pdf] [bib]
Burstein, J., Beigman Klebanov, B., Elliot, N., Molloy, H. A Left Turn: Automated Feedback & Activity Generation for Student Writers. Language Teaching and Language Technology 2016 [pdf]
Beigman Klebanov, B., Flor, M., Gyawali, B. Topicality-based indices for Essay Scoring. NLP for Building Educational Applications 2016 [pdf] [bib]
Beigman Klebanov, B., Burstein, J., Harackiewicz, J., Priniski, S., Mulholland, M. Enhancing STEM Motivation through Personal and Communal Values: NLP for Assessment of Utility Value in Student Writing. NLP for Building Educational Applications 2016 [pdf] [bib]
Beigman Klebanov, B., Leong, C., Flor, M. Supervised Word-Level Metaphor Detection: Experiments with Concreteness and Reweighting of Examples. Metaphor in NLP 2015 [pdf] [bib] [data]
Flor, M., Beigman Klebanov, B. ETS Lexical Associations System for the CogALex Shared Task. Cognitive Aspects of the Lexicon (CogALex) 2014 [pdf] [bib]
Beigman Klebanov, B., Madnani, N., Burstein, J., Somasundaran, S. Content Importance Models for Scoring Writing from Sources. ACL 2014 [pdf] [bib]
Beigman Klebanov, B., Beigman, E. Difficult cases: From Learning to Annotation, and Back. ACL 2014 [pdf] [bib]
Beigman Klebanov, B., Leong, C., Heilman, M., Flor, M. Different Texts, Same Metaphors: Unigrams and Beyond. Metaphor in NLP 2014 [pdf] [bib]
Song, Y., Heilman, M., Beigman Klebanov, B., Deane, P. Applying Argumentation Schemes to Essay Scoring. Argumentation Mining 2014 [pdf] [bib]
Beigman Klebanov, B., Flor, M. Word Association Profiles and their Use for Automated Scoring of Essays. ACL 2013 [pdf] [bib]
Beigman Klebanov, B., Flor, M. Associative Texture is Lost in Translation. Discourse in Machine Translation 2013 [pdf] [bib]
Beigman Klebanov, B., Flor, M. Argumentation-relevant Metaphors in Text-Taker Essays. Metaphor in NLP 2013 [pdf] [bib]
Flor, M., Beigman Klebanov, B., Sheehan, K. Lexical Tightness and Lexical Complexity. NLP for Improving Textual Accessibility 2013 [pdf] [bib]
Beigman Klebanov, B., Higgins, D. Measuring the use of factual information in test-taker essays. NLP for Building Educational Applications 2012 [pdf] [bib]
Beigman Klebanov, B., Burstein, J., Madnani, N., Faulkner, A., Tetreault, J. Building Subjectivity Lexicon(s) from Scratch for Essay Data. CICLING 2012
Beigman Klebanov, B., Beigman, E. A Game-Theoretic Model of Metaphorical Bargaining. ACL 2010 [pdf] [bib]
Beigman Klebanov, B., Beigman, E., Diermeier, D. Vocabulary choice as an indicator of perspective. ACL 2010 [pdf] [bib]
Beigman Klebanov, B., Beigman, E. Some empirical evidence for annotation noise in a benchmarked dataset. NAACL 2010 [pdf] [bib]
Beigman, E., Beigman Klebanov, B. Learning with annotation noise. ACL 2009 [pdf] [bib]
Beigman Klebanov, B., Beigman, E., Diermeier, D. Discourse topics and metaphors. Computational Approaches to Linguistic Creativity 2009 [pdf] [bib]
Beigman Klebanov, B., Beigman, E., Diermeier, D. Analyzing disagreements. Human Judgments in Computational Linguistics 2008 [pdf]
Beigman Klebanov, B. Measuring semantic relatedness using people and WordNet. NAACL 2006 [pdf] [bib]
Beigman Klebanov, B. Using readers to identify lexical cohesive structures in text. Student Research Workshop at ACL 2005 [pdf] [bib] * This paper is superseded by the Language Resources and Evaluation paper "Reader-based exploration of lexical cohesion."
Beigman Klebanov, B., Knight, K., Marcu, D. Text simplification for information-seeking applications. Ontologies, Databases, and Applications of Semantics 2004 [pdf]
Klebanov, B., Wiemar-Hastings, P. The role of world knowledge in pronominal anaphora resolution. Reference Resolution for NLP Symposium 2002
Klebanov, B., Wiemar-Hastings, P. Using LSA for pronominal anaphora resolution. CICLING 2002