Research Statement

Accurately grading open-ended assignments in large, online courses is an increasingly necessary, non-trivial problem [1]. For example, UC San Diego Professor Scott Klemmer has taught online courses in Human Computer Interaction on Coursera that have garnered over 3,600 students, with weekly website design tasks. For an individual or small team to grade or provide feedback on such an open-ended assignment is effectively impossible. To solve this scalability problem while maintaining a project-based architecture, I (like Dr. Klemmer and others) employ the widely-used pedagogical evaluation methodology of peer assessment [2, 3]. Peer assessment is a promising solution but can be unreliable due to a low number of reviewers (e.g. 3-5) and an un-evaluated review form (rubric). A scalable, reliable, data-driven, and crowd-sourced assessment solution is required and possible with natural language processing (NLP), data mining, and data visualization techniques [4].

I am the first to 1) leverage sentiment analysis (SA) in the peer assessment process to provide or validate grades and 2) utilize aspect extraction to develop a review form based on what students actually communicate [2]. Rather than waste student data, I data mine past review form open-ended comments to build a sentiment lexicon (dictionary) and modify the review form based on key words and phrases found. I then utilize SA to provide a fine-grained score from the written text alone and present a visualization/summary to deliver the maximum information to the professor. I chose a lexicon-based approach rather than a recurrent neural network because explainability is paramount — students often request justification for their grade. To validate my approach, I compare my 1) domain-specific lexicon to other generic lexicons (e.g. ANEW, SentiWordNet, MPQA, SlangSD) and 2) sentiment algorithm to other publicly available algorithms (e.g. AFINN, SentiWordNet, OpinionFinder, VADER).

My interdisciplinary work contributes to both computer science (e.g. the fine-grained “rater inference problem” in NLP and SA) and education (e.g. applying data-driven methods to creating review forms and accurately scaling assessment of open-ended assignments in large courses). The implications of this research are wide-reaching: my first-author work has been published in engineering education, computer-aided design, and data visualization peer-reviewed journals and conferences. I currently utilize a number of growing corpora of 1) over 6,800 peer assessments from nine USF computer graphics and software engineering courses (325 student works), 2) over 2,500 peer assessments from PeerLogic, an open repository from several peer review systems, and 3) over 2,200 peer assessments from three USF data visualization courses (840 student works). This data is used to improve the review form, infer engagement, and determine the number of peer reviewing students necessary for a reliable grade. The final result of my innovative work is a semi-automated peer assessment process based on actual student work that can be employed by a professor to confidently grade meaningful open-ended assignments in any size course.

I currently have five first author and two second author publications as well as one first author publication under review. I am on a research proposal to improve visual peer review via an open-sourced dashboard that increases reliability and reproducibility of visualizations throughout their iterative development (visualpeerreview.org) and a second proposal to improve the expressiveness of peer feedback using multimodal data collection. My future research plans include improving the performance and parallelization of the SA algorithms I have written for peer review, contrasting information and scores from open-source recurrent neural networks, and applying domain adaptation/transfer learning techniques to leverage SA in other genres. I am also interested in the application of document summarization and aggregation to extract information from noisy corpora and determine its truthfulness and helpfulness. Finally, my goal is to use data mining, machine learning, and AI to improve student retention, experience, and performance in the classroom.

References

[1] Beasley, Z. J., Piegl, L. A., & Rosen, P. (2018). Ten challenges in CAD cyber education. Computer-Aided Design and Applications, 15(3), 432-442.

[2] Piegl, L. A., Beasley, Z. J., & Rosen, P. (2019). Assessing Student Design Work using the Intelligence of the Crowd.

[3] Beasley, Z., Friedman, A., Piegl, L., & Rosen, P. (2020). Leveraging Peer Feedback to Improve Visualization Education. arXiv preprint arXiv:2001.07549.

[4] Beasley, Z. J., Piegl, L. A., & Rosen, P. (2019, June). Board 39: Designing Intelligent Review Forms for Peer Assessment: A Data-driven Approach. In 2019 ASEE Annual Conference & Exposition.