Student Perception of Collaborative Annotation Activities

Eliza Ballantyne

Wyant College of Optical Sciences | University of Arizona

Background

Class Information

Background

Both academic and informal sources indicate the students read much less than assigned and much less than their peers did in the past (Orlando 2023 and Porter 2022). I do not find a decrease concerning, especially if indicators suggest students are still gaining the core skills demanded by a course. Students may be taking advantage of free and conveniently presented online resources or instructors could be trading an traditional study requirement for more versatile approaches. On the other hand, academic primary literature in engineering is the main mechanism for conveying new technical ideas. Failing to prepare students for this kind of close reading fails to prepare them for success in a career where it is the standard.

Porter et. al. suggests that assigning as task, something with recognizable value to student's future careers, and using a reading assignment to provide scaffolding mimics the behavior of researchers and engineers beyond the classroom. Collaborative annotation is one such task where students interact with each other by accessing and responding to each other's comments on an assigned reading, mimicking asynchronously a group discussion of new ideas. Reported benefits in peer reviewed literature include a heightened sense of community among students and enhanced comprehension (Bjorn 2023), along with rapid peer feedback (Porter 2022). While formal studies acknowledge limitations including widely varying experience with annotation between participants and unaccounted AI impact (Bjorn 2023), most challenges with collaboration activates are discussed in informal settings, including teaching blogs and discussion forms. These include the disadvantage students face when they consistently do early work and cannot respond to comments (Reddit User ProfessorHomeBrew 2023), and a race to comment early to avoid being left with no low hanging fruit (Reddit User OneMoreProf 2024), and student perception of collaborative annotation activities as busywork (Reddit User ProfessorHomeBrew). My project centered on putting these benefits and challenges to the test on the scale of a single undergraduate enviornmental engineering course.

Class Information

The goal of CHEE 377 is to help students understand fundamental microbial processes and their significance in environmental engineering. Twenty junior and senior students selected this course as an elective and participated in my project. They plan to work in desalination, water management, mining, and other environment-focused careers and recognized the relevance of this class to their future careers. Most said they had little microbiology experience since high school.

Before each class, students read and collaboratively annotate a textbook excerpts or peer reviewed article relating to current concepts. An example assignment is included in figure one. Clicking on a highlight in the Perusall software allows students to access the thread of interactions and to contribute to the conversation themselves. Annotations are graded automatically using the Perusall software, which should detect "copy-and-paste" style annotations relying on AI and reward students for authentic interactions with each other.

In class, students participate in brief conceptual quizzes corresponding to the material covered in the assigned reading. My goal was to determine if participating in collaborative annotation activities improved student preformance on exams and to survey their perception of the activity.

Figure 1 | Example of assigned reading and student highlights

TAR Questions:

Do collaborative annotation activities improve student performance on in class quizzes?
What do students think of collaborative annotation activities?

Methods

Figure 2 | Project Timeline

Figure 3 | Student opinion survey

Data Collection

The project was broken down into three phases: pre-study spanning January and February, data collection for two weeks in March, and surveys conducted in April. Since varying experience with annotation is a reported challenge of collaborative annotation activities, students participated in a guided asynchronous activity to help them understand the Persuall website and provided examples of high scoring annotations (Bjorn 2023). Then, the first two months of the course were dedicated to giving students time to get feedback from their peers and the automatic grading scheme, and develop confidence in collaborative annotation activities.

After students returned from spring break, they were split into two groups. Each group was assigned to participate in two of the next four collaborative annotation activities and asked to only read, without annotating, the other two. I tracked student participation in reading and annotation activities and linked it with their in class quiz scores. For the sake of consistency, I excluded from my quantitative results any student who did not both participate in the assigned reading/annotation activity and take the in class quiz. As a result, the number of participants varied throughout the study as indicated in figure two. I also collected all student annotations during this two week period for qualitative coding.

April survey's were conducted in two sessions. My initial plan was only to ask students to rank four statements about Perusall readings on a Likert scale and answer an open ended "what would you change about these assignments" question using a google survey. Seventeen of the twenty students participating in this class responded to my survey.

After qualitative analysis of the comments from phase two, I was concerned that some students were offloading their annotations to an AI. This was outlined in the assignment as cheating and students were warned the automatic grading scheme would flag AI-generated annotations. While the survey responses were anonymous, I was worried students would not report their AI usage accurately in a survey linked to their coursework. I returned to class and passed around post it notes where I asked them to write down only a "yes" or "no." I tallied up these anonymized responses and, as I promised the students, threw out the post it notes. 45% of the class admitted to using AI at least once to generate their collaborative annotation assignment. This confounding variable led me instead to focus on a qualitative analysis of student comments.

Analysis and Results

I planned to use the Student's t-test to determine if participation in collaborative annotation activities improved student's quiz scores. At face value, it seems like the students who participated in the activity actually performed worse than their peers. Due to the small sample size, the resulting statistics were very low power. When taken in the light of student's AI usage, I instead turned most of my focus to qualitative analysis of students comments.

Figure 4 | Student's AI usage to complete collaborative annotation activities

Figure 5 | Student's quiz scores segregated by participation in collaborative annotation activites

Using free Taguette software, I combed through every comment made in the assignments and assigned codes to repeating trends, such as students asking a question or summarizing what they had read.

In figure four, each row represents a type of student annotation or interaction while each column indicates the corresponding assignment. I've highlighted an example from each category in the carousel below.

I find it telling that students ask many questions, but only ever answered one during the study. Students may lack the foundation to answer a peer's question with confidence or they may simply prioritize their own questions, as indicated by the "external research" category where students linked articles, videos, or other supplemental resources. Perhaps the extra effort and risk to answer someone else's questions would be more likely if the grading scheme prioritized this kind of interaction.

Students also favored summaries of material, which seemed promising for their retention. Variants on the theme "its interesting" also appeared frequently, although I doubt they added value to the experience. Students consistently connected their readings to applications in the real world, mentioning their upcoming internships. These kind of comments tended to gain several reactions from peers which improved their performance in the Perusall grading scheme.

Figure 6 | Taguette qualitative coding

Figure 7 | Frequency table of coded student annotations

While beyond the scope of this project, I am interested in determining if the increase in the number of questions students ask when reading a published, peer-reviewed article is statistically significant when compared to reading a selection from a textbook. Bjorn, who reported benefits of collaborative annotation including increased confidence, acknowledged that the kind of assigned reading could be significant. Future work that includes more samples of peer-reviewed work in the assignments could provide an answer to this question, easing the tensions between the benefits of collaborative annotation described in formal studies and the experiences reported informally.

To asses the Likert scale responses, I excluded all neutral responses and tallied the number of students who disagreed or agreed with the statement. Most students reported they viewed collaborative annotation as busywork and disagreed that it was a valuable use of time. This data, combined with the AI usage of the students in this course, highlights the vicious cycle of students delegating the collaborative annotation activity to AI, subsequently failing to gain anything from the activity, and rienforncing their perception that the activity is worthless.

Figure 8 | Likert Survey Responses

In the open ended portion of the survey, students suggested altering the grading scheme, requiring fewer comments in each assignment, replacing annotations with a secondary quiz, shortening the readings, or asking for a single summary. Many students expressed frustration that the grading scheme seemed to focus on the length of a comment, instead of its quality.

Summary:

45% of students used AI at least once to complete the collaborative annotation activity. This counfounding variable makes it challenging to accuratley asses if collaborative annotation actually improves student's preformance on corresponding quizzes.
While they readily ask questions in their annotations, students do not often answer each other's questions. Additional scaffolding or incentives may be required before students can complete the question-answer cycle.
Students view collaborative annotation as busywork, which may contribute to their willingness to offload the activity to AI. This, of course, garnatees the activity does not contribute to their learning and rienforces their perception that is is not a valuable use of their time or attention in a vicous cycle.

Future Work

As a result of this project and student's feedback, Perusall assignments require fewer comments and now include a short reading quiz in CHEE 377. This effectively creates two new TAR questions about reading quizzes, identical to the ones I just addressed about collaborative annotation. Besides those questions, future work could also focus on the differences in student interaction between primary scientific literature and textbook reading and how to encourage students to answer each other's questions on the Perusall platform.

Reflection

I wondered initially if I could somehow strongarm a future class into completing the required activities without using AI. I was exasperated that my quantitative data was undermined by their corner-cutting and disappointed that I ignored roughly a fourth of each assignment's data due to students missing an assignment or quiz. Somewhere between imagining the perfect data gathered from a class of robot students who enthusiastically participated in every activity and never missed a deadline, I realized how pointless that study would be to anyone teaching in the real world where students are board, excited, tired, and busy living life. Beyond being impractical, it would fundamentally miss that teaching involves the whole student, in all their messy humanity. That leaves me certain that a key part of teaching has to be an invitation to students to understand the how and why behind what I ask them to do. There's no guarantee that students will be willing to engage or have the bandwidth to participate in metacognition at every step, but my conclusion from this project is that the invitation itself is essential to teaching.

Acknowledgements

This work was made possible thanks to Dr. Byron Hempel providing access to his microbiology for engineers course and to students who participated and provided feedback. I have not used AI for any part of this project.

About the Author

Eliza is an optical engineering graduate student whose usual research interests include nanophotonics, microassembly using optical tweezers, and photonic integrated circuits. When she isn't dabbling in educational research, Eliza enjoys hiking in Saguaro National Park, visiting every bakery she can find, and playing flag football.

References

Bjorn, G. (2023). The power of peer engagement: Exploring the effects of social collaborative annotation on reading comprehension of primary literature. AI, Computer Science and Robotics Technology, 2. https://doi.org/10.5772/acrt.24

Orlando, J., Weimer, M., Imad, M., Kale, C., Chew, S. L., & Franz, L. V. and T. (2023, September 18). A flipped method for assigning readings. The Teaching Professor. https://www.teachingprofessor.com/topics/preparing-to-teach/assignments-and-activities/a-flipped-method-for-assigning-readings/

Porter, G. W. (2022). Collaborative online annotation: Pedagogy, assessment and platform comparisons. Frontiers in Education, 7. https://doi.org/10.3389/feduc.2022.852849

Reddit User ProfessorHomeBrew. (2023). R/professors on reddit: Perusall report back. Reddit/Professors. https://www.reddit.com/r/Professors/comments/yns1wk/perusall_report_back/

Reddit User OneMoreProf. (2024). R/professors on reddit: Hypothes.is vs Perusall for social annotation assignments?. Reddit/Professors. https://www.reddit.com/r/Professors/comments/1ctr678/hypothesis_vs_perusall_for_social_annotation/

Spencer, R. B. and E., Weimer, M., Imad, M., Kale, C., Orlando, J., Chew, S. L., & Franz, L. V. and T. (2018, October 22). 10 strategies for promoting accountability and investment in reading assignments. The Teaching Professor. https://www.teachingprofessor.com/topics/teaching-strategies/engagement-and-motivation/10-strategies-for-promoting-accountability-and-investment-in-reading-assignments/

Rampin, R., Rampin, V., & DeMott , S. (n.d.). Taguette. Taguette, the free and open-source qualitative data analysis tool. https://www.taguette.org/about.html

Page updated

Report abuse