Do online exams help or hurt?

Keila S. Espinoza, MS

Physiological Sciences GIDP

Introduction

TAR Question

Approach and Methods

Findings from Survey 1 - Sentiments, Preperation, and Perceptions

Findings from Assessment 1 - Justifying an Answer Choice

Sample Student Responses and Scoring

Findings from Survey 2 - Sentiments about Assessments and Gauging Resource Use

Discussion and Lessons Learned

About the Author

Introduction

After the whirlwind of COVID-19, the online realm of teaching persisted after the return to normalcy. One such artifact is online exams, which have been documented as having many positive effects for students (Bengtsson, 2019). Weekly, online exams were utilized throughout the 2023 spring semester of PSIO431/531, an in person upper divison (or graduate) Immune Systems Physiology course. In contrast, most STEM courses utilize in person exams. While multiple choice questions continue to reign supreme, the preparation and stress of an in-person exam varies to that from an online exam.

However, are the alleged benefits of online exams truly present; are they appreciated by the students? More importantly, are the students forming key learning connections with low-stakes, un-proctored online exams?

{Image from https://uknow.uky.edu/campus-news/navigating-mental-health-conversation-your-college-student-2020}

TAR Question

What are students’ sentiments of weekly online tests, and how does changing exam format from traditional exams impact student’s understanding of immune system physiology?

Approach and Methods

The Goals:

1) determine students' sentiments about weekly online exams

and

2) assess learning and understanding of course material

The Methods:

We used a pre-survey before administering any assessments. Afterwards, we tasked students with justifying their answer to a more difficult multiple choice (MC) question during one of their weekly online exams. We then administered a short answer question (SAQ) homework assignment, in which the questions integrated multiple topics into one -- to compare, usual assignments are specific to the lecture week's content. After the assessments were completed, students then completed a post survey to reflect on the assessment and gather additional habits on studying and test taking.

The Data:

Statistics reported were obtained through the native Qualtrics engine that surveys were administered through, or calculated via Microsft Excel. Figures were generated through Prism GraphPad Vers.9 software.

Findings from Survey 1 - Sentiments, Preperation, and Perceptions

Online exam sentiments

Within 61 responses, an overwhelming majority 93.5% of participants found weekly online exams to be a positive in some aspect. Interestingly, while one student had a mixed view of the exam format, no respondents found weekly online exams to be detrimental. Through additional commentary, many students expressed that it was much less stressful. Further, a statement confessed that they hoped other courses would adopt similar online exams. However, some students shared their frustration that some questions were too complex and that the exam time, being 20 minutes, is too short of a time to answer. One student additionally stated that it "may be helpful... to put some emphasis on certain things [they] need to know for exams... Sometimes it feels like [they] brush by things that would be helpful to know for the exams."

exam preperation

Out of 56 responses, a total of 12 studying habits were identified, in which students used anywhere from one up to five methods. Out of a total of 97 studying incidences, the major studying tool used were notes obtained either from attending class or from the provided, annotated notes the professor provides. Online course tools encompasses utilizing lecture slides, objectives, questions, and recorded lecture videos. Help seeking encompassed in-person aid or outside resources. Miscellaneous encompassed review, designated study time, and attending class. Of 61 responses, students reported studying for an average of 2.4 ± 1.45 hours studying for these online exams.

In-depth sentiments and confidence

61 students ranked their agreeability to 6 statements with the stem 'I find weekly online exams to be...' :

1) less stressful than in-person exams

2) less stressful than one larger exam

3) helpful in retaining information from class

4) easy to make time for

5) good test of what I know

6) a good test for bringing many concepts together

Similar to the online exam sentiments results, the majority of responses agreed in some manner with the statements posed. However, statement 5, which asked if online exams were a good test of what they know, showed the most uncertainty of all the statements. 20% of students did not vote in a positive manner. Furthermore, statement 3, which asked if online exams were helpful in retaining information, had the most disagreement of all statements. 8% of students disagreed with this statement.

These two notable differences among other statements may highlight that students may not be retaining or understanding course content as well as previous exam methods may illicit. On a scale of 0-10, with zero being not confident at all and 10 being very confident, students ranked themselves at a 5.8 ± 2.4 on their confidence on doing equally as well or better when retaking a previous exam. Similarly, students ranked themselves at a 4.8 ± 2.5 confidence in doing well on an SAQ which integrated several course concepts. At fairly modest to low scores, student's self perception in the confidence of their success is not very high.

Findings from Assessment 1 - Justifying an Answer Choice

Garnering student understanding in a multiple choice exam

All 177 students participated in a justification assessment during one of their usual weekly online exams. Students were given a multiple choice question to answer, and were then tasked to justify their answer. Answers were scored on a scale of 0-5 with the following scoring description:

0) Answer left blank

1) Answer is irrelevant with no knowledge based on content

2) No link, with some relevant ideas, but none about the answer

3) Partial link, surface ideas are present but not connected

4) Full link, two ideas are connected and elaborated

5) Complex link, three or more ideas are connected in several in depth ways.

This justification method and scoring technique were adapted from 'Assessing Integrated Understanding of Science' by DeBoer et al., 2008.

Overall, 8.5% students (n=15) chose the incorrect answer to the question, whereas 91.5% of students (n=162) responded correctly. 138 total students both responded correctly and provided their justification. The average score excluding blanks was 3.7 ± 1.4. 51.5% of these students (n=71) demonstrated complex or full links within their justifications, while 26.8% of students (n=37) demonstrated a partial link, and 21.7% of students (n=30) did not demonstrate an appropriate link. Given these responses, half of the class seemingly has an understanding of the in-depth concepts and intricacies behind the question and answer choices, whereas a quarter of students have some understanding about the meaning behind their answers. Concernedly, up to one fifth of students who answered correctly were unable to bring together the proper understanding for why they chose their answer. However, one must keep in mind that students do have their notes available to them, in which some justifications lend themselves to the possibility of being written with the aid of notes given their intricacies. One such response is shared below:

"The Old Friends hypothesis suggests that there are certain microorganisms that are important to instruct our immune systems to do what it's supposed to do better. Wienstock built off of this theory and investigated what would happen if people with IBD (an inappropriate IgE response) were infected with a parasite to better re-direct IgE. He found that most of the people with IBD had improved symptoms after being infected with a parasite, showing that it may be possible for our immune systems to be re-educated to improve our immune responses. In question 10, people with MS are exhibiting an inappropriate Th1 response. According to the Old Friends hypothesis and Wienstock's findings, it can be suggested that their MS symptoms improved after parasitic infection because their immune systems were being "re-educated." Therefore, the most plausible answer choice is that after infection, there would be increased numbers of Treg cells, since Treg cells help turn off other T helper cell responses, which would decrease the inappropriate Th1 response seen in MS, thus also decreasing MS symptoms."

curious to know the questions, answers, and how Justifications were scored? here they are!

Test Question:

In Brazil, when some patients with multiple sclerosis developed infections with parasitic worms, their MS symptoms improved significantly. (MS is probably caused by Th1 cells attacking myelin in the nervous system). Their lymph nodes were biopsied with they were in remission, and it was found that, compared to before the worm infection:

Answer options, correct response bolded:

There were large B cell germinal centers

There were increased numbers of Treg cells

There were numerous worm ova in the lymph nodes

There were increased numbers of Th1 cells

Sample Student Responses and Scoring

When someone has an autoimmune disease they produce a lot of white blood cells to create inflammation. With worms, I believe they can eat away at those over produced white blood cells

Treg cells help the parasite survive longer. In addition to this, treg cells help regulate the immune system, which is compromised in patients with MS. An immune system that is better regulated will work in favor of the patient (regardless of them having an autoimmune disease) and will help relieve symptoms.

Treg cells, otherwise known as regulatory T cells, are specific T cells that when enacted, suppress the body's immune response. When the body suffers the infection from the worms, the treg cells that are activated are able to suppress further immune response, returning the system back to homeostasis.

Since multiple sclerosis is caused by Th1 cells attacking the myelin in the nervous system. It would make sense to me that the number of T regulatory cells needs to increase to counteract this and tell the T helper cells specifically Th1s to stop their action of attacking the nervous system. This increased activation of T regulatory cells turns all the other T helpers off.

One of the main roles of Treg cells is to suppress the immune response and ensure that the body maintains homeostasis. It can do this in a variety of ways, one of which is to inhibit the actions of other T cells such as Th1 cells. If MS is probably caused by Th1 cells attacking the myelin in the nervous system, and patients with MS saw an improvement with their symptoms after having a parasitic infection, this leads me to believe that something is suppressing the actions of the Th1 cells. This could be due to the parasitic infection leading to an increase in the number of Treg cells in the lymph nodes which in turn will lead to a larger suppression of Th1 activity. This will ultimately lead to a decrease in the symptoms of MS.

Findings from Survey 2 - Sentiments about Assessments and Gauging Resource Use

Student sentiments about the justification exercise

Although there are 177 students in the class, only 35 students participated in our close-out survey. Of the responses, 28 shared their sentiments on whether they enjoyed, felt neutral, or disliked the exercise. Overall, 71.4% of students enjoyed the exercise, while 10.7% disliked the exercise. Most interestingly, the majority of additional commentary came from the students who enjoyed the exercise expressing they enjoyed the extra credit (EC); how it made them think over the thought process of their answer; and other responses like boosted confidence, the reinforcements of idea, and simple fun.

Student Resource use Before and After Exams

30 to 35 students ranked their use of various course resources prior to and during exams.

Up to 80% of students did not utilize office hours if they had questions or content concerns, while 68.6% and 67.7% did not utilize communicating with mentors of the course or facilitated review sessions, respectfully. In fact, communication with the course mentors (professor and preceptors) was the most under utilized resource reported. Unsurprisingly, based off the results from the 'Methods of Preparation' data, 91.4% and 88.6% of students reported utilizing their notes and lecture notes, respectively, to prepare for exams.

It was not surprising to find that students did indeed utilize recourses during their weekly online exams. Up to 74.1% reported frequently using their notes during online exams. Being just under a quarter of respondents, or 23 students, it is difficult to ascertain if this value can be projected onto the rest of the class standing of 177 students. Even still, should we consider the lower end of note usage, being 53.3% of lecture note users, this would still mean over half of the course body is utilizing a resource while taking their exams. In a pool of 30 students, only 2 individuals, being 6.7% of the group, reported never using any of the resources during online exams.

With the rise of AIs such as ChatGTP shaking up the realm of education, we decided to include it as a resource. While students may only have 20 minutes to complete their exams, we were uncertain to see if students may be using it. Surprisingly, 10% of respondents, or 3 students, did indicate that they sometimes used AI to help them. AI has been known to not be wholly accurate, and output responses from only what it gleans in its information banks. It would be interesting to see how these students fed AI their questions, what it gave back to them, and how they fared on their responses due to AI input.

Discussion and Lessons Learned

This project still needs to be further analyzed to better assess how students are understanding the content and material of the course. As one may notice, findings from Assessment 2 are currently not included. Due to the nature and number of submissions, it will take a considerable amount of time to analyze this final piece.

Given the findings of this project, I personally believe that current format of online exams are providing a disservice to students in part of their ability to connect various content together into coherent understanding. However, because students are enjoying the format of these exams, it is difficult to determine if they should be completely removed. A happy medium of the two may involve additional written exercises in order to reinforce material and concepts, such that they are better understood in the long run.

Through doing this project, I've learned that there are many small and large manners that educators can test and see if something is working for students or not. Something that particularly sticks out to me is that these projects are done in mind for the students, such that they are having better experiences and are being able to learn the best. I would like to keep these values and lessons in mind as I continue to move forward as a mentor in the lab as well as educator in future instances. Further, TAR projects and questions can be done in various scales -- most importantly on smaller scales...! There does not need to be much ingenuity behind a project other than a change and tracking the effect of the change, which I think projects such as these find their charm.

I was most excited about being able to perform the justification assessment. Once I found DeBoer et al.'s paper, it sparked something within me that I wanted to immediately try it out. One of the major challenges of this project was the hand scoring and coding of the written responses by the students. While I knew this was going to be challenging to begin with, once I began, the amount of time and struggle to code was more apparent.

About the Author

Keila obtained her Bachelors in Physiology at the University of Arizona in 2020, followed by a Masters in Physiological Sciences in 2022. Today, she is training as a PhD student within the PSGIDP. If you've been keeping track -- thats 7+ years at the UofA! Her previous work has focused on sex differences in metabolic disease, while her current research is focused on the role that sphingolipids play in immune cells and intestinal disease. On the teaching side of endeavors, Keila has taught lab sections at the UofA, and hopes to teach in some respect after graduation (2026, still some time to go...!).

Outside of the lab, Keila enjoys spending time with family and friends. Her hobbies include playing video games, being involved in game nights, and trying new restaurants throughout Tucson.

Should there be any comments, concerns, or general inquiries please feel welcome to email at espinoza5@arizona.edu.

This project was reviewed by the University of Arizona's IRB and deemed not to be human research.

Page updated

Report abuse