Evaluating the Impact of GenAI on Assessments

In AIO, we use the term "assessments" as an umbrella term for any mechanism by which an instructor gathers information about student learning and evaluates the extent of that learning. So, tests, exams, quizzes, assignments, papers, projects, homework, and so on, are all forms of assessment. 

All assessments should be clearly tied to the course learning outcomes.

Giving and evaluating assessments are mission critical tasks for instructors because the results of these assessments lead to a course grade and the accumulation of course grades leads to the conferring of a certification (e.g., degree or diploma). So, for our certifications to have and retain value, there must be integrity within the assessment process. This means that:

• instructors have to design fair and honest assessments

• students have to fairly and honestly demonstrate their learning on those assessments, and then 

• instructors have to fairly and honestly evaluate the learning that the student demonstrated.

If anything corrupts that process, then the assessments, and eventually the certification, lose value.

There are two main types of assessments:

1. Formative assessments are those that we give students to facilitate their learning and maybe to ascertain if they are making progress as we expect.

2. Summative assessments are those intended to evaluate the extent to which the students have achieved the learning outcomes.

Since the main goal of formative assessments are to facilitate learning or the acquisition of skills, then they usually do not involve a judgment or evaluation in the form of a grade (which would up their stakes). More importantly for formative assessments is providing the students with feedback so they understand if they are making progress and if not, what they need to do differently to make progress. Examples of formative assessments include:

• practice problems

• drafts

• peer reviews

• discussion board posts

• pre-lab or pre-discussion quizzes

On the other hand, the main goal of summative assessments is to evaluate if a student has achieved a learning outcome and to what extent. Such assessments determine if a student should pass the course, move on to the next course in the sequence, and, eventually, graduate. Summative assessments usually take the form of:

• tests/exams

• final papers

• projects

Obviously, then, it's most important that we ensure integrity in summative rather than in formative assessments.

There are several reasons why assessments might need redesigning in the age of AI:

1. We can't "ban" our way out of this.

Many instructors turned to simply banning the use of AI on assessments, yet continued to deliver those assessments in unsecure environments (e.g., remote tests; out-of-class written assignments). But 50% of college students say they'll use the tools even when faculty ban them, so by banning AI on unsecured assessments doesn't reduce cheating and therefore creates an unfair learning environment.

2. Trying to detect AI use is not sustainable

Instructors thought they could ban AI use on unsecured assessments because they also thought violations of the ban could be detected. And, when ChatGPT was first released, it was fairly easy to tell when a student used it. It might be getting harder and harder though. There are plenty of tools to "humanize" AI output and students are learning how to work with the tools to disguise their use (you can skip ahead to about 2:50 time). In AIO's experience, it's often difficult to tell a bad paper (that the student wrote) versus a dishonest paper (that an AI wrote) - the evidence is the same!

3. Securing every assessment isn't feasible

One might think that they can secure every assessment to deal with AI. However, it's not necessarily doable for most instructors to secure every assessment, perhaps because a secure assessment (like a test) is not the best way to assess a student's knowledge in a particular course, or because there isn't sufficient class time to give as many assessments as are needed to evaluate the achievement of course learning outcomes.

4. Our graduates may be expected to be AI competent

For our students to be prepared for life after graduation, they may need to have AI Literacy and experience using AI. Employers, for example, seem to "consider AI part of a larger technology competency students will need for the future and to be applied alongside other skills". While it seems many students might actually be resistant to using AI, it could be just from a lack of exposure. So, redesigning assessments to incorporate in AI use might be advantageous for the students now (reducing unethical use) and for the students in the future.

5. Assessment refreshing raises intrinsic motivations

Even prior to GenAI, assessments needed to be revised from time to time due to advances in technology, changes in student population, advancements in the discipline, and changes to learning outcomes. These changes to keep assessments up-to-date are necessary to engender intrinsic motivations for learning. When students see assessments as "out-of-touch" and easy to complete with GenAI, they are more likely to cognitively offload their thinking and doing to GenAI. 

Before you begin redesigning any assessments, you will want to experiment using GenAI tools to respond to them. Here's a simple way to do this:

1. put a test question or assignment prompt into a GenAI tool (ChatGPT seems to be the most commonly used tool by students)

2. evaluate the output according to your answer key or grading rubric

3. If the tool provides an output that garners a C grade or higher, consider changing the question or prompt because it's likely that a student could use GenAI to complete it and earn a passing grade. Considering even "Cs get degrees", it's important that we address