Transforming an Open Book, Online Assessment Using AI 

What we did

Upon evaluating our assessments, we realised that we could use AI to generate essays that students could critique and evaluate, drawing on the knowledge and understanding they developed during the module. We therefore implemented a new essay based, open book, online assessment.  Instead of giving students a title, we gave them an essay, and asked them to critically evaluate the content, in particular looking for what was incorrect and what was missing.  We also got them to review citations for authenticity and appropriate use within the essay.  Instead of simply providing information on a topic, students are now required to demonstrate critical analysis.  

Of course, it was important that we also embedded support for this new assessment approach into the module.  We discussed strategies on how to write the essay with students, and provided marking criteria in the form of a rubric.  

In our active learning sessions, instead of asking them questions, we provided sample essays, and asked them to work in groups to evaluate these essays, capturing their ideas in Wooclap, so that everyone could benefit from seeing points made by the whole cohort.  Once these teaching sessions were complete, a marked up version of the essay was provided, so that students could see what they had (and had not) identified in their discussions.       

Reflections

This assessment approach worked really well, with the majority of students engaging with the materials and activities we had provided.  In Tell Us responses, many students mentioned that module sessions were particularly useful in developing their skills and preparing them for the assessment.  Key to this success was the support we provided; a guidance document, assessment video and marking criteria.  Importantly we also evaluated sample essays in active learning sessions.  

However, even with all this support some students worried about writing, what was for them, a very different type of essay.  Students grasped the critical evaluation aspect of the assessment, but were worried about articulating their thoughts and ideas (the most common question we were asked was about how to write the essay). 

Reflecting on this, we have decided to rework our “how to write your essay” section, writing the guidance in a more direct way (rather than keep it 3rd person), and share the cohort assessment feedback, so that students can see direct information on what was done well, what needed to improve and the strategies to improve. Students who completed this assessment in 2023/24 are now taking modules in third year, and they tell me that the module and assessment approach we took has helped prepare them for critical analysis of research papers, a key skill for success.

The other consideration is the changing nature of AI, and the need to constantly evaluate our assessment approach in the context of these changes.  We are currently evaluating what changes we may need to make going forward, looking to potentially incorporate student use of AI into the assessment. AI clearly forms a key aspect of how our graduates will be working in the future, and we need to ensure they have the skillset needed for future success.  

What our students said

In the Student Staff Committee meeting, students reported that they found the active learning sessions helpful regarding how to compare essays.

In Tell Us responses, many students mentioned that module sessions were particularly useful in developing their skills and preparing them for the assessment.