Task Design for the AI Era

(Adrian Cotterell)

Three paradigm shifts needed as we move into the AI era

1) Verification > AI Detection

Teachers didn't get into teaching to become 'cheating police'. Too much time is being wasted trying to detect the inclusion of AI generated content in unsupervised assessment tasks.

Let's redefine our definition of cheating! Rather than expecting students not to use Generative AI across their unsupervised assessments, we need to assume AI will be used and adjust our assessment practice.

2) Start with the Assessment Construct

When redesigning assessments, start with the assessment construct... not with the activity itself or the 'level' of AI.

The assessment construct is the purpose of the assessment itself.

What is the 'thing' you are trying to evaluate?

For example, here are the assessment constructs of an academic essay. We are attempting to assess an individual's ability to:

produce a clear and coherent thesis
organise and structure arguments
demonstrate content knowledge
demonstrate critical analysis and evaluation
support arguments with synthesised and well-referenced evidence
write accurately with engaging style

Using the product of the essay, we are actually trying to assess the above processes. The value is not in the product itself, but the processes that create the product.

Thus, the academic essay becomes useless when Generative AI completes an essay for a student.

We can no longer assess the human thinking behind the product.

3) Three categories of assessment in the AI era

Supervised Tasks - AI is not permitted
Layered Homework Tasks - AI is assumed
AI Tasks - AI is expected

Three categories of assessment in the AI era

Supervised Tasks - AI is not permitted

The purpose of supervised tasks in the AI era is to hear the human voice of our students.

Generative AI is very noisy! The only way we can validly hear the voice of our students across many assessment constructs is for the students to complete the task in supervised conditions.

This means we don't need to chuck out the academic essay! Just don't set essays for homework anymore!

Make sure they are completed in supervised assessment conditions.

Supervised assessments in AI era should be supportive.

Supervised tasks are not just tests and exams!

They can be:

Open book
Open Teacher (Stop saying "I can't help" in supervised assessments! Instead, provide personalised support based on individual needs)
Collaborative Tasks
Oral Reports with no script (students can use notes for a reference... but let's raise our expectations. Don't tolerate students consistently reading from a script).

Layered Homework Tasks - AI is assumed

If we don’t want students using AI for any part of a task... it should be a Supervised Task.

But not every assessment should be supervised!

In most subjects, we can still give students homework-based assessments.

It's important to note that we can't aim for 'AI-proof' unsupervised assessments. That is nearly impossible as Generative AI is only getting more powerful.

But let's get real! We've never had 'cheating-proof' assessments! So let's not aim for something we've never had.

The key is that we add additional 'layers' to our unsupervised tasks to increase the level of assessment security.

Here are some examples:

Part A (supervised) & Part B (unsupervised)
Adding very niche content
Document tracking
In-class written reflection
1:1 ‘thesis defence’ or 'oral defence'
- (check out this AI-powered assessment and verification platform I am developing to help educators automate this process)

These layers are not focused on stopping students from using AI... or trying to detect the inclusion of AI generated content.

Rather, we add these layers to help us verify student understanding of the assignments they submit.

AI Tasks - AI is expected

Supervised Tasks and Layered Homework Tasks are responding to the challenge of AI in Education.

AI Tasks are embracing the opportunity generative AI is bringing to educational assessment.

The key assessment construct of an AI task is to evaluate a student's ability to use generative AI productively and creatively.

If we can teach our students to leverage this powerful emerging technology, we should then assess their ability to use it.

This doesn't mean we "teach students to use AI to write an essay"... what a terribly boring and pointless task.

Rather, this is where teachers can get creative themselves to explore how we can integrate generative AI authentically into our subject domains.

Here are some examples for an English and History context (click on URLs for further information):

Students use AI technologies to create a multimodal advertising product.
Interactive Fiction - turn a student narrative into a 'choose your own adventure' using generative AI
Socratic Dialogue AI - Rather than having AI chatbots giving answers, use the simple prompts provided to get the AI to only give adaptive questions.
Historical Roleplays - Have students experience the first person perspective of an ordinary person living in a specific historical era and context.

Model of Assessment of Redesign

Page updated

Google Sites

Report abuse