Assess: Redesign, Evaluate, & Reward

Ethical implementation requires maintaining ultimate human oversight, safeguarding student data privacy, and clearly communicating assessment guidelines. Here are a few principles to keep in mind for that scenario:‍

1. Transparency: If the AI system can’t explain a given score, the system isn’t really providing a thorough evaluation. A clear rubric and specific feedback against the criteria are required.

2. Reproducible: For an evaluation to be fair, it must be reliable, meaning that the given system with the same input, and the same rubric, will provide identical scores and feedback. 

3. Bias-aware: AI grading tools can unintentionally score students unfairly due to flaws in their training data and design. 

   • AI trains on historical data, which mirrors past systemic grading disparities. 

   • Systems prioritize "Standard Academic English," which penalizes English Language 

   • Learners and regional dialects. Algorithms can score identical work differently based on names or demographic markers. 

   • Models built on data from affluent schools fail to accurately assess diverse student backgrounds.

4. Humans in the loop: 

   • Student-focused.
     Require students to read AI feedback prior to submission for grading, rate of the feedback on a scale of 1 to 5, redo if needed, and write a brief reflection on which critiques they accept or reject. 

   • Faculty-centered.
     Consider 3-Way Rubrics using a matrix where the AI assesses technical constraints (e.g., citation structure), the student self-assesses, and you grade core concepts.

5. Honesty and Integrity: For better or worse, people are using generative AIs for things they are judged on. To address this challenge, explore alternative assignments where learners can use AI as part of their process but are ultimately responsible for something that goes beyond what the AI produces.‍ 

An AI-resilient assessment evaluates skills that generative AI struggles to replicate, such as critical thinking, lived experience, process over product, and personal reflection. 

Rather than trying to "AI-proof" assignments by banning technology, AI-resilient assessments encourage uniquely human capabilities or integrate AI as a collaborative tool. Key characteristics of an AI-resilient assessment include:

• Focus on Process, Not Just the Final Product: Evaluate the journey of learning by requiring process journals, drafts, peer reviews, or reflective writing.

• Authentic and Contextual Tasks: Ask students to apply concepts to real-world scenarios, local communities, or recent current events. AI lacks access to personal or hyper-localized context.

• Dialogue and Defense: Utilize oral examinations, Socratic seminars, or brief Q&A presentations to verify a student's intellectual ownership of the material.

• Multimodal Submissions: Require students to represent knowledge beyond text, such as through hand-drawn diagrams, concept maps, or recorded audio/video.

• AI as a Tool: Explicitly incorporate generative AI into the prompt, such as:

• Have students generate a first draft with AI, critique its mistakes, and rewrite it, citing their prompt history.

• Have students collaborate with AI to research a specific topic, followed by an oral defense of their work.

• Encourage students to feed their own work into AI, request a critique, and compare the AI’s feedback with their own reflections. 

See AI Proofing Assignments and Ensuring Authentic Student Work (2:47) to learn strategies for ensuring students genuinely engage with course material and produce authentic work. 

Overview

• Assessment and Rubric Development (Heaps, 2024). This chapter explores how AI tools can: assist with creating assessment questions based on supplied prompts or materials; generate project topics and assessment scenarios; and help to develop rubrics.

• Leveraging AI for Creating Rubrics (3:10). Shares how educators can use GenAI to create grading rubrics to support student learning outcomes.

Criteria

Here are copy-and-paste rubric criteria descriptions for your syllabus or assignment guidelines.

Process & Metacognition

• Draft Evolution & Revision: Demonstrates clear, substantive growth across multiple iterations. The final product explicitly incorporates and builds upon feedback received on early drafts, logic, or structure.

• Metacognitive Reflection: Provides a deep, self-aware analysis of the learning journey. The student accurately explains why they made specific choices, how they overcame obstacles, and how their thinking changed.

• AI Collaboration & Prompting: Displays advanced skill in guiding AI tools. Prompts are structured, iterative, and strategic, showing a clear evolution of thought to refine the generated output.

Critique & Verification

• AI Output Critique: Demonstrates sharp critical analysis of AI-generated content. Successfully identifies hallucinations, factual gaps, biased assumptions, or surface-level arguments within the AI output.

• Fact-Checking & Source Verification: Shows rigorous attention to detail by manually verifying all claims. Every fact, quote, and piece of data is cross-referenced with trustworthy external evidence.

• Advanced Sourcing & Attribution: Uses highly specific, niche, or hyper-local citations. Sources include class readings, specific archives, or specialized data that standard AI models cannot easily scrape or access.

Authentic Application

• Localized & Contextual Application: Applies theoretical concepts directly to a specific community, neighborhood, or recent real-world event. The analysis relies on unique, localized details rather than generic information.

• Lived Experience & Voice: Integrates the student’s unique lived experiences, professional background, or personal perspective. The writing maintains a distinct, authentic human voice throughout.

• Real-World Problem Solving: Proposes a practical, nuanced solution to a complex, unpredictable human scenario. The strategy accounts for messy real-world constraints like budget, politics, or human emotion.

Interactive & Multimodal Mastery

• Oral Defense & Live Q&A: Demonstrates deep intellectual ownership of the material during a live conversation. The student responds to spontaneous questions confidently, accurately, and without relying on pre-written scripts.

• Multimodal Synthesis: Integrates non-text elements—such as hand-drawn diagrams, video recordings, or concept maps—to explain complex ideas. These elements directly enhance and clarify the text.

• Collaborative Process & Peer Feedback: Provides constructive, actionable feedback to peers and documents individual contributions to the group. The student shows active engagement in the collaborative learning environment.

The following Quality Matters™ specific standards provide a good checklist for course assessments:

• 3.1 The assessments measure the achievement of the stated learning objectives or competencies.

• 3.2 The course grading policy is stated clearly at the beginning of the course.

• 3.3 Specific and descriptive criteria are provided for the evaluation of learners’ work, and their connection to the course grading policy is clearly explained.

• 3.4 The assessments used are sequenced, varied, and suited to the level of the course.

• 3.5 The course provides learners with multiple opportunities to track their learning progress with timely feedback.