abstract

This case study explores the ethical and procedural gaps that emerge when students and educators are given no avenue to contest or appeal AI-generated decisions. Set in a senior secondary college, the fictionalised scenario describes the rollout of an AI-powered assessment moderation tool. Despite its promise to ensure consistency and fairness, the tool began generating questionable grades and behaviour alerts—without clear pathways for students or teachers to challenge the outcomes. This case highlights the fundamental need for transparent contestability, human review, and accountability frameworks that uphold due process in education systems using AI.

The System Said So

In 2024, Everton Senior College implemented an AI-based assessment moderation system that analysed student essays, marked them using predictive grading, and flagged any anomalies for review. The tool was designed to ensure equity across subjects and reduce teacher bias. However, over time, concerns grew.

Students began receiving grades that did not align with their past performance or teacher expectations. Appeals were dismissed on the basis that the algorithm had “detected deviation patterns.” Teachers were instructed not to override the system unless a formal error could be proven—but the system's logic was not explainable, and its decisions were opaque.

When one student, Zara, received a failing mark on a major assessment—despite positive feedback from her teacher—the family attempted to appeal. There was no process in place. The AI’s decision was final. Frustrated and disillusioned, Zara withdrew from the subject. Her teacher later discovered that the system had flagged her essay due to “high lexical similarity” with public texts, though it wasn’t plagiarised—it referenced published research.

The situation sparked widespread concern. A collective of staff and students demanded the establishment of a right to contest AI decisions. The school responded by creating an independent review process, introducing a transparent appeals policy, and mandating human moderation for all critical decisions. They also initiated workshops to help students and staff better understand how AI assessments work—and how to challenge them if needed.

This case underscores the need for explainability, transparency, and accessible redress in all AI systems used in education—particularly those that impact grades, wellbeing, or progression.

Overview

discussion and application 

This case challenges educational institutions to ensure students and teachers retain agency, dignity, and procedural fairness in AI-influenced decision-making processes.﻿
Discussion Questions 

Learning Objectives

Participants will:

Understand the critical importance of contestability and human oversight in AI decision-making.

Identify gaps in current institutional procedures related to AI transparency and redress.

Explore strategies to build formal, accessible, and inclusive pathways for appeal and review.

Consider how to communicate student and educator rights clearly within AI-integrated systems.

What risks arise when students and staff cannot challenge AI-generated outcomes?

How can educational institutions build fair and transparent appeal processes for AI-based decisions?

What role should human review play in moderating automated outputs?

What mechanisms can ensure students understand their rights when affected by AI tools?

How can schools and universities foster a culture of accountability and responsiveness in AI use?

‘The System Said So’ A case to explore the Right to Contest and Challenge AI Outcomes in Education

How to cite this learning scenario

abstract

AI is not infallible. When students and teachers cannot question decisions made by algorithms, we risk turning education into an automated injustice system. Contestability is not optional—it’s a right.

The System Said So

Overview

discussion and application

Prompts for practice:

AI Governance for Education: case studies

CONTACT