Testing, Monitoring & Continuous Evaluation
It Wasn’t Ready
This case study explores the consequences of failing to adequately test AI models before deploying them in educational contexts. Based on sector-wide trends and current concerns, this fictionalised scenario follows a public education department that fast-tracked a student risk profiling tool without sufficient testing. The result was a series of inaccurate predictions, reputational damage, and harm to student wellbeing. This case highlights the urgent need for robust pre-deployment testing protocols that assess AI models for reliability, fairness, usability, and alignment with pedagogical values—before they are used to make or influence high-stakes decisions.
It Drifted
It Solved the Wrong Problem
This case study explores how AI systems in education can drift from their original design intentions over time—leading to harmful or unintended consequences if not routinely monitored. The fictionalised scenario follows a university that implemented an AI tool to support student wellbeing through predictive alerts and engagement tracking. Initially successful, the system began to generate false positives and stigmatise vulnerable students due to model drift and unmonitored updates. This case highlights the need for ongoing performance audits, user feedback loops, and ethical reviews to ensure AI remains aligned with educational values and community expectations.
This case study explores the risks of educational AI tools drifting from their original pedagogical goals and institutional values. It follows a fictionalised scenario in which a school network adopts an AI platform to support student learning—but over time, the tool begins to prioritise efficiency and performance metrics over deep learning, creativity, and critical thinking. As teachers and students are pushed to adapt to the tool’s preferences, the broader mission of the institution is quietly compromised. This case highlights the need for regular reflection, value-alignment audits, and shared leadership to ensure AI tools support—not override—educational purpose.