Human in the Loop



'Hands Off Learning’ A case to explore the Role of Human Oversight Throughout AI System Lifecycles in Education

MEANINGFUL HUMAN CONTROL IN AI SYSTEMS

How to cite this learning scenario

Arantes, J. (2025). Human in the Loop. Case Studies in AI Governance for Education. www.AI4education.org. Licensed under a Creative Commons Attribution 4.0 International License.
abstract
This case study investigates the dangers of diminishing human control in AI-driven educational environments. Drawing on real-world practices, the fictionalised scenario follows a K–12 school system that implemented AI systems to handle student engagement tracking, curriculum delivery, and even behavioural interventions. While initially welcomed as time-saving, these systems gradually replaced teacher judgement and student voice. As the AI tools became more autonomous, opportunities for critical reflection and human intervention disappeared. This case reinforces the need for meaningful human control at all stages of the AI lifecycle—from design and deployment to review and retirement.

AI should assist human decision-making, not replace it. Without meaningful oversight, we risk turning educators into observers—and students into data points.

Hands Off Learning

In 2024, Brightstream Schools adopted a suite of AI-powered learning tools under a “Smart Education” initiative. The tools promised personalised content delivery, real-time engagement analytics, and automated behavioural nudges. Teachers were encouraged to rely on AI-generated reports to guide lesson adjustments, attendance interventions, and even conflict resolution strategies. At first, staff appreciated the automation—but over time, it became clear that the AI was shaping not just how learning occurred, but what was taught, who received attention, and when disciplinary action was triggered. Teachers reported they had less time for relational learning, and more pressure to follow AI recommendations, even when they conflicted with their own professional judgement. Some students received automatic warnings or were denied extensions based on the algorithm’s predictions of “low effort.” Crucially, the system didn’t allow teachers or students to question or override many of its functions. There was no embedded process for review, and updates were made without educator consultation. A growing sense of disempowerment emerged among teachers, while students expressed frustration that they felt “managed” rather than taught. A review commissioned after a high-profile error revealed that human control had been gradually eroded—not deliberately, but through a series of defaults and design decisions. The system lacked override features, didn’t prompt for human confirmation on high-impact decisions, and had no built-in requirement for ongoing educator training. In response, Brightstream implemented new policies mandating human-in-the-loop protocols, including teacher approval for automated feedback, redress options for students, and routine audits of AI outputs. Teachers were retrained in critical data literacy, and students were invited into review panels to ensure transparency and trust. This case shows that meaningful human oversight must be structurally embedded—not assumed—and maintained across the full lifecycle of AI in education.

Research Topics

Research Questions

Understand the concept of meaningful human control in the context of educational AI systems. Identify risks associated with over-reliance on automated systems and the erosion of teacher and student agency. Explore ways to design, deploy, and monitor AI tools with embedded human oversight mechanisms. Develop strategies to strengthen human-led review, reflection, and intervention processes throughout the AI lifecycle.
What does “meaningful human control” look like at different stages of an AI system’s lifecycle in education? How can human judgement be preserved when AI recommendations are seen as “more objective”? What risks emerge when AI systems operate with limited or no override capacity? How can institutions ensure human oversight remains central as AI systems become more autonomous? What design and policy features support ongoing educator and student involvement in AI decision-making?

Data collection

Facilitate a mapping activity with staff and students to identify points of human oversight and gaps in accountability within a currently used AI tool.
Do you want to know more?
Acknowledgement of CountryWe acknowledge the Ancestors, Elders, and families of the Kulin Nation, who are the Traditional Owners of the land where this work has been predominantly completed. As we share our own knowledge practices, we pay respect to the deep knowledge embedded within the Aboriginal community and recognise their custodianship of Country. We acknowledge that the land on which we meet, learn, and share knowledge is a place of age-old ceremonies of celebration, initiation, and renewal, and that the Traditional Owners’ living culture and practices continue to have a unique role in the life of this region.
Subscribe to the AIGE Newsletter
© Copyright 2024 Web.com Group, Inc. All rights reserved. All registered trademarks herein are the property of their respective owners.