‘The Algorithm Didn’t See Me’ A case to explore Diversity, Inclusion, and Accessibility in AI for Education

ENGAGEMENT & DEMOCRATIC GOVERNANCE IN AI

How to cite this learning scenario

Arantes, J. (2025). The Algorithm Didn’t See Me. Case Studies in AI Governance for Education. www.AI4education.org. Licensed under a Creative Commons Attribution 4.0 International License.
abstract
This case study investigates how AI tools in education can unintentionally exclude or harm students when diversity, inclusion, and accessibility are not built into their design. Set in a multicultural urban school district, the fictionalized but research-informed narrative follows the deployment of a generative AI tool used for academic writing support. While intended to improve student outcomes, the tool failed to accommodate multilingual learners, neurodivergent students, and those with disabilities. Teachers reported bias in the AI’s feedback, with student writing penalized for non-standard English or divergent thought patterns. The case asks: who is AI really designed for—and who gets left behind?

If AI in education doesn’t recognise diverse identities, languages, and ways of learning—it reinforces the very barriers it promises to break. Inclusion must be designed, not assumed.

The Algorithm Didn’t See Me

In 2024, a large public education system introduced an AI writing assistant to support student literacy across secondary schools. The tool, designed to provide real-time feedback on grammar, structure, and clarity, was praised for its potential to reduce teacher workload and support struggling writers. However, within weeks of deployment, teachers began noticing troubling patterns: students from culturally and linguistically diverse backgrounds were receiving disproportionately negative feedback. The AI flagged Indigenous expressions as “incorrect,” penalized African American Vernacular English, and recommended simplifying content from neurodivergent students who used vivid, creative language. Students with disabilities found the platform inaccessible—screen readers were incompatible, and there were no options for voice input or alternative communication modes. English as an Additional Language (EAL) students felt demoralized, reporting that the tool seemed to “erase their voice.” Teachers were forced to intervene regularly, undermining the tool’s time-saving premise. An internal audit later revealed that the AI model had been trained primarily on standard academic English from U.S. and UK-based datasets, with limited representation from other dialects or linguistic patterns. After advocacy from disability and multicultural education groups, the education department paused the program and committed to a full review, co-led by affected communities. This case highlights the critical importance of embedding equity, accessibility, and cultural responsiveness into AI systems from the outset—not retrofitting them after harm occurs.

research topics

research questions

Evaluate how AI tools may reinforce linguistic, cultural, and ableist biases if inclusion is not intentional. Identify inclusive design principles to guide AI tool development in education. Analyze the relationship between accessibility, justice, and student agency in digital learning environments. Develop strategies for selecting or auditing AI tools that promote equity, linguistic diversity, and universal design.
What design and training practices could have made the AI tool more inclusive from the start? How can educators recognize and respond when AI technologies replicate or amplify bias? What principles should guide the development and procurement of AI tools in diverse school contexts? How can students with disabilities or those from linguistically diverse backgrounds be part of the AI design process? In what ways can AI support—not suppress—plurality in student expression and identity?

data collection

Conduct a small-scale audit of current digital tools in your school or system. Who benefits most? Who is left out? Create a checklist of inclusive design principles to apply to future AI procurement or development projects. 

Do you want to know more?
© Copyright 2024 Web.com Group, Inc. All rights reserved. All registered trademarks herein are the property of their respective owners.
Subscribe to the AIGE Newsletter
Acknowledgement of Country We acknowledge the Ancestors, Elders and families of the Kulin Nation (who are the traditional owners of the land. where this work has bene predominantly completed. As we share our own knowledge practices, we pay respect to the deep knowledge embedded within the Aboriginal community and recognise their ownership of Country. We acknowledge that the land on which we meet, learn, and share knowledge is a place of age-old ceremonies of celebration, initiation and renewal, and that the Traditional Owners' living culture and practices have a unique role in the life of this region.