AI detectors falsely accuse students of cheating

Occurred: October 2024

AI writing detection systems regularly falsely accuse students of cheating, resulting in anxiety, paranoia and time-wasting, according to a Bloomberg BusinessWeek investigation.

What happened

Moira Olmstead, a student at Central Methodist University, was falsely accused of cheating on an assignment due to an AI detection tool's misidentification of her work as AI-generated. 

After returning to school following maternity leave, Olmstead submitted a written assignment that was later flagged by the AI detector, resulting in a grade of zero. 

Olmstead's writing style, influenced by her autism spectrum disorder, tends to be formulaic, making it more susceptible to being flagged incorrectly by AI detection systems.

Why it happened

Turnitin, the AI detection tool used in Olmstead's case, is part of a broader trend where nearly two-thirds of educators employ these technologies to combat academic dishonesty. 

However, these tools have been shown to misidentify 1-2 percent of human-written texts as AI-generated. And even a small error rate can impact large numbers of students.

Despite their known error rates, the pressure on educators to maintain academic integrity has led to an increased use of these tools.

What it means

The false accusation against Olmstead underscores the potential dangers of relying on AI detection systems in education. 

Such inaccuracies can lead to severe academic repercussions for students - including anxiety, paranoia, failing grades, inappropriate disciplinary actions and damaged reputations - and reduced trust for education systems using AI detection tools.

The situation raises ethical questions about the implementation of AI in educational contexts and the need for more accurate detection methods. 

Artificial intelligence content detection

Artificial intelligence detection software aims to determine whether some content (text, image, video or audio) was generated using artificial intelligence (AI).

However, the reliability of such software is a topic of debate, and there are concerns about the potential misapplication of AI detection software by educators.

Source: Wikipedia 🔗

System 🤖

Operator: Moira Olmstead
Developer: Turnitin
Country: USA
Sector: Education
Purpose: Detect AI writing
Technology: Machine learning
Issue: Accuracy/reliability; Ethics/values

Research, advocacy 🧮

Investigations, assessments, audits 👁️

Page info
Type: Incident
Published: October 2024