Netherlands education fraud algorithm ruled discriminatory and illegal
Netherlands education fraud algorithm ruled discriminatory and illegal
Occurred: 2012-2023
Page published: December 2024
An algorithm used by the Netherlands' Education Executive Agency (Dienst Uitvoering Onderwijs - DUO) to detect student finance fraud was found by the government to be discriminatory, leading to a commitment to refund over 10,000 affected students.
Implemented in 2012, DUO's algorithm flagged students for fraud using a rule-based algorithm to calculate a risk score for all students ranging between 0 and 180 points. Profiles with a higher points were deemed to have a higher risk score.
A June 2023 media investigation indicated that 97 percent of the students flagged for fraud were immigrants from ethnically diverse backgrounds, particularly individuals of Turkish and Moroccan descent.
The investigation prompted accusations that the algorithm was based on seemingly arbitrary criteria and lacked any scientific basis.
In response, the Minister of Education discontinued the system and commissioned additional external research. The country's privacy watchdog also announced an investigation.
The government acknowledged that DUO's selection methodology amounted to "indirect discrimination"; in November 2024, it announced a refund plan of 61 million euros to compensate students regardless of the initial fraud findings.
The algorithm's design, which drew on data about students' age, level of education and the distance lived from their parents' address - is considered to have inadvertently led to ethnic profiling.
It is unclear why DUO selected the criteria it used to assess student fraud risk.
The case highights the urgent need for ethical considerations and accountability in the deployment of algorithms within public services.
The government's acknowledgement and offer of recompense is seen to have marked a shift in Dutch policy regarding the use of algorithms in public administration.
That said, organisations such as Amnesty International caution that significant risks remain unless comprehensive safeguards against ethnic profiling are established across government agencies.
DUO Fraud Detection System
Developer: Dienst Uitvoering Onderwijs
Country: Netherlands
Sector: Education
Purpose: Detect fraud
Technology: Risk profiling algorithm
Issue: Accountability; Bias/discrimination
2012: DUO starts using its Fraud Detection risk-profiling algorithm to detect grant fraud.
2012–2022: Over 350,000 students are screened; approximately 21,500 are selected for deep checks.
June 2023: Media investigations expose that the system almost exclusively targets students with migration backgrounds; the Minister of Education halts the system.
March 2024: An external audit by PwC and Algorithm Audit confirms "indirect discrimination."
November 2024: The Dutch Data Protection Authority rules the algorithm illegal and discriminatory; the government announces EUR 61 million in compensation for roughly 10,000 victims.
Autoriteit Persoonsgegevens. Onderzoeksrapport fraudeaanpak DUO
Amnesty. Profiled without Protection
https://nltimes.nl/2024/11/12/government-refund-10000-students-discriminatory-duo-fraud-checks
https://www.dutchnews.nl/2024/07/discriminatory-algorithms-still-widely-used-watchdog-warns/
https://nltimes.nl/2023/06/21/ethnically-diverse-students-screened-fraud-remarkably-often
https://www.groene.nl/artikel/bewijs-tegen-vermeend-frauderende-studenten-was-ontoelaatbaar
AIAAIC Repository ID: AIAAIC1834