Netherlands education fraud algorithm ruled discriminatory and illegal
Netherlands education fraud algorithm ruled discriminatory and illegal
Occurred: 2012-2023
Report incident ๐ฅ | Improve page ๐ | Access database ๐ข
An algorithm used by the Netherlands' Education Executive Agency (Dienst Uitvoering Onderwijs - DUO) to detect student finance fraud was recognised by the government as discriminatory, leading to a commitment to refund over 10,000 affected students.
Implemented in 2012, DUO's algorithm flagged students for fraud using a rule-based algorithm to calculate a risk score for all students ranging between 0 and 180 points. Profiles with a higher points were deemed to have a higher risk score.ย
A June 2023 media investigation indicated that 97 percent of the students flagged for fraud were immigrants from ethnically diverse backgrounds, particularly individuals of Turkish and Moroccan descent.ย
The investigation prompted accusations that the algorithm was based on seemingly arbitrary criteria and lacked any scientific basis.
In response, the Minister of Education discontinued the system and commissioned additional external research. The country's privacy watchdog also announced an investigation.
The government acknowledged that DUO's selection methodology amounted to "indirect discrimination"; in November 2024, it announced a refund plan of 61 million euros to compensate students regardless of the initial fraud findings.
The algorithm's design, which drew on data about students' age, level of education and the distance lived from their parents' address - is considered to have inadvertently led to ethnic profiling.
It is unclear why DUO selected the criteria it used to assess student fraud risk.
The case underscores the urgent need for ethical considerations and accountability in the deployment of algorithms within public services.ย
The government's acknowledgement and offer of recompense is seen to have marked a shift in Dutch policy regarding the use of algorithms in public administration.ย
That said, organisations such as Amnesty International caution that significant risks remain unless comprehensive safeguards against ethnic profiling are established across government agencies.
Rule-based system
In computer science, a rule-based system is a computer system in which domain-specific knowledge is represented in the form of rules and general-purpose reasoning is used to solve problems in the domain.
Source: Wikipedia ๐
DUO
Operator:ย
Developer: Dienst Uitvoering Onderwijs (DUO)
Country: Netherlands
Sector: Education
Purpose: Detect fraud
Technology: Rule-based algorithm
Issue: Bias/discrimination
Autoriteit Persoonsgegevens. Onderzoeksrapport fraudeaanpak DUO
Amnesty. Profiled without Protection (pdf)
https://nltimes.nl/2024/11/12/government-refund-10000-students-discriminatory-duo-fraud-checks
https://www.dutchnews.nl/2024/07/discriminatory-algorithms-still-widely-used-watchdog-warns/
https://nltimes.nl/2023/06/21/ethnically-diverse-students-screened-fraud-remarkably-often
https://www.groene.nl/artikel/bewijs-tegen-vermeend-frauderende-studenten-was-ontoelaatbaar
Page info
Type: Incident
Published: Deceember 2024