Netherlands education fraud algorithm ruled discriminatory and illegal

Occurred: 2012-2023

Report incident ๐Ÿ”ฅ | Improve page ๐Ÿ’ | Access database ๐Ÿ”ข

An algorithm used by the Netherlands' Education Executive Agency (Dienst Uitvoering Onderwijs - DUO) to detect student finance fraud was recognised by the government as discriminatory, leading to a commitment to refund over 10,000 affected students.

What happened

Implemented in 2012, DUO's algorithm flagged students for fraud using a rule-based algorithm to calculate a risk score for all students ranging between 0 and 180 points. Profiles with a higher points were deemed to have a higher risk score.ย 

A June 2023 media investigation indicated that 97 percent of the students flagged for fraud were immigrants from ethnically diverse backgrounds, particularly individuals of Turkish and Moroccan descent.ย 

The investigation prompted accusations that the algorithm was based on seemingly arbitrary criteria and lacked any scientific basis.

In response, the Minister of Education discontinued the system and commissioned additional external research. The country's privacy watchdog also announced an investigation.

The government acknowledged that DUO's selection methodology amounted to "indirect discrimination"; in November 2024, it announced a refund plan of 61 million euros to compensate students regardless of the initial fraud findings.

Why it happened

The algorithm's design, which drew on data about students' age, level of education and the distance lived from their parents' address - is considered to have inadvertently led to ethnic profiling.

It is unclear why DUO selected the criteria it used to assess student fraud risk.

What it means

The case underscores the urgent need for ethical considerations and accountability in the deployment of algorithms within public services.ย 

The government's acknowledgement and offer of recompense is seen to have marked a shift in Dutch policy regarding the use of algorithms in public administration.ย 

That said, organisations such as Amnesty International caution that significant risks remain unless comprehensive safeguards against ethnic profiling are established across government agencies.

Rule-based system

In computer science, a rule-based system is a computer system in which domain-specific knowledge is represented in the form of rules and general-purpose reasoning is used to solve problems in the domain.

Source: Wikipedia ๐Ÿ”—

System ๐Ÿค–

Operator:ย 
Developer: Dienst Uitvoering Onderwijs (DUO)
Country: Netherlands
Sector: Education
Purpose: Detect fraud
Technology: Rule-based algorithm
Issue: Bias/discrimination

Legal, regulatory ๐Ÿ‘ฉ๐Ÿผโ€โš–๏ธ

Research, advocacy ๐Ÿงฎ