Rotterdam welfare fraud risk algorithm

The Municipality of Rotterdam used a machine learning algorithm to detect welfare fraud by flagging individuals with high-risk profiles for investigation, resulting in allegations of poor governance and discrimination.

Developed by Accenture and introduced in 2017, the algorithm assigns risk scores based on citizens' gender identity, age, children, language skills, and other criteria.

Governance

A critical external review (pdf) of the system commissioned by the Dutch government had led to the system's suspension in 2021. The  government auditor discovered 'insufficient coordination' between the developers of the algorithms and people using them, potentially resulting in poor ethical decision-making.

The auditors also took issue with the Municipality's failure to assess whether the algorithms were better than the human systems they replaced.

Discrimination

A 2023 Lighthouse Reports/WIRED investigation revealed that individual criteria such as being female, young, having children or having failed a Dutch language test increased a citizen’s risk profile and made them more likely to be flagged for investigation. 

Tests revealed that increases of any one of these variables could also unintentionally increase the likelihood of discrimination, particularly against vulnerable groups. 

Transparency

The Municipality of Rotterdam had jealously guarded its welfare fraud algorithm, blocking attempts by journalists, researchers and others to gain access to information and data showing how it worked, including its data, code, and model. And, to its credit, the city opted to engage with the Lighthouse Reports/WIRED's investigation - the only government entity to do so across Europe.

That said, per The Markup, 'Rotterdam was the only city to provide the model file and other technical documentation, and that was after a year of negotiations. In the end, it still took a stroke of luck for the reporters to get the training data: City officials unknowingly included it in the source code of histograms they sent to reporters.'

Investigations, assessments, audits

• Rotterdam Risk Score Calculator

• WIRED (2023). This Algorithm Could Ruin Your Life

• WIRED (2023). Inside a Misfiring Government Data Machine

• Lighthouse Reports (2023). Suspicion Machines

• Lighthouse Reports (2022). The Algorithm Addiction. Mass profiling system SyRI resurfaces in the Netherlands despite ban

• Rekenkamer Rotterdam (2021). Gekleurde Technologie (pdf)