Netherlands SyRI welfare fraud detection automation

SyRI (or 'System Risk Indication') is a risk classification system developed and operated by the Netherlands government to detect and predict social security, tax and employment fraud.

Deployed by the Department of Social Affairs and Employment, the system used data about employment, fines, penalties, taxes, property, housing, education, retirement, debts, benefits, allowances, subsidies, permits and exemptions, amongst others, to determine whether a welfare claimant should be investigated.

Risks and harms 🛑

Discrimination

In February 2020, SyRI was found to be in breach of human rights law by a Dutch court, which ordered an immediate stop to its use. The government had been taken to court (pdf) by a number of civil rights organisations and two citizens. 

The court ruled that SyRI violated article 8 of the European Convention on Human Rights (ECHR), which protects the right to respect for private and family life, and that by primarily targeting poor neighbourhoods, the system may have discriminated against people on on the basis of socioeconomic or migrant status. 

Privacy

The court also ruled that the legislation which permitted SyRI contained insufficient safeguards against privacy intrusions, and that the 'fair balance' between its objectives and the violation of privacy that its use entailed, meant the legislation was unlawful. 

Scope creep

Despite the Dutch government's decision not to appeal the ruling and stop using SyRI, a Lighthouse Reports investigation discovered that it had quietly continued to deploy an adapted SyRI in some of the country’s most vulnerable neighbourhoods.

The finding led to a joint investigation by Lighthouse Reports and WIRED that found that a machine learning algorithm used by the Municipality of Rotterdam to detect welfare fraud discriminates against welfare claimants based on ethnicity, age, gender, and parenthood.

Legal, regulatory 👩🏼‍⚖️

• Court of the Hague (2020). NJCM v State of the Netherlands - Judgement

• The Office of the High Commissioner for Human Rights (2019). Brief by the United Nations Special Rapporteur

• Deikwijs Advocaten/NJCM PILP (2019). Subpoena vs State of the Netherlands (pdf)

Research, advocacy 🧮

• Platform Bescherming Burgerrechten. Bij Voorbat Verdacht

• de Bruijn H., Warnier M., Janssen M. (2022). The perils and pitfalls of explainable AI: Strategies for explaining algorithmic decision-making

• Rachovitsa A., Johann N. (2022). The Human Rights Implications of the Use of AI in the Digital Welfare State: Lessons Learned from the Dutch SyRI Case

• van Bekkum M., Borgesius F.Z. (2021). Digital welfare fraud detection and the Dutch SyRI judgment

• Privacy International (2020). The SyRI case: A landmark ruling for benefits claimants around the world

• Human Rights Watch (2019). Welfare surveillance on trial in the Netherlands

• Center for Human Rights and Global Justice (2019). Profiling the poor in the Dutch welfare state

• Digital Freedom Fund. NJCM, Platform Bescherming Burgerrechten and others v. The Netherlands (the SyRI case) (pdf)

• The Public Interest Litigation Project (PILP) (2015). Profiling and SyRI

Investigations, assessments, audits 🧐

• WIRED (2023). This Algorithm Could Ruin Your Life

• Lighthouse Reports (2023). Suspicion Machines

• Lighthouse Reports (2022). The Algorithm Addiction. Mass profiling system SyRI resurfaces in the Netherlands despite ban

• Rekenkamer Rotterdam (2021). Gekleurde Technologie (pdf)

News, commentary, analysis 🗞️

• https://techcrunch.com/2020/02/06/blackbox-welfare-fraud-detection-system-breaches-human-rights-dutch-court-rules/

• https://www.theguardian.com/technology/2020/feb/05/welfare-surveillance-system-violates-human-rights-dutch-court-rules

• https://www.irishtimes.com/news/world/europe/dutch-court-halts-programme-where-algorithms-hunt-for-welfare-fraudsters-1.4191784

• https://www.economist.com/technology-quarterly/2020/06/11/humans-will-add-to-ais-limitations

• https://diginomica.com/un-report-our-algorithmic-world-creating-social-welfare-dystopia

• https://algorithmwatch.org/en/story/syri-netherlands-algorithm/

• https://algorithmwatch.org/en/high-risk-citizens/