What happened

Internal assessments revealed by a Guardian investigation revealed that "statistically significant outcome disparities" in the AI system's recommendations for fraud investigations. 

Designed to identify fraudulent claims for Universal Credit, the system disproportionately flags individuals from certain demographic groups.

The incident sparked outrage and distrust in the UK government's approach to AI, with campaigners criticising the Department of Work and Pensions' (DWP) "hurt first, fix later" approach and calling for increased transparency and accountability in government's use of the technology. 

Why it happened

The DWP failed to conduct comprehensive fairness analyses for all protected characteristics. No evaluations have been made regarding potential biases related to race, sex, sexual orientation, religion, pregnancy and gender reassignment status. 

Critics argue that this lack of thorough assessment undermines the system's fairness and inclusivity.