What happened

An "adversarial" audit of RisCanvi conducted by algorithmic auditing company Eticas concluded that the algorithm suffered from “substantial deficiencies” in the reliability of its decisions by failing to standardise outcomes and for being potentially biased against specific types of people, particularly those with challenging backgrounds.

Eticas also found that the system had not adhered to Spain’s regulations on the use of automated decision-making systems, which have required audits since 2016, and that it had not undergone an Algorithmic Impact Assessment or rigorous testing.

Furthermore, the audit concluded that inmates, who are the people most affected by the system, were unaware of its existence, while staff using it lacked proper training and understanding of how it worked.

Why it happened

RisCanvi's shortcomings appear to stem from the poor design and training of the system, and from inadequate oversight and poor transparency.

It is unclear whether its transparency and accountability shortcomings constituted a deliberate attempt to conceal the system's inadequacies.