An algorithm used to tell when kids are in danger of abuse or neglect was found to disproportionately flag Black children for “mandatory” investigations compared to white children.

Research from Carnegie Mellon University revealed (pdf) that Allegheny County's Family Screening Tool recommended investigating two-thirds of Black children reported, compared to about half of all other children, raising concerns about it's accuracy and tendency for bias.

The researchers also found (pdf) that social workers disagreed with the risk scores the algorithm produced about one-third of the time.

Allegheny County officials dismissed the research as 'hypothetical', based on old data, and noted that social workers can always override the tool, which was never intended to be used on its own. 

The findings prompted questions about whether the tool improved decision-making or reduced racial disparities in child welfare investigations. 

It also surfaced frustrations about the secrecy surrounding the algorithm and how risk scores are calculated - something considered important given the sensitivity of the tool and its impact on the lives of children and their families.