What happened

In an eight-month study across local councils in England and Scotland, Ada Lovelace Institute researchers found that AI-powered transcription tools deployed in social work settings in 58 councils across the UK regularly generate “hallucinations” (fabricated or incorrect text) when transcribing and summarising conversations between social workers and clients.

One social worker reported that an AI tool falsely recorded suicidal ideation that the client never actually expressed. Other errors included nonsensical words or phrases in place of actual spoken content, such as “fishfingers" for "parents flighting".

If left uncorrected, these AI-generated transcripts could influence future decision-making about care plans, safeguarding, or risk assessments, and unduly harm patients and their families. 

Why it happened

The incident stems from the systemic prioritisation of efficiency over accuracy by the local councils, combined with technical limitations of Large Language Models (LLMs).

LLMs are designed to predict the next word in a sequence, not to verify facts. In high-stakes environments like social work, the AI often "fills in the gaps" with stereotypes or formal, "academic" language that misrepresents the person-centered nature of the actual conversation.

Many local authorities implemented these tools without rigorous evaluation of their impact on social care service users, focusing instead on Return on Investment and efficiency in terms of increasing the number of home visits social workers could perform per day.

Some practitioners received as little as one hour of training. This led to inconsistent review practices, with some workers spending only minutes proofreading AI outputs before finalising legal documents.