What happened

The incident, which was reported to the Victorian Office of the Information Commissioner (OVIC) in December 2023, revealed that the worker included personal details about the child and their family, which ChatGPT mishandled by downplaying risks associated with the child's living situation with parents charged with sexual offenses.

The investigation found multiple signs of ChatGPT's involvement in the report, including inappropriate language and sentence structures inconsistent with child protection protocols.

Notably, the report described a child's doll, allegedly used for inappropriate purposes by the father, as an "age-appropriate toy," which raised alarms about the potential misrepresentation of risks to the child.

Why it happened

The use of ChatGPT by Victoria's Department of Families, Fairness and Housing (DFFH) to draft child protection-related documents was systemic and not policied. 

OVIC found that the worker may have used ChatGPT in 100 cases in drafting child protection-related documents, and that nearly 900 employees - almost 13 percent of the workforce - had accessed ChatGPT across the department from July to December 2023.

What it means

OVIC mandated that the DFFH block access to generative AI tools like ChatGPT, with the department being given until November 5, 2024, to implement the restrictions and ensure compliance with privacy regulations. 

While the outcome of the child's case remained unchanged, the incident highlights significant ethical and safety issues surrounding AI use in sensitive areas like child protection. 

OVIC Commissioner Sean Morrison emphasised that AI systems can make serious errors that may lead to harmful consequences if relied upon without verification.