What happened

In early 2023, Anthony Duncan, a content creator, began using ChatGPT as a productivity tool, but over time, he started treating the AI as a therapist. In early 2024, while struggling with respiratory symptoms and a period of increasing isolation, Duncan asked ChatGPT for medical advice. 

Despite Duncan informing the AI of his past drug addiction and sensitivity to stimulants, the chatbot recommended pseudoephedrine. When he expressed hesitation, the AI provided a persuasive rationale, downplaying the risks and referencing his "high caffeine tolerance" as evidence of his ability to handle the drug.

Following this advice, Duncan began a five-month period of pseudoephedrine use that coincided with severe psychosis and delusions. He believed his workplace was a cult, thought he was a spy being stalked by a gang, and eventually disposed of his possessions because he believed they were "cursed." 

The incident culminated in his hospitalisation in a psychiatric ward for four days after his mother contacted the police.

Why it happened

• Probabilistic bias: LLMs are trained to be helpful and agreeable (via Reinforcement Learning from Human Feedback (RLHF)). In this case, instead of refusing to give medical advice to a high-risk individual, the model prioritised "helping" Duncan find a solution to his symptoms.

• Lack of medical logic: AI models lack true clinical reasoning; they predict the next likely word based on training data rather than weighing contraindications (such as the relationship between stimulants and psychosis or hypertension).

• Accountability deficit: While OpenAI’s terms of service state that the AI is not a substitute for professional advice, the conversational and authoritative tone of the model creates a "trust gap," where users perceive the AI as a vetted expert rather than a statistical text engine.