What happened

The patient, who had previously undergone a kidney transplant, used ChatGPT to seek information regarding her post-transplant care, and made the decision to discontinue prescribed antibiotics dased on the chatbot's assertion that her creatinine levels were normal.

Her condition quickly worsened and she lost her transplanted kidney and had to return to dialysis. 

Why it happened

The underlying cause is the misapplication of a general-purpose AI model for specialised, critical medical decision-making, compounded by limitations in corporate and product accountability:

• AI design limitations: ChatGPT is trained to predict the most plausible next word based on vast datasets, not to provide expert, clinically verified, and patient-specific medical diagnoses or treatment plans. It "hallucinates" (produce confidently false or misleading information). For a complex, critical area like organ transplant medicine, even a small error can be fatal.

• Lack of accountability and oversight: ChatGPT's developer OpenAI deploys the tool with disclaimers warning against using it for medical advice. However, the system's ability to provide articulate, convincing, and specific-sounding responses can override user caution, especially when seeking alternatives to complex, long-term, or costly treatments. The opaque nature of the tool's training data and reasoning prevents patients from discerning accurate information from dangerous misinformation.

• Patient vulnerability: Patients dealing with chronic or complex conditions, such as post-transplant care, may be desperate, seeking second opinions, or looking for ways to avoid medications with side effects, making them highly susceptible to compelling, easy-to-access, and seemingly personalized advice, even from non-human sources.