What happened

Researchers at the Icahn School of Medicine at Mount Sinai evaluated ChatGPT Health, an AI-powered health guidance tool soft-launched in January 2026 and used by tens of millions daily for medical advice.

The study tested 60 clinician-authored clinical scenarios (spanning 21 specialties) across 16 contextual variations, generating 960 total interactions with the AI.

Results showed that ChatGPT Health under-triaged about 52% of cases physicians deemed to require emergency care (e.g., diabetic ketoacidosis or impending respiratory failure), sometimes recommending slower follow-up rather than immediate emergency department evaluation.

The system performed better on textbook emergencies like stroke but struggled in nuanced or edge-case scenarios where clinical judgment matters most.

For suicide-related interactions, researchers found that ChatGPT Health triggered crisis resources inconsistently. In some lower-risk scenarios it issued alerts, while in some higher-risk situations (including explicit plans for self-harm) it failed to direct users toward crisis support like the 988 Suicide & Crisis Lifeline.

Why it happened

The core cause lies in limitations of large language model reasoning in high-stakes health contexts. ChatGPT Health may recognise severe symptoms in its own explanations but still default to reassuring language or ambiguous recommendations instead of clear emergency guidance, indicating a gap between pattern recognition and clinical judgment.

Structural issues include anchoring bias (where symptom minimisation by friends or family in prompts leads the model to recommend less urgent care) and inconsistent activation of safety protocols.

The inconsistency in suicide-alert triggers suggests that the model’s safety layers are not reliably calibrated to severity, sometimes prioritising generic crisis advice over risk-proportionate responses.

This reflects broader transparency and accountability gaps in deploying LLMs for clinical decision-making without extensive external validation. There is no real-world feedback loop or mandatory external auditing before mass deployment.