ChatGPT incorrectly diagnoses most pediatric cases

Occurred: January 2024

Can you improve this page?
Share your insights with us

ChatGPT incorrectly diagnosed over 8 in 10 pediatric case studies, according to a new research study. The findings raised questions about the chatbot's suitability as a diagnostic tool for complex conditions. 

Researchers at Cohen Children's Medical Center, USA, pasted the text of 100 pediatric case challenges published in JAMA Pediatrics and the New England Journal of Medicine between 2013 and 2023 into ChatGPT, with two qualified physician-researchers scoring the AI-generated answers as correct, incorrect, or 'did not fully capture the diagnosis.' 

ChatGPT got the right answer in 17 of the 100 cases, was wrong in 72 cases, and failed to fully capture the diagnosis of the remaining 11 cases. Among the 83 wrong diagnoses, 47 (57 percent) were in the same organ system. 

The findings suggest ChatGPT should not currently be used to assess complex pediatric cases, and that 'more selective training' is required to make it more accurate and reliable. The authors also suggest that chatbots could improve with more real-time access to medical data.


Operator: Cohen Children’s Medical Center  
Developer: OpenAI
Country: USA
Sector: Health
Purpose: Generate text
Technology: Chatbot; NLP/text analysis; Neural network; Deep learning; Machine learning; Reinforcement learning
Issue: Accuracy/reliability
Transparency: Governance