IBM AI oncology tool suggests unsafe treatments
IBM AI oncology tool suggests unsafe treatments
Occurred: February 2017
Page published: August 2024 | Page last updated: December 2025
Touted as a revolutionary AI tool for cancer treatment, IBM’s Oncology Expert Advisor (OEA) failed to meet its clinical promises due to technical limitations and reliance on hypothetical data, ultimately leading to the dissolution of high-profile partnerships and a multi-billion dollar scale-back.
IBM Watson for Oncology, developed in collaboration with Memorial Sloan Kettering Cancer Center and launched in the early 2010s, promised to revolutionise cancer care by providing treatment recommendations based on patient data and medical literature.
However, reports revealed that the system frequently suggested unsafe or incorrect treatments, relied on limited datasets, and struggled to handle real-world clinical complexity. In one instance, it suggested a drug that could cause severe bleeding for a patient already experiencing heavy hemorrhaging.
Hospitals in the U.S. and India that implemented the system reported minimal practical benefit, and in some cases, physicians had to override or ignore its recommendations, highlighting a gap between marketing claims and actual performance.
The failures potentially endangered patients, misallocated hospital resources, and undermined trust in AI in medicine.
A 2017 audit by the University of Texas System found that the project had spent USD 62 million but had failed to deliver a functional product. The audit also raised concerns about the project's governance, management, and the lack of clear goals and objectives.
The controversy surrounding the project led to a number of high-profile departures, including the resignation of the head of MD Anderson's IT department and the departure of several top executives from IBM Watson Health.
The project was ultimately shut down in 2018, with MD Anderson writing off the entire investment.
The failure of the project was seen as a major setback for IBM Watson Health, which had been touted as a leader in the field of AI-powered healthcare.
The failure was rooted in a fundamental disconnect between marketing hype and engineering reality.
Synthetic data bias: Instead of being trained on vast amounts of real-world patient data, Watson was primarily trained on a small set of "synthetic" or hypothetical cases curated by a few doctors at Memorial Sloan Kettering Cancer Center (MSKCC). This resulted in a "MSK-in-a-box" that reflected the specific biases and preferences of a single institution rather than a global standard of care.
NLP limitations: Watson struggled with the "messiness" of real medicine. It could not reliably parse unstructured doctor notes or distinguish between acronyms (e.g., "ALL" meaning Acute Lymphoblastic Leukemia vs. "all" as a common word).
Transparency and accountability: IBM maintained a "black-box" approach. The system provided recommendations without explaining the underlying reasoning or the specific literature used to reach a conclusion. This lack of transparency made it impossible for clinicians to trust or audit the machine's logic.
Corporate hype: IBM’s marketing department outpaced its R&D. While advertisements showed Watson "curing" cancer, the underlying technology remained essentially a sophisticated search engine for clinical guidelines rather than a truly predictive intelligence.
For patients, the system’s failures appear to have resulted in poor recommendations and the deterioration of their health.
Clinicians faced increased cognitive burden and skepticism toward AI tools.
For the healthcare industry, the incident underlined the dangers of prematurely deploying AI in high-stakes environments and raised broader societal concerns about trust, oversight, and the responsible communication of AI capabilities.
Oncology Expert Advisor
Developer: IBM; PwC
Country: India; USA
Sector: Health
Purpose: Diagnose cancer; Recommend treatments
Technology: Machine learning
Issue: Accountability; Accuracy/reliability; Transparency
AIAAIC Repository ID: AIAAIC0106