Epic sepsis prediction model

Incidents and issues 🔥

• A June 2021 study published in JAMA Internal Medicine by University of Michigan researchers analysing a retrospective sample of over 27,000 adult Michigan Medicine patients concluded the algorithm is only correct 63 percent of the time, and raises many false alarms. Part of the problem, Stat News reported, is that the algorithm was trained to flag when doctors would submit bills for sepsis treatment, which doesn’t always line up with patients’ first signs of symptoms. In response, Epic pointed to previous research that found the model can accurately predict sepsis, and argued customers have 'complete transparency' into the model. In an accompanying editorial, medical researchers argue the findings highlight the need for the external validation of proprietary healthcare prediction models before clinical use.

• An August 2023 Atrium Health research study found that Epic’s product was more accurate at the highest scoring thresholds, when it was most confident that a patient had sepsis. However, it also found that those thresholds were only reached after clinicians had already taken steps to treat the condition. 

• In February 2022, Stat published the findings of a research study conducted with the Massachusetts Institute of Technology that small shifts in data fed into well-known health care algorithms, including the Epic Sepsis Model can cause their accuracy to degrade over time. Instead of transforming care, the study found, the algorithms are unable to keep pace with fast-moving clinical conditions, potentially resulting in mis-diagnoses and raising the prospect AI could do more harm than good.

• In October 2022, Epic Systems said it had overhauled its sepsis prediction model to improve its accuracy and make its alerts more meaningful to clinicians. It also said it had changed its definition of sepsis to match the international consensus definition.