A popular algorithm to predict sepsis misses most cases and sends false alarms

https://www.statnews.com/2021/06/21/epic-sepsis-prediction-tool/

by hhs • 5 years ago

120 38 5 years ago

statnews.com

bimboler38• 5 years ago

Paywalled

ameetgaitonde• 5 years ago

Here's an alternative article on the same topic: https://labblog.uofmhealth.org/lab-report/popular-sepsis-pre...

lstamour• 5 years ago

Or https://www.wired.com/story/algorithm-predicts-deadly-infect...

mooneater• 5 years ago

Podcast on this was good: https://edhub.ama-assn.org/jn-learning/audio-player/18620183

MattGaiser2• 5 years ago

The problem of looking at supposedly representative (but not actually) proxies for actual information is widespread in society, whether it be in hiring or medicine.

Here they just replicated all the same issues the current model of diagnosis faced.

tgv• 5 years ago

Looks like they picked data that didn't require any work on their part.

MattGaiser2• 5 years ago

I work for an AI company, so I can easily see that.

raducu2• 5 years ago

My wife recently had a burst apendice with perithonitis and a lot of puss in her abdomen and she had no fever.

After the surgery my parents(both doctors) told me to watch for fever as a sign of reinfection. I had to remind them there was no fever in the first place and that was likely a bad proxy for infection in her case.

tuukkah7• 5 years ago

As CRP testing is easy and cheap, why do we still cling to heuristics such as measuring fever instead of measuring the infection?

skmurphy• 5 years ago

Original article https://jamanetwork.com/journals/jamainternalmedicine/fullar...

Not that "Epic" is the name of an Electronic Health Record system that embeds this proprietary algorithm for detecting sepsis.

Key Points

Question How accurately does the Epic Sepsis Model, a proprietary sepsis prediction model implemented at hundreds of US hospitals, predict the onset of sepsis?

Findings In this cohort study of 27 697 patients undergoing 38 455 hospitalizations, sepsis occurred in 7% of the hosptalizations. The Epic Sepsis Model predicted the onset of sepsis with an area under the curve of 0.63, which is substantially worse than the performance reported by its developer.

Meaning This study suggests that the Epic Sepsis Model poorly predicts sepsis; its widespread adoption despite poor performance raises fundamental concerns about sepsis management on a national level.

Results We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.

Conclusions and Relevance This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.

farseer• 5 years ago

So that's euphemism for the algorithm doesn't work.

tuukkah7• 5 years ago

This case is worse than just not working: the algorithm generates alert fatigue and misleads doctors who would do a better job and come to better decisions without the algorithm. In the hospital context, a fair characterisation might be that the algorithm causes deaths.

modmans2nd• 5 years ago

Prior to this algorithm most Epic customers had an alert for providers to check for Sepsis in a patient. In most cases the Epic tool is an improvement in over alerting.

cratermoon2• 5 years ago

Just like the days when every program would pop up warning after warning asking, "are you sure?". After a while users started to treat them as annoyances interfering with the task and automatically close them. They became worse than useless because among the noise there would sometimes be serious events that would lead to corruption or loss of data.

stuartbman2• 5 years ago

The issue is that sepsis is very poorly defined, difficult to distinguish from other conditions, and very serious if missed. This means that I get automated alerts for patients all day querying sepsis, but probably only 5% (maybe fewer) have any signs of infection clinically. This reflects clinical practice where we overtreat initially and then row back treatment if things improve.

I can go for a run on a hot day- I'll be pyrexic, tachypnoeic, and tachycardic- enough to trigger a sepsis alert. Whereas an elderly patient who is a bit more confused than normal may have a very severe raging infection with few changes in their markers.

mjburgess• 5 years ago

It seems like this should be a sepsis-risk calculation then, based on extrinsic factors (age, admittance reason, etc.); rather than intrinsic factors (hr, bp..).

The problem here then, i'd guess, is how many of those extrinsic factors are unmeasurable in practice; and how little data we'd have on them.

joshgel4• 5 years ago

Basically everyone admitted to the hospital is at some level or risk from sepsis which is why the proxy markers become important.

stuartbman2• 5 years ago

Sure, and there's a lot of work in this area to predict sepsis in specific subgroups, but as with a lot of AI I think we're a long, long way off a 'general purpose' sepsis detector. For various reasons sepsis just presents differently in different patient groups so one person's risk factor is another's protective factor

joshgel4• 5 years ago

Sepsis is pretty well defined between SIRS and qSOFA. But the definition has a subjective component which has very poor inter-rater reliability. And you’re 100% right that the definition overlaps with other very common conditions, such as pain or having just had an operation.

epmaybe• 5 years ago

I have this conversation with my attendings regularly regarding scaling back treatment. Everyone has their own comfort level with how aggressive they are doing that, and it would be interesting to see better heuristics that would allow for more appropriate de-escalation sooner (or longer, I guess)

cratermoon2• 5 years ago

Hot, breathing heard, with a fast heartbeat. For the laypeople out there. Saved you three lookups. Just call me the human memoizer.

SQueeeeeL• 5 years ago

I'm gonna call you the "being decent at communicating"izer, for using commonly understandable terms and not ego stroking about how much medical vocab you know

dr_house• 5 years ago

check out these guys: https://www.luminaremed.com ; started because of the false alarms

A popular algorithm to predict sepsis misses most cases and sends false alarms

Related Stories