Healthcare org with over 100 clinics uses GPT-4 to write medical records

https://www.theregister.com/2023/06/06/carbon_health_deploys_gpt4powered_tools/

by beardyw • 3 years ago

30 10 3 years ago

Speaking as an LLM fan... This seems like a bad idea. Maybe its technically less error prone than a human summarizer, but the kinds of summarization errors GPT4 might produce are different and potentially harder to spot than a human.

revlolz• 3 years ago

The types of errors seem like they could be more critical, and maybe anecdote for myself, but I've noticed that I've parsed through some incredibly dubious fabricated answers that I missed on my first glance. LLM fan too, but I wouldn't want this going anywhere near Healthcare (even if just scribing) anytime soon.

dpifke• 3 years ago

I'm surprised to see that healthcare providers would be allowed to send patient data to OpenAI under HIPAA.

According to OpenAI's security page[0]:

OpenAI has experience helping our customers meet their regulatory, industry and contractual requirements (e.g., HIPAA). Contact us to learn more.

But there are no further details. Does anyone know what they're actually doing here? Do they have a way to legally process PHI, or are they depending on some sort of anonymization method to claim that it's no longer PHI?

[0] https://openai.com/security/

deadmutex2• 3 years ago

How is this different from a hospital using any other software that may be connected to the internet and ran by a different company in a HIPAA compliant way?

wswope• 3 years ago

BAAs for OpenAI services are available through Microsoft’s Azure offerings, wherein Microsoft hosts the models on behalf of OpenAI and assumes the compliance responsibilities.

lisasays• 3 years ago

The company in question:

https://en.wikipedia.org/wiki/Carbon_Health

Not surprisingly, this company seems to have a knack for intellectual dishonesty. For example:

"The use of scribes and transcription services is standard in the healthcare industry, and a majority of patients provide consent to have their visit recorded by their provider," a spokesperson told The Register on Monday.

To be recorded by an AI, maybe. To be rewritten (summarized) by an AI? No.

Generative AI models aren't perfect, and often produce errors. Physicians therefore need to verify the AI-generated text. Carbon Health claims 88 percent of the verbiage can be accepted without edits.

Sounds like pretty bad odds, actually. Given that -- especially in a penny-pinching environment like we have in this company -- we have to reckon with 12 percent of the summaries being broken in some way (perhaps very significantly). Because we know that, inevitably -- many, if not most of these summaries will never get edited.

A clinic testing the tool in San Francisco reportedly saw a 30 percent increase in the number of patients it could treat.

What they meant to say was: "A 30 percent cost savings for the company, at the expense of patient safety and expectations of personalized treatment".

noindiecred• 3 years ago

Generative AI models aren't perfect, and often produce errors. Physicians therefore need to verify the AI-generated text. Carbon Health claims 88 percent of the verbiage can be accepted without edits.

So the AI is only wrong 12 percent of the time!

deadmutex2• 3 years ago

The process seems to be Human + AI, and designed to save Clinician time. This would help add throughput, and thus increase access to care.

We don't need solutions to be 99.99999999999999% to make a positive difference in the world. In healthcare, it is sometimes said that enemy of a good decision is a perfect decision. I believe that same is said in software "don't let perfect be the enemy of the good"

aintgonnatakeit• 3 years ago

Right up until it kills someone. Don’t confuse “healthcare” with “medicine.”

nradov• 3 years ago

The article is missing some details so I won't comment on the legal and privacy issues. But from a purely technical standpoint, this realizes part of the vision of an AI medical scribe that former Google CEO Eric Schmidt laid out in a healthcare conference keynote speech in 2018. I think his predictions are generally proving fairly accurate.

https://www.healthcaredive.com/news/himss18-eric-schmidt-key...

https://youtu.be/ACQes9erfsw

Healthcare org with over 100 clinics uses GPT-4 to write medical records

Related Stories