UnitedHealth uses AI model with 90% error rate to deny care, lawsuit alleges

https://arstechnica.com/health/2023/11/ai-with-90-error-rate-forces-elderly-out-of-rehab-nursing-homes-suit-claims/

by kurthr • 3 years ago

163 57 3 years ago

arstechnica.com

jghn2• 3 years ago

So glad to be a United customer!

kelseyfrog2• 3 years ago

What's an acceptable error rate for such a model? What's the human baseline?

I ask these questions[1][2] for nearly every model I'm requested to develop. It usually throws off PMs who aren't used to any acceptable errors.

It's always fun to get a request that's several orders of magnitude better than human performance. Sometimes it's actually possible to deliver, but often it shifts the conversation into feasibility and expectations.

Anyone developing a model like this will have errors and definitionally makes systemic. Regardless of performance it's important to address that.

1. In this order 2. Sometimes I study human baselines but ask before revealing because it prevents anchoring.

tekacs2• 3 years ago

Absolutely -- an academic environment I was in used to have a lot of government callers come by, asking for predictive power to solve any number of humanitarian crises, terrorism, crime and so on.

We collectively used to ask them every time,

1. What data they had to work with.

2. What was the human baseline that they were currently executing with.

3. What performance would make a difference to this process.

... and as you say, the answers we would get would tend to sound something like.

1. Almost nothing.

2. Unclear, un-measured.

3. Something well beyond incredible, well-intentioned humans with incredible data.

In that case the lesson was often to simply not try, but it does sometimes feel like other areas of government like E2E-breaking legislation, in that well-intentioned folks will try and try and try again and eventually get someone to fund and execute on the project, feasibility notwithstanding.

In this instance with United... the data that we're getting via Ars and this case is so unclear and one-sided that it's hard to know how well things actually work... it'd be fun to consign them (for now) to "don't even try" if we had the info to do so, but it feels like it's a matter of time either way and so... I think the ideal with most of these cases is to eventually ship something that's actually high quality.

raverbashing• 3 years ago

What's an acceptable error rate for such a model? What's the human baseline?

Funny, I though it was socialized medicine that had "death panels" and such /s

kelnos3• 3 years ago

If anything, the "death panels" on private insurers (whatever form they might actually take) are likely much worse than a government-run "death panel".

kelnos3• 3 years ago

What's an acceptable error rate for such a model?

Ideally, 0%. We're talking about people's lives and health here.

But, right, realistically:

What's the human baseline?

Good question, but I would hope it's much better than a 90% error rate. And if it's not, any sane company in any sane industry would figure out why things are that bad, and fix it. Of course, insurance companies are the scum of the earth, and have a huge incentive to get things wrong in this way.

kelseyfrog2• 3 years ago

Why is there a different standard between the model and a human?

From the perspective of the business, whether a model or human causes the error is immaterial.

Any sane company in any sane industry would figure out why things are that bad, and fix it.

The function of a thing is its consequences. So we have to conclude that the reason the model had a 90% error rate was to have a 90% error rate. It's aligned with business incentives to be so.

tekacs2• 3 years ago

Though few patients appeal coverage denials generally, when UnitedHealth members appeal denials based on nH Predict estimates—through internal appeals processes or through the federal Administrative Law Judge proceedings—over 90 percent of the denials are reversed, the lawsuit claims. This makes it obvious that the algorithm is wrongly denying coverage, it argues.

The '90% error rate' in the title sounded like a pretty serious twisting of some numbers so I scanned the article for where it's quoted from ^, which... makes a mockery of the title.

In case it's not obvious and to save parsing that, 'few people appeal' (which makes sense given the internal appeals process or Administrative Law Judge proceedings). Then, /of those cases/ 90% are overturned.

- If 50 / 10,000 cases and 45 were overturned or 500 / 100,000 cases appealed and 450 were overturned... would all meet this '90% error' mark.

- Similarly, 'overturned != error', because it may be as simple as a policy on United's side to not fight super hard and allow overturns (for any number of reasons, including just saving costs on the proceedings, it needn't be altruistic).

Did Ars used to be better than this, or were they always like this on some issues?

The /business/ side of medical insurance isn't really a topic that I know enough to weigh in on, but hopefully people can read the above sentence and parse what it's actually saying for themselves before reading the article.

Irrespective of how I/we feel about the outcomes, this headline is awful.

michaelmrose2• 3 years ago

Of the only claims that have been adjudicated 90% were found to have been erroneous.

Without analysis there is no particular reason to believe that other denials were more justified.

Few doctors on average make nonsensical requests for patient care while insurers issue spurious denials as a matter of course.

You dont need to know the total number of denials instead you need a neutral party to analyze a random sample of denials.

We could do this as matter of course and tie failing an adequate benchmark to ruinous penalties to company and CEO.

hayksaakian• 3 years ago

I think the OP is suggesting that hypothetically speaking, people would only go through the hassle of appealing if they were pretty sure they would win to begin with

miki123211• 3 years ago

For a model whose only output is allow/deny, 90% error rate is pretty good, you can just flip the answer and then you have a 10% error rate.

If you know the first thing about AI (or even statistics in general), that headline sounds very suspicious.

michaeljx• 3 years ago

What kind of an educated scientific person, working in health insurance and dealing with sick people, develops anything that makes this possible?

lispisok2• 3 years ago

Regular people will do horrible things to keep their paycheck coming. It's an underrated phenomena of humanity.

HellDunkel2• 3 years ago

I strongly disagree with how this is presented. Doing half-assed or even passionate work for greedy insurance companies is not a horrible thing to do. Not thinking of how these companies can be reigned in and acting accordingly is far worse. We need to stop offloading moral responsibility from the people in charge to the regulae folks.

twiceaday• 3 years ago

One person doesn't do this, a company does, a hierarchy of jobs. An entity without a conscience built from entities with a conscience. Very clever.

t-writescode• 3 years ago

The banality of evil.

corethree• 3 years ago

The irony here is that the entities with a conscience are very much aware of what their doing. Very very much aware.

So there's not much difference is there?

Actually there is. The difference is in a quirk of moral evolutionary psychology. The tragedy of the commons. As an individual each of those entities contributes negligible evil to society. In aggregate that negligible evil becomes prominent.

Morality is divided up into fractional shares and sold like a collateral debt obligation. Every time you step into a car and let that thing spew green house gases into the air you are contributing to the destruction of the world. The individual is still guilty. We call corporations evil, but what we don't realize is that corporations are a mirror of humanity.

thelastgallon• 3 years ago

There is only one rule: shareholder profits

jghn2• 3 years ago

You're making a lot of assumptions here. Insurance is there to make a buck. Nothing more.

ranting-moth• 3 years ago

"Would you do X for $Y?"

Most of the time you can increase Y to get the answer "yes".

In many cases Y can even be factored out by changing it to "Would you do X so you wouldn't get fired?"

Empirical answers differ a lot from people's theoretical principles.

wahnfrieden• 3 years ago

Plenty of folks here work on bombs for killing people, surveillance instruments for govt and police to use for spying on citizens, etc - people take paychecks and they take orders

The business of insurance is to make money by denying care that people paid for

FpUser• 3 years ago

What on Earth makes you think that there are less assholes in educated part of the population?

renewiltord• 3 years ago

It's a common expression on HN and Reddit that people don't "care" about their job or their coworkers. They produce output, they get paid. Simple exchange. Code for money. So why does it surprise you that someone would see building this as just an exchange?

manicennui• 3 years ago

I worked on healthcare software for over a decade. The healthcare industry attracts the lowest quality people generally. Software engineers aren't giving up working at FAANG to go work in healthcare. A significant portion of UHG's revenue comes from billing and other healthcare software. Most of it is under their Optum brand.

RedCardRef• 3 years ago

In my previous role, I worked on something similar to this. We built a model to predict which claims(CMS 1500) would be denied by the carrier(Insurance Provider) before they were submitted and allowing the hospital staff to adjust the Diagnosis/Procedure codes and validate the combination. The client had 20% denials, our model brought that denial rate down to 10%.

We wanted to compare the performance of our model with commercial third party offerings and we enrolled into an early beta for a similar offering from ChangeHealthcare one of the big players in the healthcare industry, their model was wildly off(<50% accuracy), we thought we were missing something and went back and forth with their team, turns out they didnt know what they were doing, they just slapped AI on the API and tried to see if anyone would actually use it. That model never saw the light of day. After studying the data model and seeing the number of variables involved I would be skeptical of prediction models retaled to insurance in the healthcare industry.

jpambrun• 3 years ago

Now Change is owned by United Healthcare, so maybe the model you mentioned did live on..

catchnear4321• 3 years ago

you either die the benevolent language model, or live long enough to become the…

no, don’t like that. do not like that.

alberth2• 3 years ago

Was denying 90% not the desired outcome?

For an insurance company, this sounds like the ideal model.

lusus_naturae• 3 years ago

Stupid question—if an insurance provider keeps denying claims which could keep elderly patients healthy, and those patients then die, isn't the insurance provider just cutting off its own source of income?

But I agree that it's about incentives, and insurance carriers are incorrectly incentivized for the service they provide vs. what they're expected to provide.

alberth2• 3 years ago

The old adage goes: “those who don’t use insurance, pay too much. And those who use insurance, pay too little”.

It becomes a math problem for insurance companies to keep that equation positive. And since claims far exceed the premiums any single member pays, the easiest way to stay positive is to deny claims. Even if that sacrifices losing a few premiums.

bigfudge• 3 years ago

die, isn't the insurance provider just cutting off its own source of income?

No because that person will have paid right up the the point they become expensive.

Most people incur the vast majority of their health costs in the final year or even months of life. This is why insurance based medicine will always be problematic - there’s just too much incentive to cheap out on people when they are weakest and most powerless.

heavyset_go3• 3 years ago

Not really, the highest medical costs are incurred in old age. Insurance companies would have to make less costly payouts if they didn't have to cover older people.

gorjusborg2• 3 years ago

Another angle is premiums paid vs claims awarded. If an elderly patient has paid in 50k over a lifetime and is making a 100k claim, now is an okay time to lose that customer.

ahoka• 3 years ago

Less known for it, but EU GDPR allows for asking for a non automated decision.

causi• 3 years ago

We need regulations about what computers should and should not be allowed to do. Computers should not be allowed to make healthcare decisions. Computers should not be allowed to make legal threats or actions.

heavyset_go3• 3 years ago

Bureaucracies and authorities love black box models because they can always point to them, shirk their own responsibility for decision making and say they're only following orders and that computers aren't biased.

It's just a coincidence that the model they're paying for keeps making decisions that are beneficial to their own interests and reflect their own biases.

michaelteter• 3 years ago

I predict this that AI will be (intentionally) used as a scapegoat for plausible deniability in some industries. "We just followed our AI model." "We thought it was doing the job." "It's too complex to understand."

Obviously once you know or believe that your AI model is wrong, and you are doing a disservice to your clients/patients... and especially once you begin discussing this internally, then you are aware of the problem and should not (get caught) sweeping it under the rug.

iforgotpassword• 3 years ago

You don't need AI for this, software is already treated as something God given by people. Government somewhere wrongly denying foodstamps to a huge amount of people? Software error, nothing anyone could possibly be responsible for. Getting hacked and all your customers data stolen (because you don't follow basic security best practices)? Well that just happens, just like a taiphoon or earthquake, nothing anyone could've done about this. But bonus points for demanding the government do something about those hacker gangs.

blitzar2• 3 years ago

"Computer says no" is a catchphrase first used in the British sketch comedy television programme Little Britain in 2004.

https://www.youtube.com/watch?v=0n_Ty_72Qds

SigmundA• 3 years ago

CEO of Brawndo: "The stock has dropped to zero and the computer did that auto-layoff thing to everybody!"

gorjusborg2• 3 years ago

There needs to be a legal respponsibility of users of AI. AI isn't responsible, those acting with the aid of AI are.

zzzcsgo2• 3 years ago

It's call UnitedHealthcare

zzzcsgo2• 3 years ago

I guess the stupid HN title rule limited the characters

rchaud• 3 years ago

Though few patients appeal coverage denials generally, when UnitedHealth members appeal denials based on nH Predict estimates—through internal appeals processes or through the federal Administrative Law Judge proceedings—over 90 percent of the denials are reversed, the lawsuit claims.

That's the power of system defaults. Make "No" the default response, regardless of the validity of the request, and there's a good chance patients that are trying to manage the logistics of appointments, prescriptions and treatments will simply give up.

AI will be used to hold up these existing practices; when glittering visions of lowered healthcare costs are trotted out by the op-ed writers and techno-optimists, this is really how that sausage is being made.

The-Old-Hacker• 3 years ago

66.5% of US bankruptcies are caused directly by medical expenses.

https://www.retireguide.com/retirement-planning/risks/medica...

UnitedHealth uses AI model with 90% error rate to deny care, lawsuit alleges

Related Stories