Large Language Models Are Few-Shot Health Learners

by rish-b • 3 years ago

53 47 3 years ago

At this point is it fair to ask: in which spaces, which we describe with natural language, have LLM's have not been few shot learners in?

ResearchCode• 3 years ago

What is a "learner"?

simonw2• 3 years ago

I find the healthcare applications of this stuff so interesting.

On the one hand, there are SO many reasons using LLMs to help people make health decisions should be an utterly terrible idea, to the point of immorality:

- They hallucinate

- They can't do mathematical calculations

- They're incredibly good at being convincing, no matter what junk they are outputting

And yet, despite being very aware of these limitations, I've already found myself using them for medical advice (for pets so far, not yet for humans). And the advice I got seemed useful, and helped kick off additional research and useful conversations with veterinary staff.

Plenty of people have very limited access to useful medical advice.

There are plenty of medical topics which people find embarrassing, and would prefer to - at least initially - talk to a chatbot than to their own doctor.

Do the benefits outweight the risks? As with pretty much every ethical question involving LLMs, there are no obviously correct answers here.

deadmutex• 3 years ago

found myself using them for medical advice (for pets so far, not yet for humans)

Which model did you use?

simonw2• 3 years ago

GPT-4

I like running "can my dog eat avocado?" through smaller models to see what happens.

omeze• 3 years ago

I agree, Ive found combining LLMs with Google has worked well for research. Use it for all sorts if random things, usually starting with search then hopping to chatgpt or bard when I cant understand the results. Then back to search when I know what to look for again.

rvz3• 3 years ago

I have been saying this for months for deep learning in general (and now the new hype in LLMs) in high risk situations such as medical, legal and financial advice and even transportation. The only common use-case which makes sense is summarization and even then, a human expert ends up reviewing the output before post it anyway.

There are plenty of medical topics which people find embarrassing, and would prefer to - at least initially - talk to a chatbot than to their own doctor.

I don't think you would trust an AI chatbot alone to take a number of pills for any medication instead of going to a human doctor, especially when these AI models risk hallucinating terrible advice and its output is unexplainable and as transparent as a black-box. The same goes for 'full self-driving'.

I don't think one would trust these deep learning-based AI systems in very high risk situations unless they are highly transparent and can thoroughly explain themselves rather than regurgitate back what it has been trained on already.

It is like trusting a AI to pilot a Boeing 737 Max with zero human pilots on board end-to-end. No one would board a plane that has an black-box AI piloting it. (Autopilot is not the same thing)

flangola7• 3 years ago

At least for legal there is far more potential than just summarization. Harvey is already producing legal documents with error rates lower than humans.

MrPatan2• 3 years ago

People take pills from known criminals, with high risk of fentanyl OD, just for fun.

Yes, I think people would indeed take pills prescribed by AI, just make it a robot wearing a lab coat.

Also pilots! I mean, pilots kill themselves and a planeload of people more than you think. Of course people would take black box ai that works.

pstuart2• 3 years ago

I'd be happy with summarizing and aggregating of health and longevity articles/papers to have a concise digest of strategies.

Case in point, I'm a big fan of Andrew Huberman (https://www.youtube.com/@hubermanlab). He's quite prolific and his presentations pack a lot of data. Just taking all of that in would require a lot of time. Being able to have it condensed and indexed would be wonderful.

Plenty of others like him (e.g., Rhonda Patrick, Peter Attia, etc.) High quality stuff but there's literally not enough time to take all of it in.

ke88y3• 3 years ago

However useful his advice might be, Andrew Huberman isn't a doctor (of medicine).

Summarizing academic research is almost entirely unrelated to the practice of medicine. Medical diagnosis and treatment are different from more typical uses of LLMs in lots of important ways.

Baeocystin• 3 years ago

Whatever else its ills, the bot actually will pay attention to the tokens you're submitting to it to formulate its answer. That puts it well ahead of a majority of the doctors I've seen over the years.

I say this without snark- it is simply true. I should also mention that a good quarter of the medical care folks who have assisted me have gone above and beyond in exceptional ways. It is a field of extremes.

brucethemoose23• 3 years ago

Most doctors/vets I've seen recently are just massively overbooked. You wait 3 hours, then you have 4 minutes of conversation time for one (out of multiple) ailments before you’re booted out the door. Its like you're on an assembly line and the workers can't even keep up.

leobg• 3 years ago

Attention is all you need, doctors.

ke88y3• 3 years ago

they can't do mathematical calculations

Tell me you never taught service courses for pre-meds without telling me you never taught service courses for pre-meds ;)

They hallucinate, They're incredibly good at being convincing, no matter what junk they are outputting

Describes about a third of the doctors I've interacted with, tbh.

And the advice I got seemed useful, and helped kick off additional research and useful conversations with veterinary staff.

It's similar to "Dr. Google". Possible to misuse. But also, there's nothing magical about the medical guild initiation process. Lots of people are smart enough to learn and understand the bits of knowledge they need to accurately self-diagnose and understand tradeoffs of treatment options, then use a medical professional as a consultant to fill in the gaps and validate mental models.

Unfortunately, most medical professionals aren't willing to engage with patients in that mode and would rather misdiagnose than work with an educated patient. (My bil -- a medical doctor, and a fairly accomplished one at that -- has been chided for using "Dr Google" at an urgent care before.)

Do the benefits outweight the risks? As with pretty much every ethical question involving LLMs, there are no obviously correct answers here.

At the end of the day, it doesn't matter. At least in the US, you won't have access to any meaningful treatment without going through the guild anyways.

I don't think that using LLMs for medical diagnosis is a good idea, but it's important to admit when the status quo is so thoroughly hollowed out of any moral or practical justification that even terrible ideas are better than the alternative of leaving things as they are.

haldujai6• 3 years ago

Lots of people are smart enough to learn and understand the bits of knowledge they need to accurately self-diagnose and understand tradeoffs of treatment options, then use a medical professional as a consultant to fill in the gaps and validate mental models.

This is incredibly dangerous, lots of people are smart enough that they can research questions about their condition/care to discuss with their medical professional but should absolutely not be self-diagnosing. It is very reasonable to ask "I read about X what do you think" but you (and even physicians cannot do this for themselves by the way) should not be self-diagnosing anything.

This is like saying lots of doctors are smart enough to learn and understand the bits of knowledge they need to accurately train LLMs and put them in charge of [life threatening system].

But also, there's nothing magical about the medical guild initiation process.

You're right, it's not magical. It's just 10+ years of medical training.

gok• 3 years ago

Those bullet points make LLMs sound just like most human doctors.

throwaway14356• 3 years ago

it seems to me that you can hardcode the answers to the riddles and exactly match symptoms with illnesses then sort them by likelyhood and provide again hardcoded tests (proposals for) to gather further data.

It also seems capable of anonymizing a large chunk of medical data that we would not want to share normally. Who knows, perhaps it could even be a means of payment.

zoogeny• 3 years ago

It seems that time and time again, transformers are the swiss army knife of learning systems. And specifically LLMs are proving to be like chameleons. In some ways that shouldn't be surprising. Some say that math is a universal language after all, and we seem to agree that math is unreasonably effective at describing reality.

carbocation• 3 years ago

I find chat GPT to be very helpful for working with programming languages that I’m less comfortable using (shell, python). I know enough to evaluate correct code in these languages, but producing it from scratch is more difficult, which seems like a sweet spot for carefully using ChatGPT for code.

As a physician, I would not be surprised if the medical use of these tools ends up having similar value.

haldujai6• 3 years ago

Unlike search/random chatbots the bar to beat in medicine (UpToDate) is much higher, when that happens I agree (in terms of diagnosis and management).

tstrimple• 3 years ago

I think the key here is that experts can take better advantage of tools like these because they have more ability to see when it's going off the rails. If you're a brand new programmer, you might be stumped if ChatGPT "hallucinates" a function which doesn't exist within an API. But an experience developer can pick up on the problem pretty quickly and either correct for it or know they need to pursue more traditional routes to solve the problem.

I recently used ChatGPT because my Google was failing to help me remember the name of the standard for securely sharing passwords between systems. My searches kept turning up end user password management related topics. ChatGPT got me to SCIM after one question and one correction.

I could absolutely see a doctor using something like a ChatGPT to help supplement their memory in a way I did. I don't think anyone recommends that doctors just trust ChatGPT, but to use it as a supplementary tool for their own expertise. Even if it's outside of their specific medical domain, it could help them get a basis for having a conversation with one of their specialist colleagues.

MrPatan2• 3 years ago

Do you reckon there's pharma people right now wondering how to make LLMs push their drugs?

Take fine-tuning trainers to "conferences", perhaps?

Will they try to make their own?

What a next few years...

arthurcolle• 3 years ago

Definitely ongoing right now

Large Language Models Are Few-Shot Health Learners

Related Stories