Study: AI responses to healthcare queries are nearly 76% accurate

https://www.psu.edu/news/research/story/calling-doctor-gpt-ai-responses-healthcare-queries-are-nearly-76-accurate

6 3 2 months ago

A total of 34 participants — comprising faculty, staff and undergraduate and graduate students — submitted 212 prompts and AI-generated responses to real and imaginary health concerns written from both patient and doctor perspectives. Participants were allowed to choose one of four LLMs to use for the contest: ChatGPT-4o, ChatGPT-3.5, Gemini-1.5 Pro and Llama3-8b.

xyzal• 2 months ago

Or: Almost a quarter of AI responses to healthcare queries wrong

jimrandomh• 2 months ago

This press release links to an arXiv article dated a year ago, which ran tests using AI models that were already seriously out of date at that time. The practical upshot of which is that, with respect to the question people care about, and which the headline claims to answer, this is basically pseudoscience.

Study: AI responses to healthcare queries are nearly 76% accurate

Related Stories