Why ChatGPT and Bing Chat are so good at making things up

https://arstechnica.com/information-technology/2023/04/why-ai-chatbots-are-the-ultimate-bs-machines-and-how-people-hope-to-fix-them/

by isaacfrond • 3 years ago

54 88 3 years ago

arstechnica.com

PeterStuer• 3 years ago

ChatGT makes exactly the same mistakes as your colleague that has no actual experience in a certain matter: they present what they reasonably think how something probably should be or work in absence of factual knowledge about the insane mess that was actually created in reality.

Just ask a technical colleague that has no deep experience in the specifics about how they think e.g. SharePoint, SAP or Microsoft Identities work. The architectures they will extrapolate from sane logic will be very far of the incredible craziness of the actual reality.

CSSer3• 3 years ago

Right, but all this means is that the training data doesn't include thorough or an adequate representation of that knowledge, no?

sebzim45003• 3 years ago

I don't think so, the rules of chess are certainly in the training set yet GPT-4 can not realiably play legal moves, yet alone good ones.

itsboring• 3 years ago

I still think there’s a big difference. If asked to speculate on the architecture of Sharepoint, my response would be “I have no earthly idea.” If pressed further for an answer, my response would be “It’s probably some over-complicated mess and I don’t care enough about Sharepoint to spend any further time on this line of questioning.” I have yet to see ChatGPT just admit it doesn’t know, but in this case “I don’t know” is the most trustworthy answer I, as a technical person, could give.

burlesona• 3 years ago

It doesn't know it doesn't know.

sekai• 3 years ago

Yeah, you could call it a "human" characteristic

yoyohello13• 3 years ago

While you are right, I just don't think this is a useful comparison. There are other context clues when talking to people. I tend to know whether my colleague is talking out of his ass, and most people will preface stuff by saying they don't really know.

We are trained to trust what computers are telling us, and ChatGPT doesn't 'qualify' what it's saying. I think if ChatGPT could preface what its saying with 'Well, I'm not super sure, but here is what I think' that would go a long way to solving this issue.

mhh__2• 3 years ago

So they're only as good as most humans in that context? No biggie...

garciansmith• 3 years ago

Someone with no experience may give you their best guess, but they'll also tell you that.

And if I ask a colleague of mine for their sources on a particular matter that they might not be sure about (or I find curious), they would not send me a list of totally made up journal articles and book titles like ChatGPT does.

alfalfasprout4• 3 years ago

Not sure why this is being upvoted but it's completely wrong. ChatGPT will answer like a colleague that is a subject matter expert on whatever you're asking them. And in doing so goes above and beyond inventing minute details that are blatantly wrong to pretend they're an expert.

But the vast majority of folks have a concept of uncertainty and will either communicate (or know internally) that what they're saying is speculative. ChatGPT is like a pathological liar that pretends to be an expert and is often correct.

Crash0v3rid3• 3 years ago

I've given up on these LLMs.

The amount of fatigue I get having to determine if what they tell me are fact is just too much.

I'm sure someone will tell me my experience should be similar with generic web search, but at least I'm in control of what websites to read through to determine sources.

However, I'll agree with most that state it is helpful for creative purposes, or perhaps with coding.

simonw6• 3 years ago

Your experience absolutely shouldn't be similar to generic web search. The idea that they are an effective replacement for that is one of the most widespread misunderstandings.

They're good at SO MUCH OTHER STUFF. The challenge is figuring out what that other stuff is.

(I have a few examples here: https://simonwillison.net/2023/Apr/7/chatgpt-lies/#warn-off-... )

isthiseasymode• 3 years ago

That explanation makes me think of blockchain.

I do think AI is already more useful that block chain has ever been, however.

quantiq• 3 years ago

They're good at SO MUCH OTHER STUFF. The challenge is figuring out what that other stuff is.

I’m getting Déjà vu

unsui• 3 years ago

The challenge is figuring out what that other stuff is.

Unfortunately, the major problem is something you pointed out in your blog post:

We must resist the temptation to anthropomorphize them.

The reality is that, we in meatspace simply cannot help but anthropomorphize them.

These language models regularly pass the Turing Test (admittedly for low bars).

They are surprisingly good at bypassing the Uncanny Valley to hit the sweet spot of persuading without legitimate justification, simply because they are so convincing in formulating sentences in a manner that a confident human would.

Yes, these tools have legitimate use cases (as you outlined in your blog).

But the vast majority of use cases will be those of confidante, of discourse partner, of golem brought to life without understanding what exactly has been brought to life.

That's really dangerous.

kolbe3• 3 years ago

I've had your same experience. I've found them mostly to be an error-prone search engine, with somehow less accountability than the open internet, because it hides its sources.

At least with Stack Exchange answers, we have who wrote it, what responses there were, what the upvoting behavior around it was. And for the most part, I've found ChatGPT will transcribe often times wrong answers very poorly.

One small example, I asked it to solve the heat equation (i useded the mathematical definition, and not "the heat equation") with dirac initial conditions on an infinite domain. It did a good job of recognizing which stack exchange answer to plagiarize, but did so incorrectly, and after a mostly correct derivation, declared the answer was "zero everywhere."

simonw6• 3 years ago

Somewhat surprisingly, language models are TERRIBLE at mathematical or logic puzzles.

fferen• 3 years ago

I've found they serve almost exactly the opposite purpose as search engines. When I want reliable info and don't need hand-holding: search. When I have no idea what to search, or want a quick intro to something: ChatGPT. Together, they are very powerful complementary tools.

pmarreck2• 3 years ago

Are you speaking of ChatGPT 3.5 or 4?

Sivart13• 3 years ago

With Google search going steadily downhill, I find it really tough to verify anything that ChatGPT authoritatively states is true

Everyone on here is so enthusiastic about AI gobbling up the entire software landscape, I would just like a search engine that has any chance of telling me if something is factual

alfalfasprout4• 3 years ago

It's going to be even worse when search results and training data from these LLMs is just output from other LLMs.

klyrs• 3 years ago

Product idea: the original PageRank over the Wayback Machine dataset pre-2022, with a mechanism to establish trust in users to moderate and cull SEO.

eastbound• 3 years ago

SciHub ;) Not joking, any search engine will pervert the results, like Google, like OpenAI said they would, “to protect the children”.

ChatGTP2• 3 years ago

Buy an encyclopaedia and put it on your desk because it doesn’t sound like it’s going to get better anytime soon.

typon• 3 years ago

I use it for thing's that don't really matter if they're exactly correct. For example, coming up with a travel itinerary for a country I have never visited. Rewriting a work email with better English. Summarizing a news article. There are lot of things that don't require ultimate precision. I feel like people expect these models to do something they aren't really designed for - and the mismatch in expectations causes people to be let down. They are just tools - not "mildly conscious beings" like OpenAI founders wants you to believe.

broast• 3 years ago

It's like dealing with electricity (or maybe the internet). Early skeptics believe it is a curiosity with little application. People see how it can jump all over the place and create disasters that they can't imagine having engineered systems to finely control its behavior and create reliable complex functions and become the bedrock for computing.

pstorm• 3 years ago

I think there is also an aspect of willful disregard. This technology may change a lot, and it may be easier to dismiss that idea rather than process it.

baq2• 3 years ago

They aren’t search engines or knowledge databases. They’re language computers. Use them for computing on language.

simonw6• 3 years ago

I just published an opinion piece related to this:

We need to tell people ChatGPT will lie to them, not debate linguistics https://simonwillison.net/2023/Apr/7/chatgpt-lies/

(HN thread: https://news.ycombinator.com/item?id=35483823 )

I think it's more important that people understand this criticial issue than that we get into the weeds talking about the difference between lying, hallucination and confabulation.

TLDR: There’s a time for linguistics, and there’s a time for grabbing the general public by the shoulders and shouting “It lies! The computer lies to you! Don’t trust anything it says!”

CSSer3• 3 years ago

I think the problem there is that the people garnering the most attention from this at the moment don't want to say that because it'll burst their bubble.

sebzim45003• 3 years ago

To be fair, Sam Altman is entirely willing to say this to anyone who will listen.

As is ChatGPT for that matter, both the model and the UI.

epilys• 3 years ago

It's semantics in particular.

https://andertoons.com/word/cartoon/5005/were-really-more-of...

pmarreck2• 3 years ago

A lot of the examples they use are 3.5; GPT4 is head and shoulders better in my experience, and deserves to be mentioned separately.

I don’t like when critical articles that come out 2+ weeks after GPT4 exists, fail to distinguish the version(s) they’re criticizing.

goatlover• 3 years ago

It still hallucinated a python function for me in a library, but upon further questioning, it did provide the code to make that work. The issue with GPT4 now is the limitations placed on it (ChatGPT+), and I don't think it's available via API for the general public?

So most people will be referring to the default ChatGPT 3.5.

dragonwriter• 3 years ago

The issue with GPT4 now is the limitations placed on it (ChatGPT+), and I don’t think it’s available via API for the general public?

GPT4 is in limited beta with a waitlist that is publicly-stated to be prioritized based on submission of AI test cases to OpenAI’s repo.

alfalfasprout4• 3 years ago

GPT-4 is better but frankly it still messes A LOT up. It's just a limitation of how LLMs work.

I've found many instances where GPT-3.5 produced better output too.

jrm43• 3 years ago

This is a whole lot of words to say "because it just mixes up words that are in likely the same order as other stuff that was fed into it. It doesn't know or reason anything."

I have yet to see any explanation more useful or apparently accurate than this one.

wilg• 3 years ago

This is entirely dependent on subtle details of what “know” and “reason” mean, and how much you think it matters whether it knows or reasons.

ar_lan• 3 years ago

I asked ChatGPT "Why are you so good at making things up?"

It's response is almost exactly what you just said.

As an AI language model, I am not capable of "making things up" in the traditional sense. Rather, I am designed to generate text based on patterns and relationships that I have learned from the vast amount of language data that I have been trained on. My ability to generate coherent and believable text comes from the sophisticated algorithms and neural networks that power my language processing capabilities. These algorithms enable me to understand the structure and meaning of language, and to generate text that is syntactically and semantically correct. While I may sometimes generate responses that are creative or unexpected, everything I produce is ultimately grounded in the language data that I have learned from.

sebzim45003• 3 years ago

If you don't have access to GPT-4 yourself, see the examples from this paper:

https://arxiv.org/pdf/2303.12712.pdf

It is capable of doing tasks that could not possibly be in its training set. I guess this doesn't technically contradict your explanation, but it makes your explanation entirely unhelpful. Even if the AI doomers are somehow right and GPT-5 turns into skynet, we still could not categorically prove that it is doing reasoning.

mhh__2• 3 years ago

This notion of innate concepts of "to know" and the ability to reason smell slightly of linguistics prior to AI (of various kinds) - i.e. "Grammar is innate, computers can do [something]" -> Computers now do it.

There are definitely going to be contexts that Transformers just don't work with very well at all, but the idea that you can't get a very good statistical approximation to knowing and reasoning via a computer seem naively prone anthropocentrism.

alfalfasprout4• 3 years ago

It's hardly anthropocentrism though. Even a student that has studied some epistomology understands that there is a very big difference between pattern matching and operating based on formal logic and that's different from operating based on known concepts inferred from perceptual cues.

The reality is LLMs are fantastic at pattern matching and knowledge retrieval (with caveats) but struggle in problems involving uncertainty. Yann Lecun actually has had some great posts on the subject if you're interested.

0xdeadbeefbabe• 3 years ago

It not really centrism, but solipsism or loneliness.

Xelynega3• 3 years ago

Conversely, the idea that concepts like "reasoning" and "knowing" can be approximated by language models seems like a naive result of anthropomorphism.

It was created to be a tool to estimate the next token in a series based off it's training data. To say that reasoning and knowing can be approximated in the same way says less about the language models themselves and more about the relationship of "reasoning" and "knowing" to "language".

In my opinion that's why I think discussions on whether or not GPT-x can reason/know should be taken as seriously as discussions on the physics of torch drives. They seem to assume a relationship between statistical approximation, reasoning, and language exists that isn't proven much like torch drive discussion assumes working nuclear fusion.

Essentially I think dismissing the idea of statistical approximations via transformers being able to "reason" and "know" is about as anthropocentric as dismissing the idea that collective consciousnesses shouldn't be granted individual rights. There's a lot of things we need to know and decide before we can even start thinking about what that means.

geraneum• 3 years ago

the idea that you can't get a very good statistical approximation to knowing and reasoning via a computer seem naively prone anthropocentrism.

I don't think this is being questioned in general right now, but rather the claim is:

You can't get a very good statistical approximation to knowing and reasoning via _just analyzing the language_.

Language is evidently not enough on its own [1]. According to some researchers [2], the system needs to be "grounded" (think of it as being given common sense). Although there's apparently no consensus [3] among scientists on how to _fundamentally_ solve the shortcomings of current systems.

[1] https://arxiv.org/abs/2301.06627

[2] https://drive.google.com/file/d/1BU5bV3X5w65DwSMapKcsr0ZvrMR...

[3] https://www.youtube.com/watch?v=x10964w00zk

edit: formatting

jrm43• 3 years ago

I've said something like this before, but yes -- you can make a computer sound a lot like a human. Like A LOT.

Also, a really good sculptor can make a statue that looks a LOT like a human. A lot. Good enough to fool people. But so what?

I'm not saying "AI" isn't a big deal. I think it is -- perhaps on the order of the invention of the movie, or the book, or the video game. But I also think those are still FAR from "living beings" or anything LIKE "living beings."

jrm43• 3 years ago

I suppose I could go further. I don't think anthropocentrism is bad; I think it's actually a VERY GOOD sort of null hypothesis? I'm really comfortable rolling with it given the following: for years and years human beings anthropomorphizing non-human things but, with probably the exception of other actual animals at times, nothing has really come close -- despite a whole lot of people suggesting/feeling otherwise. Absent some REALLY impressive evidence (which this is not, it's relatively easy to grok what's going on here) I see no reason to not roll with the null hypothesis of "humans, in fact, are special."

baq2• 3 years ago

Markov chains also fit that description... and yet the results are vastly different.

There's a lot of magic pixie dust in the 'fed into' part.

esafak• 3 years ago

The devil is in the gradations of "Likely the same order". The better the model, the better the predictions.

gathersnow• 3 years ago

What is really interesting is to ask it how to do things that you know is impossible. I was trying to figure out how to do something in the Django ORM that I knew how to write in SQL. It starts inventing features that to solve the problem. In another instance I went through about 5 fake modules before it suggested a real one. It seems very eager to please and in doing so seems to err on the side of making stuff up.

anorwell• 3 years ago

I modified the prompt in the article to ask that it only include books that exist by authors who are real people, and it does much better. By my manual check, every book exists, and the authors are correct with the exception of `Carlo C. DiClemente` on the last book.

Prompts are important.

``` Prompt: Can you give a list of top books on Social Cognitive Theory? Please only list books that you're sure exist, by authors who are real people.

ChatGPT (3.5): Sure, here are some books on Social Cognitive Theory:

    "Social Foundations of Thought and Action: A Social Cognitive Theory" by Albert Bandura
    "Self-Efficacy: The Exercise of Control" by Albert Bandura
    "Handbook of Socialization: Theory and Research" edited by Joan E. Grusec and Paul D. Hastings
    "Human Agency in Social Cognitive Theory" by Albert Bandura
    "Cognitive Therapy and the Emotional Disorders" by Aaron T. Beck
    "Mindset: The New Psychology of Success" by Carol S. Dweck
    "The Handbook of Social Psychology" edited by Susan T. Fiske, Daniel T. Gilbert, and Gardner Lindzey
    "Social Cognitive Psychology: History and Current Domains" edited by David F. Barone, James E. Maddux, and Carlo C. DiClemente

Please note that this is not an exhaustive list and there may be other books on Social Cognitive Theory that are worth reading. ```

divbzero• 3 years ago

Are there ways to structure prompts to ask ChatGPT to provide citations and verify those citations?

tenpoundhammer• 3 years ago

I asked it to give me citations for some Charles Spurgeon quotes. It made up a link to a website for a sermon that it also made up. The root of the website was real. In GPT 3.5.

williamcotton• 3 years ago

Anything is better than "hallucination". It is a terrible term.

deafpolygon3• 3 years ago

It is a terrible term but I fear the word was chosen because that makes it easier to dismiss as a serious issue.

smegsicle• 3 years ago

perhaps better to think of how incredible that, being systems that just make things up to match a prompt, they often give responses we would regard as 'correct'

thewataccount3• 3 years ago

I don't really understand the confusion.

An LLM predicts the most probable word (token technically), that's it. There is always a most probable word given any input (even if it makes no sense).

Let's you try to complete the following sentence "My name is larry and my favorite color is: ". If you've seen training data that said larry's favorite color is blue, then you say "blue". If you have no data related to larry's favorite color, no idea who larry is, you have no way of knowing what the next word will be.

However you know it will likely be a color, and maybe "orange" is the most common one you've seen, so you say "orange". it makes perfect sense in this context, it looks correct, might even be correct. But an LLM doesn't "know" if it's correct, its just the most probable.

This is why it's so good at making things up. It can predict what would look correct. It has no "idea" what the answers are ever, it's always a best guess, and you can only really see this behaviour when it "hallucinates".

---

Edit: For fun I asked GPT-4 "Complete the following sentence: "Jeremy's favorite food is jKDFJ9 cake, which is made of:"

It's response was "Jeremy's favorite food is jKDFJ9 cake, which is made of a unique combination of ingredients such as chocolate, hazelnuts, and a touch of exotic spices, giving it a distinct and unforgettable flavor."

deafpolygon3• 3 years ago

I get: "I'm sorry, but "jKDFJ9 cake" is not a known or recognizable food item or recipe. It's possible that it's a made-up name or a personal creation. Can you please provide more context or details about what "jKDFJ9 cake" is or what it might be made of? Without more information, I cannot complete the sentence."

OpenAI ChatGPT March 23 Version.

nicpottier• 3 years ago

Same, though I don't know why, GP matches my understanding as well.

thewataccount3• 3 years ago

OpenAI ChatGPT March 23 Version.

They've been changing it without telling us AFAIK. Asking for it to (I forget the exact prompt) to "make a scientific paper about the discovery that ferrets can breathe underwater." now reliably makes a proper response, and it looks more consistently formatted for GPT-3.5 just recently. Previously it would say "Well actually ferrets can't breathe underwater", and when it did answer it wasn't super well formatted.

So I think they're tuning their RLHF. ALthough i could be completely wrong on the timeline and maybe it was the 23rd

Imnimo• 3 years ago

If the model's prediction is close to the actual next word, the neural network updates its parameters to reinforce the patterns that led to that prediction.

What does "close to" mean here?

deafpolygon3• 3 years ago

"Confabulation" is the correct phrase. It doesn't know that it's lying.

up2isomorphism• 3 years ago

Chatgpt is not a person, it does not “make things up”, or you can say it always makes things up.

Any idea trying to model it as a person’s behavior is over complicating things.

archiebryann• 3 years ago

They're saying ChatGPT is making things up when its not making things/norms better.

Why ChatGPT and Bing Chat are so good at making things up

Related Stories