LLMs are steroids for your Dunning-Kruger

https://bytesauna.com/post/dunning-kruger

by gridentio • 8 months ago

392 298 8 months ago

Brendinooo4• 8 months ago

I think LLMs should not be seen as knowledge engines but as confidence engines.

This is a good line, and I think it tempers the "not just misinformed, but misinformed with conviction" observation quite a bit, because sometimes moving forward with an idea at less than 100% accuracy will still bring the best outcome.

Obviously that's a less than ideal thing to say, but imo (and in my experience as the former gifted student who struggles to ship) intelligent people tend to underestimate the importance of doing stuff with confidence.

shermantanktop• 8 months ago

Confidence has multiple benefits. But one of those benefits is social - appearing confident triggers others to trust you, even when they shouldn’t.

Seeing others get burned by that pattern over and over can encourage hesitation and humility, and discourage confident action. It’s essentially an academic attitude and can be very unfortunate and self-defeating.

Chabsff5• 8 months ago

I feel like LLMs are a fairly boring technology. They are stochastic black boxes. The training is essentially run-of-the-mill statistical inference. There are some more recent innovations on software/hardware-level, but these are not LLM-specific really.

This is pretty ironic, considering the subject matter of that blog post. It's a super-common misconception that's gained very wide popularity due to reactionary (and, imo, rather poor) popular science reporting.

The author parroting that with confidence in a post about Dunner-Krugering gives me a bit of a chuckle.

miningape3• 8 months ago

I also find it hard to get excited about black boxes - imo there's no real meat to the insights they give, only the shell of a "correct" answer

yannyu6• 8 months ago

What's the misconception? LLMs are probabilistic next-token prediction based on current context, right?

Chabsff5• 8 months ago

Yeah, but that's their interface. That informs surprisingly little about their inner workings.

ANNs are arbitrary function approximators. The training process uses statistical methods to identify a set of parameters that approximate the function as best as possible. That doesn't necessarily mean that the end result is equivalent to a very fancy multi-stage linear regression. It's a possible outcome of the process, but it's not the only possible outcome.

Looking at a LLMs I/O structure and training process is not enough to conclude much of anything. And that's the misconception.

parineum• 8 months ago

I'm not sure what claim your disputing or making with this.

What more are LLMs than statistical inference machines? I don't know that I'd assert that's all they are with confidence but all the configurations options I can play with during generation (Top K, Top P, Temperature, etc.) are all ways to _not_ select the most likely next token which leads me to believe that they are, in fact, just statistical inference machines.

ACCount379• 8 months ago

What more are human brains than piles of wet meat?

It's not an argument - it's a dismissal. It's boneheaded refusal to think on the matter in any depth, or consider any of the implications.

The main reason to say "LLMs are just next token predictions" is to stop thinking about all the inconvenient things. Things like "how the fuck does training on piles of text make machines that can write new short stories" or "why is a big fat pile of matrix multiplications better at solving unseen math problems than I am".

LeroyRaz10• 8 months ago

How is that a misconception? LLMs are just advanced statistical modelling (unsupervised machine learning) with small tweaks (e.g., some fine-tuning for human preference).

At the core, they are just statistical modelling. The fact that statistical modelling can produce coherent thoughts is impressive (and basically vindicates materialism) but that doesn't change the fact it is all based on statistical modelling. ...? What is your view?

sho_hn2• 8 months ago

I'm not sure this is something I really worry about. Whenever I use an LLM I feel dumber, not smarter; there's a sensation of relying on a crutch instead of having done the due diligence of learning something myself. I'm less confident in the knowledge and less likely to present it as such. Is anyone really cocksure on the basis of LLM received knowledge?

As I ChatGPT user I notice that I’m often left with a sense of certainty.

They have almost the opposite effect on me.

Even with knowledge from books or articles I've learned to multi-source and question things, and my mind treats the LLMs as a less reliable averaging of sources.

deadbabe• 8 months ago

If you feel dumber, it’s because you’re using the LLM to do raw work instead of using it for research. It should be a google/stackoverflow replacement, not a really powerful intellisense. You should feel no dumber than using google to investigate questions.

Insanity2• 8 months ago

I don't think this is entirely accurate. If you look at this: https://www.media.mit.edu/publications/your-brain-on-chatgpt..., it shows that search engines do engage your brain _more_ than LLM usage. So you'll remember more through search engine use (and crawling the web 'manually') than by just prompting a chatbot.

pessimizer2• 8 months ago

I find that it is terrible for research, and hallucinates 25% to 90% of its references.

If you tell it to find something and give it a detailed description of what you're looking for, it will pretend like it has verified that that thing exists, and give you a bulletpoint lecture about why it is such an effective and interesting thing that 1) you didn't ask for, and 2) is really it parroting your description back to you with embellishments.

I thought I was going to be able to use LLMs primarily for research, because I have read an enormous number of things (books, papers) in my life, and I can't necessarily find them again when they would be useful. Trying to track them down through LLMs is rarely successful and always agonizing, like pulling teeth that are constantly lying to you. A surprising outcome is that I often get so frustrated by the LLM and so detailed in how I'm complaining about its stupid responses that I remind myself of something that allows me to find the reference on my own.

I have to suspect that people who find it useful for research are researching things that are easily discoverable through many other means. Those are not the things that are interesting. I totally find it useful to find something in software docs that I'm too lazy to look up myself, but it's literally saving me 10 minutes.

Insanity2• 8 months ago

I remember back when I was in secondary school, something commonly heard was

"Don't just trust wikipedia, check it's resources, because it's crowdsourced and can be wrong".

Now, almost 2 decades later, I rarely hear this stance and I see people relying on wikipedia as an authoritative source of truth. i.e, linking to wikipedia instead of the underlying sources.

In the same sense, I can see that "Don't trust LLMs" will slowly fade away and people will blindly trust them.

aabhay• 8 months ago

There’s also the fact that both Wikipedia and LLMs are non-stationary. The quality of wikipedia has grown immensely since its inception and LLMs will get more accurate (if not explicitly “smarter”)

sho_hn2• 8 months ago

I don't think the cases are really the same. With Wikipedia people have learned to trust that the probability of the information being at least reasonably good is pretty high because there's an editing crucible around it and the ability to correct misinformation surgically. No one can hotpatch a LLM in 5mins.

tayo42• 8 months ago

Wikipedia is usually close enough and most users don't require perfection for their "facts"

Ive noticed things like gemini summaries on Google searches are also generally close enough.

rsynnott2• 8 months ago

Now, almost 2 decades later, I rarely hear this stance and I see people relying on wikipedia as an authoritative source of truth. i.e, linking to wikipedia instead of the underlying sources.

That's a different scenario. You shouldn't _cite wikipedia in a paper_ (instead you should generally use its sources), but it's perfectly fine in most circumstances to link it in the course of an internet argument or whatever.

miningape3• 8 months ago

"Don't just trust wikipedia, check it's resources, because it's crowdsourced and can be wrong"

This comes from decades of teachers misremembering what the rule was, and eventually it morphed into the Wikipedia specific form we see today - the actual rule is that you cannot cite an encyclopaedia in an academic paper. full stop.

Wikipedia is an encyclopaedia and therefore should not be cited.

Wikipedia is the only encyclopaedia most people have used in the last 20 years, therefore Wikipedia = encyclopaedia in most people's minds.

There's nothing wrong with using an encyclopaedia for learning or introducing yourself to a topic (in fact this is what teachers told students to do). And there's nothing specifically wrong about Wikipedia either.

CobrastanJorji• 8 months ago

A big reason for this is that Wikipedia's source is often a book or a journal article that is either offline or behind an academic paywall. Checking the source is effectively impossible without visiting a college campus's library. The likelihood that the cited information is wrongly summarizing the contents is low enough and the cost is high enough that doing so regularly would be irrational.

Theodores• 8 months ago

It wasn't just Wikipedia, which was a relatively recent addition to the web, everything online was a 'load of rubbish'.

In turn-of-the-century boomer world, reality was what you saw on TV. If you saw something with your own eyes that contradicted the world view presented by the media, then one's eyes were to be disbelieved. The only reputable sources of news were the mainstream media outlets. The only credible history books would be those with reviews from the mainstream media, with anything else just being the 'ramblings of a nutter'.

In short, we built a beautiful post-truth world and now we are set on outsourcing our critical thinking to LLMs.

thewebguyd5• 8 months ago

I can see that "Don't trust LLMs" will slowly fade away and people will blindly trust them.

That's already happening. I don't even think we had a very long "Don't trust LLMs" phase, if we did it was very short.

The "normies" already trust whatever they spit out. At leadership meetings at my work, if I say anything that goes against the marketing hype for LLMs, such as talking about "Don't trust LLMs", it's met with eye rolls and I'm not forward thinking enough, blah blah.

Management-types have 100% bought into the hype and are increasingly more difficult to convince otherwise.

Ultimatt2• 8 months ago

Well also years of Wikipedia proving to be more accurate than anything in print and rarely and not for very long misrepresenting source materials. For LLMs to get that same respect they would have to pull off all of the same reassuring qualities.

alfalfasprout2• 8 months ago

Wikipedia also didn't have remotely the controls they have today around review + locking contentious articles, etc.

AlexCoventry• 8 months ago

That's partly because they are getting more reliable, though, just as WP did.

jalapenos2• 8 months ago

Problem is they also included newspapers in authoritative sources - except foreign ones that is - and Wikipedia at least has some kind of peer review process.

It's genuinely as authoritative as most other things called authoritative.

more_corn2• 8 months ago

The difference is that Wikipedia validates its sources and LLMS don’t and can’t.

freejazz• 8 months ago

Is anyone really cocksure on the basis of LLM received knowledge?

Yeah, the stupid.

code_for_monkey• 8 months ago

unfortunately im like you and we are in the minority. The manager class loves the llm and doesnt seem to consider its flaws like that.

lukan• 8 months ago

Nah, I feel smart to use it in a smart way to get stuff done faster than before.

everdrive3• 8 months ago

This captures my experience quite well. I can "get a lot more done," but it's not really me doing the things, and I feel like a bit of a fraud. And as the workday and the workweek roll on, I find myself needing to force myself to look things up and experiment rather than just asking the LLM. It's quite clear that for most people LLMs will make the more dependent. People with better discipline I think will really benefit in big ways, and you'll see this become a new luxury belief; the disciplined geniuses around us will genuinely be perplexed why people are saying that LLMs have made them less capable, much in the same way they wonder why people can't just limit their drug use recreationally.

Brendinooo4• 8 months ago

it's not really me doing the things, and I feel like a bit of a fraud

I've been thinking about this a bit. We don't really think this way in other areas, is it appropriate to think this way here?

My car has an automatic transmission, am I a fraud because the machine is shifting gears for me?

My tractor plows a field, am I a fraud because I'm not using draft horses or digging manually?

Spell check caught a word, am I a fraud because I didn't look it up in a dictionary?

agumonkey2• 8 months ago

Most of the time it feels like a crutch to me. There has been a few moments where it unlocked deep motivation (by having a feel for the size of a solution based on chatgpt output) and one time a research project where any crazy idea I threw, it would imagine what it would entail in terms of semantics and then I was inspired even more.

The jury is Still out on what value these things will bring

borroka• 8 months ago

I have found them to be very useful.

Here are just a few examples of how I have used them.

My fountain pen stopped working, so I tried the common solutions recommended, but they did not solve the problem. Claude told me to try using a mixture of window cleaner and water. It worked! (The solution must have been in the corpus used to train Claude.)

I switched from W2 to consulting, but I didn't know anything about taxes. ChatGPT gave me the right recommendation, saving me hours of research.

I wanted to evaluate the quality of some shirts I have, so I used ChatGPT to estimate stitches per inch and the quality of the buttonholes and other details.

I planned my diet using ChatGPT. I had it calculate calories, macros, and deficit. Could I have done it without an LLM? Of course, but it made the planning much faster.

rsynnott2• 8 months ago

Is anyone really cocksure on the basis of LLM received knowledge?

Some people certainly seem to be. You see this a lot on webforums; someone spews a lot of confident superficially plausible-looking nonsense, then when someone points out that it is nonsense, they say they got it from a magic robot.

I think this is particularly common for non-tech people, who are more likely to believe that the magic robots are actually intelligent.

jayd162• 8 months ago

Well I think your experience is, if not in the minority, at least not the overwhelming majority.

Lots of folks think it's amazing and greatly empowers them.

nathan_compton• 8 months ago

Yeah, everything I get out of the AI stinks of wrongness, even when its not materially wrong. There is a flimsiness to everything.

bootsmann• 8 months ago

Is anyone really cocksure on the basis of LLM received knowledge?

I work for a company with an open source product and the number of support requests we get from people who ask the chatbot to do their config and then end up with something nonfunctioning is quite significant. Goes up to users complaining our api is down because the chatbot hallucinated the endpoint.

threetonesun• 8 months ago

LLMs do love to make up endpoints and parameters, but I have found that ones with web access are pretty good at copy/pasting configs if they can find them, so it might be worth a few minutes of exploring what people are actually finding that's causing it to make up an endpoint. I have not (yet!) seen an instance where making something easier for LLMs to parse didn't also help human comprehension.

tiltowait• 8 months ago

I work in DevSecOps, and devs sometimes come to us with AI-slop summaries and writeups about our own tooling. Any time I see emojis in a message, I know I'm about to have a laugh.

doctorpangloss• 8 months ago

Nothing stops you from spending the paltry $60 to find out how ridiculously good coding agents are. It’s only a matter of time for other problems.

chamomeal• 8 months ago

If I use LLMs too much I swear I can feel my brain powering down. Same feeling as playing a really grindy/mindless game

collingreen• 8 months ago

I agree but I've personally seen some egregious examples of people who are not only extremely confident in their new "knowledge" and "ability" but simultaneously think everyone else is extremely stupid. It's been absolutely wild to watch people paste chatgpt output and claim they wrote it, over and over again, even though every time I actually read it and ask a few "what does this mean" questions they have no idea and simply ask chatgpt then confidently say the response. It's so bad it's like a pathology; I wouldn't believe it if I hadn't seen it with my own eyes.

Something is happening here. Hopefully it's just revealing something that was already there in society and it isn't something new.

cachius• 8 months ago

From the title I thought this was a repost of 'AI is Dunning-Kruger as a service ' https://news.ycombinator.com/item?id=45851483

It is not.

gopheryourshelf• 8 months ago

“the problem with the world is that the stupid are cocksure, while the intelligent are full of doubt.”

Is it me or does everyone find that dumb people seem to use this statement more than ever?

stevenwoo• 8 months ago

It appears to be a paraphrasing of William Butler Yeats https://en.wikipedia.org/wiki/The_Second_Coming_(poem)

0xdeadbeefbabe• 8 months ago

Ugh. You can be cocksure of your doubts. It's still confidence, duh.

aeve890• 8 months ago

Everyone thinks they're the intelligent ones, of course. Which reinforces the repetition ad nauseam of Dunning Kruger. Which is on itself dumb AF because the effect described by Dunning and Kruger has been repeatedly exaggerated and misinterpreted. Which in turn is even dumber because Dunning-Kruger effect is debatable and reproducibility is weak at best.

the_af• 8 months ago

Yeah, nobody who ever mentions the DK effect (myself included) ever stops to consider they might be in the "dumb" cohort ;)

We are all geniuses!

vehemenz• 8 months ago

I hate to comment on just a headline—thought I did read the article—but it's wrong enough to warrant correcting.

This is not what the Dunning-Kruger effect is. It's lacking metacognitive ability to understand one's own skill level. Overconfidence resulting from ignorance isn't the same thing. Joe Rogan propagated the version of this phenomenon that infiltrated public consciousness, and we've been stuck with it ever since.

Ironically, you can plug this story into your favorite LLM, and it will tell you the same thing. And, also ironically, the LLM will generally know more than you in most contexts, so anyone with a degree epistemic humility is better served taking it at least as seriously as their own thoughts and intuitions, if not at face value.

topaz0• 8 months ago

Funnily enough, DK is also not real -- just a statistical artifact of a poorly chosen analysis.

dktoao• 8 months ago

Ok, I think you are going to need to explain to me why "Overconfidence resulting from ignorance" isn't exactly the same thing as "lacking metacognitive ability to understand one's own skill level". Just worded more simply

lukev• 8 months ago

I very much agree. I've been telling folks in trainings that I do that the term "artificial intelligence" is a cognitohazard, in that it pre-consciously steers you to conceptualize a LLM as an entity.

LLMs are cool and useful technology, but if you approach them with the attitude you're talking with an other, you are leaving yourself vulnerable to all sorts of cognitive distortions.

roywiggins2• 8 months ago

It certainly isn't helped by the RLHF and chat interface encouraging this. LLM providers have every incentive to make their users engage it like an other. It was much harder to accidentally do when it was just a completion UI and not designed to roleplay as a person.

cgriswald• 8 months ago

I don't think that is actually a problem. For decades people have believed that computers can't be wrong. Why, now, suddenly, would it be worse if they believed the computer wasn't a computer?

The larger problem is cognitive offloading. The people for whom this is a problem were already not doing the cognitive work of verifying facts and forming their own opinions. Maybe they watched the news, read a Wikipedia article, or listened to a TEDtalk, but the results are the same: an opinion they felt confident in without a verified basis.

To the extent this is on 'steroids', it is because they see it as an expert (in everything) computer and because it is so much faster than watching a TED talk or reading a long form article.

roywiggins2• 8 months ago

It can also dispense agreeable confirmation on tap, with very little friction and hardly any chance of accidentally encountering something unexpected or challenging. Even TED talks occasionally have a point of view that isn't perfectly crafted for each hearer.

mmaunder• 8 months ago

Use an agent to create something with a non-negotiable outcome. Eg software that does something useful, or fails to, in a language you don’t program in. This is a helpful way to calibrate your own understanding of what LLMs are capable of.

AndrewKemendo• 8 months ago

Humans broadly have a tenuous grasp of “reality” and “truth.” Propagandists, spies and marketers know what philosophers of mind prove all too well: most humans do not perceive or interact with reality as it is, rather their perception of it as it contributes or contradicts their desired future.

Provide a person confidence in their opinion and they will not challenge it, as that would risk the reward of lend you live in a coherent universe.

The majority person has never heard the term “epistemology” despite the concept being central to how people derive coherence. Yet all these trite pieces written about AI and its intersectionality with knowledge claim some important technical distinction.

I’m hopeful that a crisis of epistemology is coming, though that’s probably too hopeful. I’m just enjoying the circus at this point

LeroyRaz10• 8 months ago

Hacker News readers, and especially commenters, are number 1!

balderdash• 8 months ago

I ascribe the effect of LLMs as similar to reading the newspaper, when I learn about something I have no knowledge base in I come away feeling like I learned a lot. When I interact with a newspaper or LLM in an area where I have real domain expertise I realize they don’t know what they are talking about - which is concerning about the information I get from them about topics I don’t have that high level of domain expertise.

moffkalast2• 8 months ago

And why stop at newspapers, it's been a while since one could say books have any integrity, pretty much anyone can get anything into print these days. From political shenanigans to self help books designed to confirm people's biases to sell more units. Video's by far the hardest to fake but that's changing as well.

Regardless of what media you get your info from you have to be selective of what sources you trust. It's more true today than ever before, because the bar for creating content has never been lower.

Night_Thastus• 8 months ago

The problem is that LLM output is so incredibly confident in tone. It really sounds like you're talking to an expert who has years of experience and has done the research for you - and tech companies push this angle quite hard.

That's bad when their output can be complete garbage at times.

defgeneric• 8 months ago

Also known as the "Gell-Mann amnesia effect" [1].

[1] https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect

avree• 8 months ago

The title makes this incomprehensible. The author seemingly defines Dunning-Kruger as the... opposite of the Dunning-Kruger effect.

gowld3• 8 months ago

The "Dunning-Kruger Effect" Effect: A reference to Dunning-Kruger Effect is almost certainly incorrect.

kraftman4• 8 months ago

I feel like when I talk to someone and they tell me a fact, that fact goes into a kind of holding space, where I apply a filter of 'who is this person that is telling me this thing to know what the thing they are telling me is'. There's how well I know them, there's the other beleifs I know they have, there's their professional experience and their personal experience. That fact then gets marked as 'probably a true fact' or 'mark beleives in aliens'.

When I use chatGPT I do the same before I've asked for the fact: how common is this problem? how well known is it? How likely is that chatgpt both knows it and can surface it? Afterwards I don't feel like I know something, I feel like I've got a faster broad idea of what facts might exist and where to look for them, a good set of things to investigate, etc.

giraffe_lady2• 8 months ago

The important part of this is the "I feel like" bit. There's a fair but growing bit of research that the "fact" is more durable in your memory than the context, and over time, across a lot of information, you will lose some of the mappings and integrate things you "know" to be false into model of the world.

This more closely fits our models of cognition anyway. There is nothing really very like a filter in the human mind, though there are things that feel like them.

kraftman4• 8 months ago

Maybe but then thats the same wether I talk to chatGPT or a human isnt it? except with chatgpt i instantly verify what im looking for, whereas with a human i cant do that.

medstrom• 8 months ago

Reminds me of "default to null":

The mental motion of “I didn’t really parse that paragraph, but sure, whatever, I’ll take the author’s word for it” is, in my introspective experience, absolutely identical to “I didn’t really parse that paragraph because it was bot-generated and didn’t make any sense so I couldn’t possibly have parsed it”, except that in the first case, I assume that the error lies with me rather than the text. This is not a safe assumption in a post-GPT2 world. Instead of “default to humility” (assume that when you don’t understand a passage, the passage is true and you’re just missing something) the ideal mental action in a world full of bots is “default to null” (if you don’t understand a passage, assume you’re in the same epistemic state as if you’d never read it at all.)

https://www.greaterwrong.com/posts/4AHXDwcGab5PhKhHT/humans-...

jancsika2• 8 months ago

Afterwards I don't feel like I know something, I feel like I've got a faster broad idea of what facts might exist and where to look for them, a good set of things to investigate, etc.

Can you cite a specific example where this happened for you? I'm interested in how you think you went from "broad idea" to building actual knowledge.

kraftman4• 8 months ago

Sure. I wanted to tile my bathroom, from chatgpt i learned about laser levels, ledger boards, and levelling spacers (id only seen those cross corner ones before).

chaostheory• 8 months ago

There are so many guardrails now that are being improved daily. This blog post is a year out of date. Not to mention that people know how to prompt better these days.

To make his point, you need specific examples from specific LLMs.

jakubmazanec• 8 months ago

It's possible that the Dunning-Kruger effect is not real, only a measurement or statistical artefact [1]. So it probably needs more and better studies.

[1] https://www.mcgill.ca/oss/article/critical-thinking/dunning-...

travisgriggs• 8 months ago

8 months or so ago, my quip regarding LLMs was “stochastic parrot.”

The term I’ve been using of late is “authority simulator.” My formative experiences with “authority figures” was a person who can speak with breadth and depth about a subject and who seems to have internalized it because they can answer quickly and thoroughly. Because LLMs do this so well, it’s really easy to feel like you’re talking to an authority in a subject. And even though my brain intellectually knows this isn’t true, emotionally, the simulation of authority is comforting.

Qem• 8 months ago

8 months or so ago, my quip regarding LLMs was “stochastic parrot.” The term I’ve been using of late is “authority simulator.”

I guess soon we'll hear them called weapons of mass epistemic destruction.

saltcured3• 8 months ago

And for some of us, it may be an anti-authoritarianism stimulator.

GMoromisato2• 8 months ago

Speaking of uncertainty, I wish more people would accept their uncertainty with regards to the future of LLMs rather than dash off yet another cocksure article about how LLMs are {X}, and therefore {completely useless}|{world-changing}.

Quantity has a quality of its own. The first chess engine to beat Gary Kasparov wasn't fundamentally different than earlier ones--it just had a lot more compute power.

The original Google algorithm was trivial: rank web pages by incoming links--its superhuman power at giving us answers ("I'm feeling lucky") was/is entirely due to a massive trove of data.

And remember all the articles about how unreliable Wikipedia was? How can you trust something when anyone can edit a page? But again, the power of quantity--thousands or millions of eyeballs identifying errors--swamped any simple attacks.

Yes, LLMs are literally just matmul. How can anything useful, much less intelligent, emerge from multiplying numbers really fast? But then again, how can anything intelligent emerge from a wet mass of brain cells? After all, we're just meat. How can meat think?

svieira3• 8 months ago

How can meat think?

Some of us used to think that meat spontaneously generated flies. Maybe someday we'll (re-)learn that meat doesn't spontaneously generate thought either?

ACCount379• 8 months ago

I don't give much merit to ideas that demand the existence of Magic Fairy Dust.

And especially not now. Not when LLMs can already do pretty much anything that a human can - and some of those things they can even do well.

saltcured3• 8 months ago

My light-sensing meat read this as "spontaneously generated files" about 3 times before finally seeing maggots instead of tokens...

hathawsh2• 8 months ago

Most of HN has probably seen this gem about "thinking meat", but in case you haven't: https://www.mit.edu/people/dpolicar/writing/prose/text/think...

GMoromisato2• 8 months ago

Yes, thanks for the link.

I just re-read it a few weeks ago and it was still fresh in my mind!

kayge• 8 months ago

Also a great read-through of this by H. Jon Benjamin (Archer / Bob's Burgers) and Maeve Higgins: https://www.youtube.com/watch?v=5usXhX0zaO4

zahlman5• 8 months ago

How can you trust something when anyone can edit a page? But again, the power of quantity--thousands or millions of eyeballs identifying errors--swamped any simple attacks.

Sure, but now the established power users are free to insert more subtle attacks. The https://xkcd.com/978/ problem never stopped and the "reliable sources" consideration process allows for considerable political bias.

TeMPOraL2• 8 months ago

Yes, LLMs are literally just matmul. How can anything useful, much less intelligent, emerge from multiplying numbers really fast? But then again, how can anything intelligent emerge from a wet mass of brain cells? After all, we're just meat. How can meat think?

LLMs actually hint at an answer to that, but most people seem to be focusing too much on matmuls or (on the other end) specific training inputs to pay attention to where the interesting things happen.

Training an LLM builds up a structure in high-dimensional space, and inference is a way to query the shape of that structure. That's literally the "quality of quantity", reified. This is what all those matmuls are doing.

How can anything useful, much less intelligent, emerge from a bunch of matmuls or wet mass of brain cells? That's the wrong level of abstraction. How can a general-purpose quasi-intelligence emerge from a stupidly high-dimensional latent space that embeds rich information about the world? That's the interesting question to ponder, and it starts with an important realization: it's not obvious why it couldn't.

gloomyday• 8 months ago

I've been around long enough to see this saying that "As soon as it works, no one calls it AI anymore" in action many times.

It is almost infuriating how dismissive people are of such amazing technologies when they understand it. If anything, progress is often marked by having things becoming simpler rather than more complex. The SpaceX Raptor engine versions are such a cool example of that.

johnnyanmac3• 8 months ago

I don't pretend to know the long term future of llms. But I get this dismissal everytime I suggest "this is unsustainable, this is going to crash". No matter what trends I point to.

I won't pretend to know what lies beyond that. I just know on 5 years you're not going to spam AI in your deck and get millions in funding.

phamson02• 8 months ago

I partly share the author's point that ChatGPT users (myself included) can "walk away not just misinformed, but misinformed with conviction". Sometimes I want to criticise aloud, write a post blaming this technology for those colourful, sophisticated, yet empty bullshits I hear from a colleague or read in an online post.

But I always resist the urge. Because I think: Isn't it always going to have some kinds of people like that? With or without this LLM thing.

If there is anything to hate about this technology, for the more and more bullshits we see/hear in daily life, it is: (1) Its reach: More people of all ages, of different backgrounds, expertise, and intents are using it. Some are heavily misusing it. (2) Its (ever increasing) capability: Yes, it has already become pretty easy for ChatGPT or any other LLMs to produce a sophisticated but wrong answer on a difficult topic. And I think the trend is that with later, more advanced versions, it would become harder and take more effort to spot a hidden failure lurking in a more information-dense LLM's answer.

bryanlarsen3• 8 months ago

My opinion: if LLM's speed you up, you're doing it wrong. You have to carefully review and audit every line that comes out of an LLM. You have to spend a lot of time forcing LLM's to prove that the code it wrote is correct. You should be nit-picking everything.

Despite, LLM's are useful. I could write the code faster without an LLM, but then I'd have code that wasn't carefully reviewed line-by-line because my coworkers trust me (the fools). It'd have far fewer tests because nobody forced me to prove everything. It'd have worse naming because every once in a while the LLM does that better than me. It'll be missing a few edge cases the LLM thought of that I didn't. It'd have forest/trees problems because if I was writing the code I'd be focused on the code instead of the big picture.

nzach• 8 months ago

You have to carefully review and audit every line that comes out of an LLM. You have to spend a lot of time forcing LLM's to prove that the code it wrote is correct. You should be nit-picking everything.

I'm not sure this statement is true most of the time. This kind of reasoning reminds me of the discussion around 'code correctness'. In my opinion there are very few instances where correctness is really important. Most of the time you just need something that works well enough.

Imagine you have a continuous numeric scale that goes from 'never works' to '100% formal proofs' to indicate the correctness of every piece of software. Pushing your code to the '100% formal proofs' side takes a lot of resources, that could be deployed on other places.

bryanlarsen3• 8 months ago

At least for us, every bug that makes it into a release that gets installed on a client computer costs us 100x - 1000x as much as a bug that gets caught earlier.

Foobar85682• 8 months ago

I start feeling that LLM are hallucinating less than people, no matter the fields, I am at the stage where I trust more code written by an LLM than by a person.

Typically, for the last 2 years, I don't feel that anyone can or bother to read anymore.

pants2• 8 months ago

I've seen this! Following some Math and Physics subreddits it's a regular occurrence for a new submitter to come in and post some 40 pages of incomprehensible bullshit and claim that they developed a unifying theory of physics with ChatGPT and that ChatGPT has told them it's a breakthrough in the field. Of course that used to happen regularly before LLMs but not nearly as often.

turtletontine• 8 months ago

Including the former CEO of Uber. I’m somewhat curious what these people even think they’ve discovered, what outstanding problem they think they’ve actually solved… but I’m not curious enough to actually dig through their slop.

https://gizmodo.com/billionaires-convince-themselves-ai-is-c...

danparsonson• 8 months ago

"Vibe physics" good lord... It's like reading the thoughts of a five year old - absolutely certain that they know how the world works with little or no basis in reality.

ramraj07• 8 months ago

Hey! you called me out: https://news.ycombinator.com/item?id=35178196

simianwords• 8 months ago

How often do you think a ChatGPT user walks away not just misinformed, but misinformed with conviction? I would bet this happens all the time. And I can’t help but wonder what the effects are in the big picture.

this is so wrong! i simply can't get ChatGPT to admit something clearly wrong. it can play both sides and gives nuance which is exactly what i expect. but it is so un-sycopanthic that it won't leave you feeling like you are right. any examples of it doing so are welcome! show me examples where it takes a clearly wrong or false idea and makes it look as if it is a good idea (unless you specifically ask it to do it).

uoaei• 8 months ago

Freely available online information is very often educationally incredibly shallow and commonly oversimplified to the point of being wrong. So of course an agent trained on it would be, too.

zkmon3• 8 months ago

> How often do you think a ChatGPT user walks away not just misinformed, but misinformed with conviction? I would bet this happens all the time.

Why is the "Dunning-Kruger" is not mentioned anywhere in the article body while is gloriously visible in the title? By the way, AI is not wrong "all the time".

rwolf• 8 months ago

"all the time" in this context means frequently, not constantly, and is usually understood as a global rate.

for example: people fall in love all the time.

zkmon3• 8 months ago

I don't think you get "frequently" misinformed by ChatGPT. What kind of prompts you are using, that make ChatGPT to give false info very often?

resters• 8 months ago

they do not have to be. People who seek an idea bubble end up finding one.

vanillax• 8 months ago

I'll take big - enterprise AI written code, over offshore any day of the week. I for one welcome AI for this exact use case.

djoldman2• 8 months ago

Just a reminder that the "Dunning-Kruger effect" is probably not real [0].

It makes sense to refer to it as a concept but it's probably not an appropriate assumption to make about people.

[0] https://www.mcgill.ca/oss/article/critical-thinking/dunning-...

jayd162• 8 months ago

This seems to move the idea that you might not understand how skilled you are to some kind of law that ties humility to knowledge more strictly.

Maybe this is my misunderstanding but I don't think the common invocation really took it as a law that the unknowledgeable always think their skills are higher.

nis0s2• 8 months ago

There’s a gap that LLMs are trying to fill in such cases, which is that there’s too much information that we can possibly hope to make sense of in a lifetime. Just as it’s possible to compute something incorrectly with a calculator, you can definitely be led astray by an LLM, which is why I am surprised that people think these models are good enough to replace humans at work. The only thing which makes sense is to both raise the bar for publishing, and to only take published works seriously. If something isn’t published, then authors should provide code to demonstrate the effect they’re describing.

thewebguyd5• 8 months ago

which is why I am surprised that people think these models are good enough to replace humans at work.

There are a lot of office jobs that I'd fit into the category of "bullshit jobs." They may serve some purpose in the huge bureaucracy of enterprises but the day to day ultimately boils doing to managing someone's calendar and sending emails.

Quite a few people at my work have now started using Copilot for their emails. It's obviously AI (at least to me), and yet, the content and formatting are an improvement over what they were sending before.

So much of the marketing hype on LLMs is about how it'll replace all the engineering work (the MBA's wet dream, to replace all the expensive labor). In reality, I think its more capable at replacing non-tech labor and middle management.

An LLM can send out an email to the team and analyze a project check-in faster, and better, than some overpaid middle manager can. I have no doubts an LLM could probably serve the role of a project management office, or a business analyst.

Sure, there should still be a human in the loop for now, but you need far, far less humans in those roles than previously.

nis0s2• 8 months ago

I go back and forth on the idea that some jobs are bullshit, maybe I haven’t been exposed to enough industries or work places. Every place I worked definitely didn’t have bullshit jobs to hand out as adult daycare, but I can see how some places can become bloated because an over ambitious middle manager wants to say they manage X number of people on their resume. So there are bullshit jobs in that there are people who aren’t being utilized correctly, so in that case I’d say they’re no bullshit jobs, just bullshit leadership or managers.

Foobar85682• 8 months ago

Spend half a day with me, and you'll understand why LLM can replace most people in a company.

gowld3• 8 months ago

I recently asked a leading GenAI chatbot to help me understand a certain physics concept. As I pressed it on the aspect I was confused about, the bot repeatedly explained, and in our discussion, consistently held firm that I was misunderstanding something, and made guesses about what I was misunderstanding. Eventually I realized and stated my mistake, and the chatbot confirmed and explained the difference between my wrong version and the truth. I looked at some sources and confirmed that the bot was right, and I had misremembered something.

I was quite impressed that it didn't "give in" and validate my wrong idea.

hathawsh2• 8 months ago

I've seen similar results in physics. I suspect LLMs are capable of redirecting the user accurately when there have been long discussions on the web about that topic. When an LLM can pattern-match on whole discussions, it becomes a next-level search engine.

Next, I hope we can somehow get LLMs to distinguish between reliable and less-reliable results.

littlestymaar• 8 months ago

Found somewhere on the internet a few days ago: LLMs are Dunning-Kruger as a service.

Edit: it was https://christianheilmann.com/2025/10/30/ai-is-dunning-kruge...

AaronAPU• 8 months ago

LLMs basically act as defense attorneys for all your dumbest ideas. It is very easy to assume their confidence in you is justified, especially if you already lean narcissistic.

You now see threads on X of famous people using Grok to explain how smart their ideas are. But there’s a problem: You can literally get it to do that with every single dumb idea.

pklausler2• 8 months ago

LLMs, kind of like Bill Bryson's books, are great at presenting "information" that seems completely plausible, authoritative, and convincing to the reader. But when you actually do know the truth about a subject, you realize how completely full of crap they too often are. And somehow after being given a patently counterfactual response to one query, we just blindly continue to take their responses to other queries as having value.

niccl• 8 months ago

But when you actually do know the truth about a subject, you realize how completely full of crap they too often are

The Gell-Mann Amnesia Effect https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect

serial_dev• 8 months ago

Similar to (same as?) Gell-Mann amnesia effect.

obelos• 8 months ago

I've most frequently heard this referred to as “Gell-Mann Amnesia,” and yes, LLMs are fertile ground to find it.

rockostrich3• 8 months ago

At the moment, I find them to be the perfect tool to get started with learning about something. I don't expect it to tell me everything I need to know or to even be right, but if I ask ChatGPT or another LLM a question about a subject I'm not familiar with then it will at least use a bunch of terminology that I didn't have in my vocabulary before starting.

For example, I just bought a 1990 Miata and I want to install a couple of rocker switches in the dash to individually control the pop-up headlights. I have enough circuits knowledge to safely change outlets and light switches, but I didn't know about relays. I asked ChatGPT how to add these switches and it immediately mentioned buying DPDT switches and tying in the OEM relay into a SPDT relay. It may have gotten the actual circuit diagram completely wrong, but now I know exactly what to read up on.

kaydub• 8 months ago

Yeah, it's definitely been terrific for figuring out terminology or "the right word" to use for things.

brewtide• 8 months ago

Completely not related to any LLM usage, but welcome to the world of NA Miata ownership! I think you'll find that with just general maintenance it'll treat you very well -- My '91 is the most reliable car in the drive, and by far the most whimsical. (I just got back from a Miata errand trip in the pouring rain -- Why did I drive the Miata? Winter is very soon, and it gets put away for ~3 ish months -- so at this time of year, every possible trip is a miata trip!)

goalieca• 8 months ago

Now let me ask you the more fundamental question.. did this do you any better than if you had searched a youtube video or some other source? Would this be video from 2016 be relevant? This may not be the right video but my approach for DIY in the last 10-20 years was to hit youtube up. https://www.youtube.com/watch?v=77q9KtjnNTU

I'm trying to gauge whether LLMs are truly expanding our capabilities in a fundamental way or are really just another way to search for answers without going to google or a library.

j-bos• 8 months ago

like Bill Bryson's books, are great at presenting "information" that seems completely plausible, authoritative, and convincing to the reader. But when you actually do know the truth about a subject, you realize how completely full of crap

Wow, I have a couple Bill Bryson books on my reading list, can you share some examples of that?

cynicalkane• 8 months ago

I read this good breakdown on 'The Mother Tongue' on everything2 sometime ago: https://everything2.com/title/The+Mother+Tongue%253A+English...

lexicality• 8 months ago

Honestly I wouldn't worry about it. He's a wonderful writer, the problem is that he doesn't let reality get in the way of a good story. Just classify them with the rest of the fiction-non-fiction books and enjoy the journey. If you ever find yourself asking "wow is that true?" then it probably isn't.

jb1991• 8 months ago

I think it is truly hilarious that you brought Bill Bryson into this discussion.

Panoramix• 8 months ago

I don't quite disagree but this comparison is typically unfair, because when you really know about a subject you tend to ask way more difficult questions than about other subjects, so of course the LLMs are gonna struggle more. If you ask really basic questions they will regurgitate well known bachelor-level knowledge and look good. What do I know about biology anyway? about silos for grain storage? any passable answer is enough to wow me on those topics. But on the topics I really know about, I never ask the basics.

pklausler2• 8 months ago

It's a valuable but scary experiment to query an LLM on basic subject matter in a field that you know a lot about. Ask those basic questions first.

beng-nl• 8 months ago

geLLMan amnesia

ivape• 8 months ago

Can you give me examples of topics that you know about that LLMs don’t know about?

austin-cheney• 8 months ago

I read your comment and just see the typical web developer.

I can’t count how many times developers have tried to school me with their expert wisdom. It’s typically garbage, complete Dunning-Kruger. The causes are two-fold.

On one hand most of these people’s capabilities are an inch wide. Maybe they are really good about JSX, but you take that away and wisdom becomes empty hostility. I don’t want anything to do with JSX or React, so it that’s all you got you are probably just blowing smoke.

The other cause is no experience at all. For example somebody might think they are super knowledgeable on WebSockets because they used a package off NPM. They have no idea how it really works, can’t understand RFC6455 even with Cliff Notes, and can’t write original code.

If you want to be an expert at least start with your own implementation of the thing you want to be an expert about, but most of the people doing that work can’t program.

I can’t help but wonder if the DK coming out of LLMs is really any worse.

shagie• 8 months ago

I was curious about exploring the motivations of a character (specifically Linter in The State of the Art) and asked a question to start off with (and bring existing understanding into the context) with another character (Diziet Sma)... and ChatGPT got things wrong...

The chat is https://chatgpt.com/share/691266fa-c76c-8011-876c-027206abd2... if one is curious. I continued a bit to see what else it got right and wrong.

The thing is if you don't know the story or the books mentioned... it's perfectly plausible that what was written is correct. And while a good bit of it is... maybe; that it got material facts wrong makes it "if it's working from that, then nothing it produces is based on the correct information."

I've known that ChatGPT is full of crap (and experienced in other chats).

It can be a good tool to augment some capacities - but the exploration of ideas based on facts and reality are often (at best) flawed and if one is to try to build upon those flaws and add in one's own misconceptions, then its output is even more questionable.

hyperbolablabla• 8 months ago

Indeed, a couple years ago I tried to use chatgpt to clarify some confusing parts of geometric algebra to me, and it confidently told me completely useless information which I had to blindly trust. Fast forward a couple years, I've read more on the subject from trusted sources, and realised everything it was telling me was pure BS.

seattle_spring• 8 months ago

LLMs, kind of like Bill Bryson's books

I wonder if maybe Malcolm Gladwell would be a more apt comparison?

jonahx• 8 months ago

And somehow after being given a patently counterfactual response to one query, we just blindly continue to take their responses to other queries as having value.

They have value because they are very fast, concise, and right "often enough."

People used to make your criticism about Wikipedia (and occasionally still do).

All of the following are true:

1. They regularly make errors

2. They require more caution than most people have to use effectively.

3. They have tremendous value.

bob1029• 8 months ago

I find the biggest crime with LLMs to be the size of the problems we feed them.

Every time I start getting lazy and asking ChatGPT things like "write me a singleton that tracks progression for XYZ in a unity project", I wind up with a big hole where some deeper understanding of my problem should be. A better approach is to prompt it like "Show me a few ways to persist progression-like data in a unity project. Compare and contrast them".

Having an LLM development policy where you ~blindly accept a solution simply because it works is like an HOV lane to hell. It is very tempting to do this when you are tired or in a rush. I do it all the time.

nlawalker• 8 months ago

It all depends on what you do with it - I see the first prompt just as a slightly different starting place than the second one.

cluoma• 8 months ago

I used to get this same feeling during lectures in uni. Often the information was presented well and, along with some clear examples, everything seemed to make perfect sense.

It wasn't until working through practice problems later, on my own, did it become clear how much detail I was missing.

Swizec• 8 months ago

It wasn't until working through practice problems later, on my own, did it become clear how much detail I was missing.

This is a common problem in learning. Recognition is easier than recall and smoothness is confused for understanding.

You actually need to struggle with the concepts a bit to learn effectively. Without the struggle it feels more effective, but is not.

bamboozled• 8 months ago

Detail missing and being a confidently wrong are two different things though ?

Edit: Claude told me the other day told me my entire building might have to be demolished due to a slightly bow in my newly poured stem wall, I uploaded a photo etc and it was liked, “yes this is a serious structural issue blah blah blah” , the inspector came to look at it and literally laughed that I was worried about it.

elgenie• 8 months ago

Now consider what's happening to the learning process of the (rather large) subset of current college students choosing to replace that struggle for detailed understanding with LLM queries.

brookst3• 8 months ago

It’s the biggest crisis since math students started using graphing calculators.

michaelbuckbee• 8 months ago

I've heard experts comment on this from the other side, that they'll give a quick layperson's soundbite about their subject of expertise that doesn't defensibly lay out all the possible exceptions and edge cases and weirdness for reasons of time and audience interest and then they'll be inundated with comments calling them a liar and accused of falsifying things or not actually understanding the subject.

avs733• 8 months ago

you are not the only one. There was a paper covering this exact topic in the Proceedings of the National Academy of Sciences a few years back [0].

Passive learning (lecture) scored better on:

* Student Enjoyment

* Feeling of Learning

* Instructor effectiveness

* I wish all my courses where taught this way

Active Learning (i.e., not lecture) scored better on:

* Actual learning

The differences are not small.

[0] https://www.pnas.org/doi/10.1073/pnas.1821936116

malshe• 8 months ago

I suspect a majority of my students this semester used LLMs to complete homework assignments. It is really depressing. I spent hours making these assignments and all they probably did was to copy and paste them into ChatGPT. The worst part is when they write to me asking for help, sharing their code, and I can see it was written by LLMs. The errors are mostly there because occasionally the assignments refer to something we did in the class. Without that context LLMs make assumptions and the code fails to generate the exact output. So now I am fixing the part of the code that some of my students didn't bother to write themselves.

Edit: Added "I suspect" in the beginning as I can't prove it.

WingedBadger• 8 months ago

Just going by the last 2 years of university teaching (energy focused computer science in germany), I feel like LLMs have already had a devastating effect. There has been a large influx of students who seemingly got through their entire Bachelors degree with nothing but ChatGPT. The university is slow to adapt and ill equipped to deal with this.

This is absolutely killing my enjoyment of teaching. There is nothing more disheartening than carefully preparing materials for people to grasp concepts I find extremely interesting, just for them to hand in ChatGPT generated slop and not understanding anything at all. In stark contrast, just a couple of years prior I would have quite rewarding projects and discussions with students. I also refuse to give detailed feedback on such "solutions" anymore because the asymmetry in student effort and my effort is just completely unreasonable.

This development is something very different from the often quipped "graphing calculator in maths education". For a graphing calculator you still need to know the mathematical foundations to input the correct things to get the correct results. LLMs are mostly used by just pasting in the exercise of the day.

This is not to say LLMs can't be a useful tool for learning. They absolutely can. But that is not how the majority of students uses them... to their own detriment and the detriment of those trying to teach them.

If universities don't adapt to this quickly, then the already weak signal of "university degree implies some amount of competence" will be entirely lost.

chmod775• 8 months ago

Why are you fixing their code? You're just doing the same thing as the LLM you're complaining about.

Also, as someone who attended university in Germany, the mental image of a professor helping undergrads with homework already seems strange if not funny to me. That is... at least I hope they're undergrads, because if people managed to get any sort of CS degree while having to rely on a LLM to code I might be sick.

lamontcg• 8 months ago

I'm pretty well "on the spectrum" and people glazing me in real life produce suspicion and discomfort rather than any good feelings.

I don't have a problem just ignoring all the LLM glazing, although I'd really like the ability to turn it off.

The fact that they've all been trained to do it, because so many of the "normies" fall for it, is kind of an indictment in my eyes. Bit of a mirror held up to society.

You should probably be worried about how fake flattery works so well in society, and how this enables sociopaths and narcissists to flourish and control everything.

This LLM problem is just a symptom.

jumploops2• 8 months ago

I recall trying to use GPT-4 to plan a trip through the PNW in ~Spring of 2023.

It presented a reasonable agenda, however 80% of the rockhounding spots were completely made up!

Over time, and as LLMs have gotten less sycophantic, I’ve found myself trusting them a bit more (a dangerous and slippery slope).

With that said, GPT-4o in particular, seemed to rank user satisfaction above truth.

I’ve found that GPT-5 Pro is currently the best at pushing back against silly ideas, and does a decent job of informing me that my questions could be better (:

dfxm12• 8 months ago

As always, trust, but verify! Google maps lists "made up" places or outdated info. AI isn't scouting these locations physically...

Of course, at that point, the real question is, what's the value difference (taking into account personal, external and social costs) between asking chatgpt and /r/rockhounding (or whatever message boards they frequent)? At least if you start a thread on reddit, you might meet other people in the area with the same hobby, find a spot no one's talked about yet, get expert context and leave a trail for others to find.

jumploops2• 8 months ago

One of the reasons I love rockhounding is that most of the information is _not_ online, but there is quite a bit of print literature from the last century that hasn't seemed to be scanned.

My recommendation for newcomers is to find a local rockhounding club and start there. Some of the places listed in the old books are no longer publicly accessible, so best to tread carefully!

mattcantstop• 8 months ago

I think this is true. It can super charge some bad takes.

But I've had the opposite experience. The average person is never going to read a scientific study, nor invest the time to find out the real details of any topic they are opinionated about other than simply typing a Youtube search and finding a video that is:

- Entertaining - The person has their same biases - the present the information in a short, consumable manner that doesn't require much investment.

In comparison to this dynamic LLMs are wonderful. They can reference scientific data. I have noticed that they do push back on bad takes (very gently) and steer people towards truth.

It's not that I think LLMs are perfect. They are not. But they are infinitely better than the average human at discovering truth.

LeroyRaz10• 8 months ago

You realize that they will gladly hallucinate science...

You should check the papers it claims to reference as see if the claims it makes are actually backed up.

In my experience, it can completely mischaracterize scientific literature. For example, I asked it if a codebase was a faithful implementation of an algorithm described in a CS paper, and is said "no" and then proceeded to list a dozen small changes. Every single change was incorrect. The codebase was in fact a completely faithful implementation.

ksynwa• 8 months ago

I wonder if LLMs need to be this way owing to the role of pseudo-intelligent conversation partners they've been shoehorned into or if it's a deliberate choice of the vendors.

just-another-se• 8 months ago

been thinking about this for a while - how will society progress when everyone has their own version of "yes man" confirming everything they think of?

PeaceTed• 8 months ago

This is a problem if 'everybody' is using it but I suspect there will be a few groups. It will be a 'tortoise and the hare' situation.

The LLM folks (the hare) will get the initial upper hand as it appears as though they are moving far faster than others but with limited or wrong actual results. This could change if we can solve the hallucination issue. Yes, they are in personal echo chambers but that can only get you so far when you hit the real world. It will be painful and messy but it will resolve long term. Worse case we end up with a Dune "do not make machines that think like a person".

The slow group (tortoise) are those that do not actively engadge in these things. Yes, this trying to keep up but using much slower mental faculties. I suspect long term they will do better as the fast group fail to deliver. Again if we do not solve issues of LLMs which is not certain.

So long as there is still the slow group, we probably would not go down the dark path of individual echo chambers. Long term, eventually if you trip over the same mental stumbling block, you learn to not do that any more.

Dilettante_• 8 months ago

How is "don't use LLMs as a source of truth" still news today? The machine does work, it doesn't know anything. Let the sucker fetch websites and write code.

habibur• 8 months ago

I think it's ok. When wikipedia arrived, everyone was up in arms that people are learning from something that's open for anyone to edit.

But it rectified itself.

The same thing happened when Internet arrived. "Don't believe anything you read on the Internet."

I guess the reaction was same when printed media arrived.

But the thing is, things get better over time.

radarsat1• 8 months ago

Ah so nothing bad happening anymore due to people believing what they read on the internet, huh? Interesting take.

Capricorn2481• 8 months ago

Here's a thought - improving AI is a completely different ball game.

otabdeveloper4• 8 months ago

But it rectified itself.

Or did it?

dr_hooo• 8 months ago

The same thing happened when Internet arrived. "Don't believe anything you read on the Internet."

Isn't the saying "Don't believe *everything* you read on the Internet."? Which is quite different (and still holds today).

LeroyRaz10• 8 months ago

I don't think things get better over time. What is your source for that? Here's an article (with sources) describing a massive down trend in literacy and reading comprehension: https://jmarriott.substack.com/p/the-dawn-of-the-post-litera...

In short, college students nowdays have lower reading comprehension than young children in the 1850s. That is not what I would call progress.

Speaking personally, I believe I would potentially have significantly worse critical reasoning abilities if I had grown up using LLMs. It is very clear to me the temptation of using them as an ersatz for engagement and thought.

I think you are perhaps conflating technological progress (yes technology has improved) with demographic progress. Demographic progress is far from monotonically increasing (reading comprehension is newly plummeting, maths scores are dropping in America, science per scientist is stalling compared to 50 years ago, etc...)

galaxyLogic• 8 months ago

LLMs should not be seen as knowledge engines but as confidence engines.

The thing I like best about LLM is when I ask question about some technical problem, and it tells that it is a KNOWN problem. It thus gives me confiidence that I don't need to spend time) to look for solution where there is no good soloution. Just go around it somehow. It let's me know I'm not the only person with this problem. And that way it gives me confidence that I'm not stupid, the problem is a real problem.

As an example I was working with WebStorm and tried to find a way to make the Threads-tab the default tab shown when debugger opens. AI told me there is no way it knows about. Good, problem solved, solved by finding out there is no solution.

adamhartenz• 8 months ago

This is the kind of stuff AI lies about all the time. I can get it to tell me "That is some good insight, and is a known issue..." with things I make up out of thin air.

LeroyRaz10• 8 months ago

Be careful. The models easily hallucinate problems and misdiagnose. For example, I had an issue with some GPU code, and it assured me, with utter conviction that my problem was caused by some subtle race condition ('a known issue') that the model described in great when the real issue was just a trivial typo - no race condition, no subtly or complexity.

inshard• 8 months ago

The author misses the science of emergence. Reductionist views can’t fully explain macro-level capabilities that arise in these systems. Something emerges at higher scales from the possibility space as model sizes grow; they stop being mere “stochastic parrots” or black boxes running simple regressions.

The weights develop their own inherent logic based on how they relate to each other, analogous to how brain waves encode memory at a level higher than individual neuron networks.

Ultimately, the value of AI lies in the imagination of its wielder. The Unknown Unknowns framework is a useful tool for navigating AI effectively (it powerful to help elaborate on Known Unknowns and identify Unknown Unknowns), along with a healthy dose of critical thinking and understanding how reinforcement learning and RLHF work post-pretraining.

flashfaffe2• 8 months ago

Without being to self-centred here but since I have been using LLM heavily I have always challenged the results given

The post seems to propose the following vector:

Idea-> LLm validation -> confidence -> no further checks

My process is more :

Idea-> LLm response -> skeptical reflection -> adversarial prompting -> synthesis

pedalpete• 8 months ago

I quite regularly ask LLMs to take the other side of an argument, or to tell me where something is wrong.

Unfortunately, they don't seem very good at this process, and in some ways seem to defend the previous position.

Does anyone else take this approach and have success with it?

fennec-posix• 8 months ago

This article also touches on why LLMs can be so dangerous for those who are going through a psychotic episode, it will hit you with the "That's a great idea", "You're correct", etc. Which will just further play into someone's delusions, to a point it's directly down a statistical well telling the person what they want to hear. Sadly this has ended in tragedy more than a few times.

OfflineSergio• 8 months ago

I really like to know if other stuff that make things easy would have felt the same to fols who have been doing computer programming for the past 30 years if they were presented to them with a similar speed.

kuil009• 8 months ago

I use LLMs mainly as a mirror for my own thinking, not as a source of authority.

When I explain my ideas to the model during development, I often see flaws or confusion in my own words. This is where I learn the most. The author talks about people who rely on AI for arguments or research. They let the model's smooth, but statistical, language replace their own thinking. Language is naturally uncertain. LLMs just show this uncertainty using statistics. If you understand this, LLMs are no longer a "confidence engine." Instead, they become a tool to fix and improve your thoughts.

A key point is that even if we try hard, we cannot help but react to what the AI says. We must remember that neither AI nor humans are perfect. I believe we should accept AI responses critically and always be skeptical, just like when meeting a stranger.

spacecadet• 8 months ago

Also says people are not using AI for anything meaningful. If you are trying to use AI in any meaningful way you are hyper skeptical of it and always trying to refine your dataset and understand its outputs. Anything less is mindless consumption, no different from any Joe Internet Consumer.

satisfice• 8 months ago

So weird that I have the opposite relationship with LLMs. I find them revolting and have to force myself to use them.

BabbalaGG• 8 months ago

https://www.pnas.org/doi/10.1073/pnas.2518443122

johnsmith1840• 8 months ago

Just like reading one good article is the same.

Before LLMs I was studying distributed AI training/inference. I read hundreds of sources for this blogs, book chapters, papers, reddit posts anything and everything.

If you think you understand a system via a single article or chatgpt session that's a you problem.

LLM is just giving you a weird mixture of the same content you got before with a random chance of it being different every time.

Ask it again, in a different way over and over. Eventually you will see between it's arbtrary lines. It is MASSIVELY faster at performing this task than before.

more_corn2• 8 months ago

“Does lemon juice make me invisible to cameras?”

“Nope, that won’t work. If you’re planning to rob a bank try a mask or some other way of obscuring your face. Some classics are a bandanna for that classic outlaw look, a balaclava (black is the new black!). Retro cult movie style is to use some pantyhose.”

LLMs are steroids for your Dunning-Kruger

Related Stories