Regression to the Mean: on LLMs and the quiet death of the new

https://rruxandra.github.io/regression-to-the-mean.html

by rruxandra_l • 16 days ago

71 99 16 days ago

rruxandra.github.io

QuercusMax2• 16 days ago

This appears to be a basically content-free slideshow with very shallow thinking...

joombaga• 16 days ago

It's this blog's MO

A running transmission of half-formed thoughts, sketched before they cool into language. Pages that perform their ideas instead of explaining them.

QuercusMax2• 16 days ago

So... fluff.

ben_w• 16 days ago

Reads like (and with that weird contrast and non-trivial background, visually styled like) an LLM whose goal is to campaign against LLMs.

(Or am I now like the Dwarfs in The Last Battle, seeing all things as mere simulacra?)

doitLP2• 16 days ago

And obviously written by an AI

bigfishrunning• 16 days ago

I agree with the sentiment, but the flashy website is distracting and obnoxious

utopiah• 16 days ago

OK as prototypist I can safely say "Yes!" and I've been repeating this for a while now. If you want something that is close to what everybody else does, or said, using statistical means make sense. If you are interested by genuine novelty, things on the fringe where the usual process breaks, then it still remains hard.

Anyway, going to read the actual piece but felt I needed this off my chest.

CJefferson2• 16 days ago

This is my biggest concern. Speaking as someone who recently had to read 60 AI generated reports (the whole issue of how much students are using AI is one discussion), it was genuinely soul-destroying reading the same phrases, seem sentence structures, same arguments over and over. Depressed me the whole of the next day.

olsondv2• 16 days ago

Every student is forced to read the same materials and books. Tests are designed to test how close they remember the same correct answers. It’s always been rare that a novel interpretation or idea has come out of a classroom. The modern, structured education has never been designed to generate creative people.

organsnyder• 16 days ago

I hear this sentiment repeated a lot, but I've never seen it to be true in practice. My kids' teachers absolutely do nurture creativity, and I don't think our school district is particularly unique.

jjkaczor• 16 days ago

True - but it did have the benefit of giving the majority of people who passed through it the same benefit of a baseline understanding of "things" - those with aptitude and talent (and time and privilege) could take that further and build upon those fundamentals.

Now however, if people are not even internalizing those fundamentals in order to even re-write things in their own words, using their own "mental model" (perhaps correct, sometimes not) - I fear they won't even develop "mental models" and abstractions...

AbsurdCensor• 16 days ago

Wouldn't that be the same even if AI wasn't involved? By and large, it's the same books, the same training for students, the same hammered in structure, so the output from students would be reasonably similar. No one is coming up with new allegories out of The Scarlet Letter at this point.

consp• 16 days ago

I've had a few hundred reports over the years as a teaching assistant (digital but must be your own) at the algorithmics course of my university. If I saw that LLM generated uniformity as described they'd all have gotten a plagiarism mark. There were many differences in how people describe things and you could easily see if someone understood the subject or not. That is gone with LLMs as I see it now.

CJefferson2• 16 days ago

No, not at all. Students all have different voices. Yes there are similarities but if I read two pieces of work that sound in the same voice I assume there has been cheating — except this year everything is in the same voice.

liendolucas• 16 days ago

Man if I ever had to read 60 AI reports, I'd quit on the spot.

kolinko7• 16 days ago

Why? Just get AI to grafe them and give you the outliers to judge

FatherOfCurses• 16 days ago

Our future: Fifty billion shades of beige.

adverbly• 16 days ago

Good potential for discussion here. I full agree with the underlying premise: This technology CANNOT be allowed to just give us more of the same, but lazily. It HAS TO be an empowering tool. It has to unlock NEW discoveries.

For the purpose of discussion though, this also undersells AIs:

1. They CAN be great tools for novelty + discovery! You just need to ask and explore and put in work. Its not "easy", but it does help.

2. Sometimes "the mean" is what you want. Sometimes I'm not after art. I'm after something efficient and recognizable and easy to maintain.

How this oversells AIs:

1. We are losing the muscle of forced creativity and problem solving. There is a certain kind of learned privilege that comes from facing a problem and having your instinctive reaction be to ask for and expect help from something else rather than to roll up your sleeves or sit back and have a think. If the system incentivizes loss of muscle en-masse, we're gonna lose something beautiful and powerful.

NichoPaolucci• 16 days ago

I agree with this sentiment.

Many folks have touted the "calculator" similarities as an argument, saying it's more of an efficiency gain / productivity enhancer. To me, LLMs are far more involved than this. Now, unknowingly (or knowingly), people are offloading the problem solving portion of small tasks.

- Creative Writing (Claude, make this email sound more professional)

- Coding (Handle this small logic bug for me)

- Note Taking (Generate a summary of this meeting recording)

- Strategy (Set up a roadmap for X project) and many other areas

- Design (Give me a powerpoint for a stakeholder meeting)

- Personal Life (Find a restaurant I can take my wife to for our anniversary)

Many people underestimate how many "simple" tasks required creative problem solving abilities, and we're actively handing more and more of that over to the thinking machine.

Perhaps it's human nature to give this up, and maybe it's in our best interests - but this is the first time I've ever seen people stop thinking for themselves en masse. Interesting times ahead, IMO.

marksully• 16 days ago

I agree with the whole premise, but isn't this a bit hypocritical? I mean, this website looks almost like what Claude outputs on average

zwischenzug3• 16 days ago

This touches on something I've (and many others) have felt throughout my life, not just since the advent of LLMs.

To take a simple example: I grew up with computer games in the '80s where there were no 'physics engines' or frameworks for building games. As a result, each game was an expression of the author's personality somehow. Fast forward to the noughties, games bored me as they mostly looked and felt the same, or maybe felt like 3-5 different games all packaged differently.

Another example: going abroad on holiday in Europe (I'm from London) used to be a relatively wild, vibrant experience, filled with unexpected differences and challenges (not all positive). There were no McDonalds or Starbucks and the shops were filled with unfamiliar products and foods. Now everywhere in Europe feels the same when I visit, especially with smartphone in hand.

And films went from wildly different to one another to what now feels like 'arty' and 'CGI' being the two choices.

This article continues that into the realm of ideas, or idea production. Everywhere you go looks and feels familiar.

Or am I just getting old?

toppy• 16 days ago

Alex Murrell has a great piece on this: https://www.alexmurrell.co.uk/articles/the-age-of-average

lotsofpulp3• 16 days ago

The generic city one makes no sense. Does the author want each city to invent a new physics? Not only are there quite a few different looking buildings in each of the cities, but given the constraints of not have unlimited funds, surely one can understand that many columns of steel, concrete, and glass will look like columns of steel, concrete, and glass from afar.

zemvpferreira• 16 days ago

No, culture and the means of production went global. Bespoke only makes sense when you can’t get an acceptable good at a decent price. That goes for food, building materials and physics engines. Different will only be found in fetishistic disneylands from now on.

sologub• 16 days ago

Getting old for sure, but also globalization of cultures and ideas contribute as well.

olsondv2• 16 days ago

It’s human nature to pattern-match experiences. As the number of experiences grows, more fit into something seen before. So, yes, we’re just getting old.

Schlagbohrer3• 16 days ago

No, global capitalism and franchise agglomeration have resulted in a flattening of experience. Big global cities all around the world look more and more like each other, with the same franchises and extremely similar offerings everywhere. Young people are having a more homogenized, globalized experience as they grow up online. Teenagers around the world watching the same media and existing in the same shared media-space.

q8zd3• 16 days ago

there is also the fact those tools increased the number of offerings but the quality did not improve.

phillipcarter3• 16 days ago

Or am I just getting old?

Perhaps? But I think this is more a case of just not seeking things out.

Music is as vibrant and diverse as ever, but not if you're only looking at the top charts run by the music industry.

Same deal with games, there's more experimentation and interesting concepts in gaming than ever before, but not from the AAA studios.

Now I can't speak for how you vacation, but I've had wonderfully different experiences between Hamburg, Berlin, Vienna, Prague, Rome, Paris, Montpellier, London, Amsterdam, Oslo, and Florence. I just don't go to the starbucks and instead wander around a bit, optionally picking from a few hit destinations if I feel like it. But also, it's not like this was created for nothing: https://www.itchyfeetcomic.com/2018/10/omnimappus-europeus.h...

ux2664783• 16 days ago

Same deal with games, there's more experimentation and interesting concepts in gaming than ever before, but not from the AAA studios.

I just wanted to add on to this, I wouldn't really classify modern games as "lower quality" than those of the 80s. I'm really not a fan of AAA games, I think the last one I played was Elden Ring, but I would never suggest that they're actively low quality. Uncompelling? Absolutely. But I also have spent a lot of time playing games from the 80s. Silver and golden age CRPGs, random simulators, DOS games that catch my eye. "Quality" isn't the first thing that jumps to my mind. Often they're ugly, terribly balanced, buggy, rife with all sorts of issues in any category you can think of. Games have come a very, very long way. 2400 AD (1988) and Champions of Krynn (1990) are relatively speaking highly polished masterpieces. They're still kusoge, honestly. I have very little experience with the consoles of that era, because pretty much nothing I see even remotely catches my eye.

zwischenzug3• 16 days ago

On cities, I guess my point is that I didn't have to seek out new experiences. Simply going there guaranteed new experiences. I spent a lot of time in Vienna growing up and everything was different. I literally couldn't have a familiar experience even if I'd wanted to. In fact, I remember having to spend lots of money and travelling a long way to find a specialised shop to get a familiar food item (baked beans) to relieve the homesickness.

I don't think people under a certain age can really appreciate how different it was.

coldtea3• 16 days ago

Perhaps? But I think this is more a case of just not seeking things out

"If you went for deep dives and looked at fringe stuff there's still variety" is totally compatible with "the world is way more honogenized".

There have always been fringe unique voices, but that's beside the point. The world is not defined by the long tail, but by the top 20% most listened/visited areas/games played/movies watched/etc, that all now look the same.

Music is as vibrant and diverse as ever, but not if you're only looking at the top charts run by the music industry"

So it's not "as vibrant and diverse as ever" then, since the same "top charts" were much more varied in the past, and they reflect what most people listen to, not just what some music nerds seek.

Same deal with games, there's more experimentation and interesting concepts in gaming than ever before, but not from the AAA studios.

"There is more experimentation and interesting concepts in gaming than ever before" just not in the games most people play. Well, used to be ALSO in the games most people play.

Now I can't speak for how you vacation, but I've had wonderfully different experiences between Hamburg, Berlin, Vienna, Prague, Rome, Paris, Montpellier, London, Amsterdam, Oslo, and Florence. I just don't go to the starbucks and instead wander around a bit, optionally picking from a few hit destinations if I feel like it.

Yes, if you go out of your way you can find some places in Hamburg, Berlin, Vienna, Prague, Rome, Paris, Montpellier, London, Amsterdam, Oslo, and Florence that are not all alike. Meanwhile, you could go anywhere in those cities 30 years ago and you'd find uniquely different stuff, whereas now all major streets and areas have the same globo cafe and globo shops.

rapind• 16 days ago

It's crazy to hear someone think games in the 80s were more creative than they are today. That's taking nostalgia to a whole new level! Indy games today are amazing. When you lower barriers the ratio of good to bad probably tilts more towards bad, but the absolute number of good still increases.

There's going to be reams of AI slop (already is), but I bet the amount of amazing games will also (more slowly) increase due to AI tools. The trick is in how well we can filter.

I think we're in the early stages and being overwhelmed by low quality production. We'll find ways to filter, and find some real bangers.

amelius• 16 days ago

Or how about corporate memphis, rounded corners, etc.

xpct• 16 days ago

I'll stick to games and movies, as I believe both have been moving in a similar direction, becoming more of an object to be consumed, rather than to be experienced. I've thought about this in two ways: it's either that (a) when fields are fresh, creators explore orthogonal concepts and fit to what performs best relatively quickly, or (b) the available idea space just isn't that large by itself, and novelty wears off after you watch some number of movies.

Both games and movies are predictable in the sense that we know what to expect, and they have been largely standardized. Games have common keybinding schemes, as well as user experience mechanics: how jumping feels, when we expect to autosave, what the UI/minimap symbols mean, etc. When it comes to movies, I find myself no longer turning away from the screen before gruesome scenes, because I expect in advance that they won't show it, depending on the mood of the movie. I also find that you can often predict which dialogue lines were meant as foreshadowing for a plot twist coming later. This standardization is intentional in the sense that people are more likely to consume something they are familiar with, and more likely to enjoy it if they can passively engage with it.

It's common nowadays to pay $20 for a game, play it for a few hours, and forget about it. Or, turn on a random Netflix show on the TV to pass time in the evening. Quite likely that a month later you won't reminisce about either of these experiences, but you probably didn't have high expectations either way. I think 'consuming' a travel trip is similar in the sense that it has very familiar tropes no matter where you go, but more implicitly resulting from market forces rather than intentional design from a creator.

mghackerlady• 16 days ago

For this reason, I'll always love indie games. Anyone who finds themselves bored with moder AAA gaming should really go play some of the 2000s and 2010s indie darlings. Here, I'll even give you a list of games I've been playing

The Binding of Isaac: Rebirth, Bit. Trip, Cave Story, Crypt of the Necrodancer, Cuphead, Downwell, Fez, Hollow Knight, Limbo, Octodad: Dadliest Catch, Papers, Please, Proteus, Risk of Rain, Rogue Legacy, Shovel Knight, Spelunky, Stardew Valley, Super Meat Boy, Terraria, VA-11 Hall-A, VVVVVV

Can you tell I own a PS Vita?

coldtea3• 16 days ago

Or am I just getting old?

Obviously not, since 20-somethings also dislike that and express the same things openly, which they didn't in the past.

Also obviously not, since one can just see it by comparing e.g. 10 pictures of NY or Chicago or LA vs 10 pictures of Berlin or London or Barcelona cafes from 1980s and from 2020s, and the latter all look alike. It's also been studied (even academically) and has several names: airport aesthetic, international style, placelessness, global monoculture, airspace aesthetic (air as in airbnb), etc.

zwischenzug3• 15 days ago

"placelessness" was coined the year after I was born! So if it's not 'just age', then it's been going on a long time.

dsmurrell• 16 days ago

I love this article... but is AI correcting you worse than being burned at the stake? :)

AgentMatt• 16 days ago

I agree with the basic premise: that when using LLMs, they will tend towards some mean when resolving ambiguity.

There's a very interesting opportunity here for a deeper investigation.

- What does "regression to the mean" actually mean in practice when the LLM is conditioned on a possibly large amount of context?

- How does this perceived regression to the mean affect the result in different applications? When implementing code, it may show up as keeping it simple, hence easily understandable, "nonclever". When writing documentation, it may show up as simple language, short sentences, etc. supporting the intent of communicating with little friction to a broad audience. When brainstorming product ideas, it may show up as regurgitating old and boring ideas, but dressed in fancy language and affirmations that hide the shallowness of the content.

- What can be done to alter this behavior? Now that temperature doesn't seem to be a parameter anymore in new models, how can we steer creativity of the model?

- If the model's creativity is fundamentally limited, is there a way we can use it to support us in the expression of our creativity, leveraging the different strengths of humans and LLMs in a way that the result transcends the limits of either?

Unfortunately, I don't see the article doing that. And, while I know pointing out LLM-isms is often a cheap shot these days, I feel compelled to point out that this article is full of what I perceived as LLM-ism, quite ironic given the premise and the statement ("written off-distribution · on purpose").

E.g.

Trained on the past, it answers in the past tense of thought. Not what is true. What is typical.
We converge — not on what is right, but on what is average.
Not the answer it was sure of — the one it would not stop correcting

ux2664783• 16 days ago

LLMs might be the least interesting of the statistical models for creative purposes, they're kind of nasty to work with. Creating low rank adaptations is slow and expensive because the models are so fat, and meddling with the inference flow is a much more explosive game of cat and mouse. You can tell Flux where colors, shapes, textures, etc. should live on a canvas. Trying to wrangle an LLM into "spatially/temporally" arranging text in accordance with a writing style is a nightmare and a half. It's complicated, I can't really explain it well. Probably because I still haven't been able to put in the time to figure out the "grain" of pure text transformers. I can tell you that they're very hard to work with, though.

The simple fact of the matter is that these things are best viewed as a computer implementing a continuous form of computation. If you can understand the following words: "the function between function 1 and function 2 is function 1.5" and can imagine that this is a process with "infinite" descent, where you can continue to pull functions halfway between other functions trivially, that's a pretty great mental model to have. To that end, using them is like operating a big, complicated radio setup. Or a huge collection of synthesizers and filters. You're essentially tuning in to a structure. It can't help you figure out what structure you should reach for, or what you should do with that structure.

The creativity doesn't come from the tool itself. If you were only capable of having a mediocre body of work before, they aren't going to help you after. There is no removing human brilliance from the equation. AI is immensely exciting for what it promises, there is something genuinely new and interesting here, but it's not a crutch for the untalented. That's how it's getting sold, and it's having a massive disservice done to it. What we have is a genuinely new space to pioneer in. The interesting stuff and compelling art isn't going to be found at the end of a however-many-word positive prompt, but a symphony of total control over the model itself. Drawing 50 original works for the purpose of fine-tuning a diffusion model for a single project, and only as one particular component of the fully customized inference workflow, which will be scrapped when the project is completed. That's what artistic usage of statistical models looks like. The construction of a specific program for a specific purpose. Anything else is fantasy. The model can't give you that purpose.

queenkjuul• 16 days ago

It's got LLM fonts and styles too

Alien1Being• 16 days ago

The website hurts my eyes...

isgb• 16 days ago

I agree with the point but it would've been nicer if it wasn't written by an AI

ux2664783• 16 days ago

But ask it anything and it returns the most probable continuation — the center of mass of everything already written. Trained on the past, it answers in the past tense of thought. Not what is true. What is typical.

The problem is that this is a contradiction, and a pretty common misunderstanding. When we talk about something being probable, we're talking about what we don't know. When you extrapolate, you're saying something about the unknown, you're creating something new. While you can extrapolate into the past or the future, in the way this line is talking, it's answering in a future tense. I think even worse is that beneath this it says

returned at the speed of certainty

At that point we're no longer talking about probabilities!

The problem isn't that these machines aren't capable of making new things. The whole of their mathematical grounding is in the creation of the unknown from the known. The problem is precisely that they are sold as miracle cures where they can produce great results for little effort. The law of "you get out of it what you put into it" still holds true. Undirected, uninspired usage of these statistical models gets you mediocre at-best results. Without an understanding of the underlying theory and mechanisms of how these models work (it's not just transformers, but any statistical model), driving the whole of the inference chain with maximal control as one might Max/MSP, as well as mastery over the target domain, you will effectively achieve nothing but "slop".

Of course, there's a whole other discussion here, which is that this site seems to be victim to the same grave ignorance that has caused the supposed "crisis of newness" within the arts (which has been talked about for much longer than these models have existed). That's a whole other can of worms, but essentially it's bunk. In modernity we can point to the last century of unending artistic innovation, and panic that this is slowing down, that this is the end of history. In truth, that century is anomalous. It's the most anomalous we've ever recorded, where real material changes were reacted to in real time. The innovations of modernism weren't born because of pretense to being original. It was wholly derived from the changes happening in reality irrespective of the arts, as a result of the industrial revolution, and later the information revolution. The norm in history is centuries of very slow refinement, barely perceptible on the timeline of a generation. Tiny little incremental changes stacked up over a long period of time. Bombastic, revolutionary artistic progress is the anomaly. An unending cacophony of that progress has happened exactly once in the entire history of humanity, as far as we can tell. There is a stupid expectation that the 20th century's breakneck pace was going to last forever. Obviously it wasn't. It was never a sustainable momentum, statistical models or not. People are still in the mindset it's the norm. The languishing over creative bankruptcy is simply the death of this delusional fantasy.

throwaway13337• 16 days ago

This is both true and not.

It's true that in a project, a novel idea undeclared as such will be shaved off quietly by an llm. You really need to be explicit about wanting to keep it.

You will get pushed into the mean.

However, I'd say 90% of making something (that is useful) is repeating the old thing. We stand on the shoulders of giants. Or at least we should. Getting there can be difficult for most of us.

I say this as someone who chronically re-invents things. I then later get stuck and find someone already thought through my problem and solved it better.

I don't believe being unique in all the ways is useful. You need to be unique in the important ways and not unique in the other places.

There's also a cultural coherence angle that (my) unique things often fail at. Stuff has to look like other stuff enough for people to understand intuitively what it is and how it works. Here the mean is your friend.

I am able to explore more unique spaces because I no longer deal with the minutia of getting the things that should be the same correct. So paradoxically, this has made my output more unique.

marcosdumay• 16 days ago

However, I'd say 90% of making something (that is useful) is repeating the old thing.

The entire point of software engineering is to make repetition unnecessary so people can focus on the new.

LLMs are pushing the people that use them into the worst possible set of practices.

terabytest• 16 days ago

We truly are reaching the end times when an article criticizing the use of LLM is in itself pure AI slop.

kolinko7• 16 days ago

It’s also ironic that the slop article complaining about the lack of originality has very little intellectual originality itself.

krona5• 16 days ago

The output of a GPT is an interpolation (an estimation of new data points inside the range of known data) rather than extrapolation (estimations outside that range).

99% of the time we don't need a true intellectual breakthrough to get the job done, and often 'new ideas' are simply riffs on or blends of old ones, like fashion or music genres.

The worry to me, however, is that if society comes to rely on this form of 'AI' then eventually the model collapse bleeds into academia (e.g. grant proposals reviewed by AI?) causing a kind of incremental sociocognitive atrophy. Everything becomes a reaffirmation of the status quo.

That being said I think people said something similar about electronic calculators (that if you couldn't do long division by hand then you'd be too incompetent for higher-level calculus.)

pbasista• 16 days ago

if you couldn't do long division by hand

But the people studying math and the related fields are able to do division by hand on paper. They are just slow when doing it.

I believe that the calculator was meant to solve the slowness problem rather than eliminate the need to fundamentally understand division.

orbital-decay3• 16 days ago

The output of a GPT is an interpolation (an estimation of new data points inside the range of known data) rather than extrapolation (estimations outside that range).

That's a common meme but it's the opposite of true. Everything big models, not just transformers, mathematically do is extrapolation in the feature space, almost never interpolation. They're perfectly able of combining the ideas, although of course this ability declines once they're off the distribution, just like in humans. The model is creative and its output is transformative, however it only makes sense if you define creativity in a pure manner, based on novelty for itself.

However most people use entirely different definitions of creativity, something like "surprise me in a way that still makes sense to me". This includes "me", a side observer, and depends on what the side observer considers novel. Hence the main reason for the lack of "creativity" in big models is not some hand-waved "regression to the mean" or "interpolation", but the fact that they're still insufficiently intelligent compared to a human. Current models simply don't have enough fidelity to understand humans and think as deep, that's why humans think their output isn't sufficiently novel for them! Think about this, insanely smart aliens would probably dismiss humans as non-creative too.

The contributing factor is also the lack of semantic diversity in current models. Also known as mode collapse, but the name is a bit of misnomer and describes a technicality, not the resulting phenomenon. This indirectly affects creativity as it's usually understood, because the models generate and repeat the same thing in response to the same thing, which is the opposite of novel in layman's definition. That's part of where the slop comes from. Mode collapse has many causes, e.g. post-training with current algorithms. It's likely fixable but AI shops show little to no interest in studying and fixing it.

aerodexis3• 16 days ago

so you're saying the difference b/w extrapolation and interpolation is subjective unless the difference is defined tautologically?

krona5• 16 days ago

That's a common meme but it's the opposite of true.

But you're restating what I just wrote - We're training a status quo machine and the probability of anything outside that distribution rapidly drops to zero.

aerodexis3• 16 days ago

The more I read these think-pieces on AI the clearer it is to see that they have very little to do with LLM technology, and are really just talking about the collapse of the set of values constructed during the enlightenment. The sociocognitive atrophy that you describe is inherent to the idea that newness and progress are the ultimate goal that we should organize society around. The problem is that newness and progress can only be defined relative to a status-quo, and hence are incoherent goals in themselves. When pursued far enough, they become their own strange, monstrous status-quo that betrays the original intentions of the people who pursued these values. Hence "we mistake the flattening for progress". This was the case before AI, it's just that AI makes it much harder to ignore, and in many ways encapsulates the problem.

The point w/ electronic calculators is the same point made by Plato regarding books. It used to be easy to laugh off these concerns, not so much today. Imo, this is the real progress: people are now asking meaty questions regarding the ultimate human purpose of books, calculators and technology.

da-x• 16 days ago

Noticed that most of the comments here are despairing.

I think that perhaps there's a bit of hope, that by the forces of the market, the value of human distinctiveness will rise in comparison to whatever is the generated mean. This is what I am looking into.

lotsofpulp3• 16 days ago

Markets only function if sellers (and buyers) have accurate and transparent information about prices and the goods and services. There is no force of the market if it takes too much effort to distinguish what is and isn’t human.

kidbomb• 16 days ago

For the specialists: how much of temperature settings would help on the regression to the mean? Doe

lmeyerov2• 16 days ago

I've been calling this Software Collapse

It's the same problem that AI faces of Model Collapse: AIs that train on the internet ultimately just end up training on one another, stop moving forward, and end up as identical polished versions of one another

I now think of it as a Dr. Jekyll/ Mr. Hyde situation for software projects:

- Dr. Jekyll: For makers, the only limit is your imagination, architectural guidance, and token budget. Time to build!

- Mr. Hyde: For projects to get off the treadmill of having to copy others to maintain you position, you need to redefine how the project works and provides unique value. Features and quality are no longer the answer. Time to fight!

kolinko7• 16 days ago

That is not true - a model trained in the internet can both build verifiers to remove false/poor quality data from the next training, and build synthetic datasets that will supplement its training.

Similar to a human that wants to learn something and invents exercises to practice.

1-2 years ago it was a theory, but new models are trained, successfully, on synthetic datasets.

lmeyerov2• 16 days ago

This feels like a bit of a semantic debate, but maybe a few useful perspectives, as interpret current bespoke work as not so rosy wrt collapse:

- Synthetic datasets are typically human-steered today, which points to model collapse wrt learning from the internet. (Edit: or even simpler, model x data tapped out for cost/benefit even before collapse.) I don't think standard practice is (yet) AI looking at the internet and deciding to build its own gyms to go further. When it does, model collapse may happen again, and be even more expensive

- Distillation attacks are getting interesting here. There seem to be 2 kinds: intentionally querying other models, and maybe not so intentionally, learning from reasoning traces going through shared routers, esp. coding ones

- A lot of neolabs are trying to go where the big labs might not look as directly to avoid being squashed, which suggests they aren't ready to bet on being smarter, and that means the general AI is more about collapse / $

break_the_bank• 16 days ago

impossible to read through this, ironically the entire page feels ai generated.

beyonddream• 16 days ago

The recent breakthrough of llm’s solving major open problem’s in math is a direct contradiction to the article.

But, there is some truth to the article and perhaps it is more true in chat based interactions. The agentic, hands-off mode might tell a different story.

HarHarVeryFunny• 16 days ago

The recent breakthrough of llm’s solving major open problem’s in math is a direct contradiction to the article.

I disagree. LLMs are essentially trained on all the math we know, as well as - via RL training - how we apply it to solve given types of problems.

It should be obvious that not every known mathematical technique, or sequence of techniques, have yet been tried by humans to solve every unproven conjecture, so necessarily there is some amount of low hanging fruit where an automaton tasked with throwing the kitchen sink at a problem will be successful. This is not the same as creativity.

If you are familiar with the mathematical notion of a closure, then what LLMs are capable of is generating the generative closure of what they were trained on. They can generate output that is novel in the sense of a sequence of chess moves, or mathematical moves, that was not - as a sequence - in the training set, but yet it WAS in the training set as part of the implicit closure of what an LLM would be capable of generating.

kerblang• 16 days ago

I expect that a lot of bad norms that crop up in various programming communities will get tossed out after years of dysfunction, often as a result of outsiders and younger people arriving, and stating the obvious: This doesn't make sense. It's actually nice to know this when dealing with all of the mess in the present. You can't change it now, but eventually someone will.

But yes, LLMs are likely to force permanent conformity.

One could also talk about how language in general shifts with the population, but LLMs are likely to prevent it. One would think anthropologists are already looking into this experimentally...

Schlagbohrer3• 16 days ago

"Offer it something it has never seen, and it doesn't light up. It corrects you. To a system built to predict the expected, the genuinely new is indistinguishable from a mistake.

The pushback is soft, and constant: Did you mean: the familiar thing, offered in place of yours."

Ouch, that's scary to think of.

Also why does it read like its written by ChatGPT?

golly_ned• 16 days ago

Even if something wasn't written by ChatGPT, I'm supposing ChatGPT's influence on writing has been so strong that (1) any typical reader of text on the internet will have ingested a lot of ChatGPT writing habits in their own writing, and (2) that any reader of ChatGPT text is so habituated to those writing habits that even non-ChatGPT generated text appears similar to ChatGPT generated text.

Schlagbohrer3• 14 days ago

I have started doing an XKCD style trolling online occasionally- writing like ChatGPT, but, I'm writing it out myself :-D

(Don't worry, I would never do that here on Hackernews... ;-) )

delichon• 16 days ago

This is only surprising if you see AI as a kind of independent intelligence rather than as a new way to access existing media. If it's the later then of course it rarely goes beyond that training. With that perspective it would be just as unreasonable to expect a book to be different when we re-read it. And whatever stultifying/amplifying effect that the LLM has on creativity, so does the written word.

kolinko7• 16 days ago

This is a recurring sentiment but flawed, I think.

First of all, neural nets do nit return averages per se. They construct space between the points and extrapolate outside of the points. So even if a point was not in their training data, they will be ok, in many situations, to acknowledge it.

Or in other words - LLMs don’t average. They construct world models. A novel thing that fits their world model will be accepted no prob. A thing that doesn’t may still be accepted but with challenges.

The same is true though for humans, including scientists. There is a saying that science moves one grave at a time - because often prev gen of scientists needs to die off for a new idea to take root.

Or in yet other words - even if llms produced averages, an average of a discontinuous set can lie outside of that set. And the set of all human ideas is very much discontinuous.

krona5• 16 days ago

They construct space between the points and extrapolate outside of the points.

They don't. They interpolate between the points on a manifold.

memoriyato32• 16 days ago

so the proof to the unit-distance problem was on the manifold, given it was outputed by a LLM?

is the proof to the Riehmann hypothesis also somewhere on the manifold and we just need to prod the LLM with the right prompt so it locates the point?

kolinko7• 16 days ago

Manifold vs space - yes, I meant manifold, english is not my first language and I missed this word.

The argument stands.

deepsquirrelnet• 16 days ago

I think this is only accurate when no external ideas are used, but I'd like to suggest that nearly all new discovery is built on a combination of old ideas and LLMs are really good at the latter.

If you bring something new to the table, then in my experience, AIs are really good at helping you ground it old ideas. If you want to set it and forget it, then you will get the mean. If you want to do something new, in my experience, they are enablers and not blockers.

doitLP2• 16 days ago

And this was itself written by an LLM…we read so much of this stuff now that I’m worried about is we’re all just going to start writing like this whether we use LLMs or not.

https://mapwriting.substack.com/p/living-subscription-free-i...

memoriyato32• 16 days ago

Ironic that this website has exactly the regression to the mean kind of visual design that LLMs output

MatheusFelipe• 16 days ago

pretty interesting discussion

btbuildem• 16 days ago

It's an absolute regression to the mean. We need to understand the correct place for LLM-driven coding agents et al: they're beige paste extruders. Which is actually a perfect match for 80% of corporate world.

For anything unique, they're a boon, in that they can free you from dealing with the boilerplate, but you must think on your own, must have your own insights, tap into your creativity. If you start to depend on them for ideation, it's slippery slope to the slop bowl.

yalogin• 16 days ago

This is not a new realization, it’s known. By definition llms are great summarizers not inventors. The value is in having the knowledge readily accessible and to cross check and validate against what’s already known - a huge time saver. It’s not going to come with anything new. It need not even, the value is in making things time efficient.

Of course the llm companies justifying huge investments want to frame it as a technology to invent new things and that is where the disconnect is. They are over indexing on the promise but companies are figuring out that the value is in automation and applying/showing value for existing workflows. Claude focused on this aspect and hence so successful

fellowniusmonk• 16 days ago

Discovery, which was hard for so long, is now hard again.

No longer because communication and transmission is expensive but because it is so cheap.

coldtea3• 16 days ago

Trained on the past, it answers in the past tense of thought. Not what is true. What is typical. (...) Offer it something it has never seen, and it doesn't light up. It corrects you.

While correct, the post itself is AI slop.

joshka• 16 days ago

Technically incorrect (the best type of incorrect).

A conditional probability model (LLM) is not necessarily a mean-seeker.

Possibly mode-seeking, rather than mean.

AmazingEveryDay• 16 days ago

Firefox's "Reader View" mode really shines on sites like this!

Regression to the Mean: on LLMs and the quiet death of the new

Related Stories