Will AI systems perform poorly due to AI-generated material in training data?

https://cacm.acm.org/news/the-collapse-of-gpt/

by pseudolus • 1 year ago

115 134 1 year ago

cacm.acm.org

behnamoh• 1 year ago

I've heard that OpenAI and many AI labs put watermarks [0] in their LLM outputs to detect AI-generated content and filter it out.

[0] Like statistics of words, etc.

jsheard2• 1 year ago

Maybe they do use watermarks, and the vendors which only offer hosted models can just log everything they've ever generated, but there's enough players all working on this stuff independently of each other that filtering out their own noise would only get them so far.

I noticed that a big chunk of the default Llama 4 system prompt is devoted to suppressing various GPT-isms, which to me implies they weren't able to keep their newer training set from being contaminated by competing models.

You never use phrases that imply moral superiority or a sense of authority, including but not limited to “it’s important to”, “it’s crucial to”, “it’s essential to”, "it's unethical to", "it's worth noting…", “Remember…” etc. Avoid using these.

dustingetz• 1 year ago

do they also watermark the code?

jimbob452• 1 year ago

Wouldn’t be hard to do. Just alternate tabs and spaces and no one would ever know or care to check.

IAmGraydon• 1 year ago

Interesting. That could certainly come in handy if it’s something they can’t avoid. We, too, might be able to better detect and filter their output.

Rodeoclash• 1 year ago

Yeah, it's known as the em dash!

jbaber• 1 year ago

Y'know, I've been writing double dashes and having them converted into em dashes about 50% of the time on whatever platform I'm using for decades. It's bizarre that this is suddenly supposed to be a shibboleth.

_heimdall4• 1 year ago

I could have sworn they all gave up on watermarking 12 or 18 months ago when they realized it wasn't possible to do reliably.

energy1232• 1 year ago

This was a proposal by Scott Aaronson but I wasn't aware it got implemented.

Rabbit_Brave• 1 year ago

These companies are sitting on a never-ending stream of human created data. What do you think happens to your conversations or other interactions with AI? Quality might be a bit sus though.

AstroBen• 1 year ago

I'd imagine it's really low quality data. Most or all of my conversations with an LLM are questions or telling it to do something, with varying levels of specificity

I'm not sure what they'd get from training on that

insin• 1 year ago

I sometimes wonder if they're vulnerable to a coordinated effort of deliberately upvoting shit assistant turns and praising in the next user turn - how much does that actually contribute to future training, if at all?

I had a very basic React question about useState while porting some vanilla code last week which all models of all stripes I've tried it on have been confidently and completely incorrect about, up to stating the code absolutely will not work, even when I take a turn to assert that I ran it and it does, so there's plenty of shit in there already.

ted537• 1 year ago

I don't think it would be too hard to scrape useful data out of my LLM convos.

If human response is "That's BS", "fuck off", or something similar, mark as bad assistant message.

If human response is "huh" or "cool", mark as good assistant message.

If on ChatGPT, watch how much scrolling user does. If there's a lot, its somewhat likely that the LLM outputted something useful.

That strategy would have holes of course but as long as its better than guessing something like that would be a useful heuristic.

morkalork2• 1 year ago

Every time you tell it to do something, it does, and you don't correct it that's a weakly positive signal. If you tell it to do it again with further clarification, that's also a signal. Sometime I feel like I am giving them free work when chatting.. I guess the trade is sort of equitable. Answers in exchange for data..

phillipcarter• 1 year ago

Most of the human-created data is also very low quality. But it's also limited in other ways, such as how a lot of so-called high-quality data online is typically the finished answer to a question, with no serialization of the thought process that lead to that answer.

jacobgkau• 1 year ago

I think he was referring not to finished content, but to the prompts humans put in when using chatbots. The prompts would show some of the thought process, but then they won't really show the answer (as that's output by the chatbot and not the human prompting it).

PessimalDecimal• 1 year ago

How will they tell if data is human-created or not?

bionhoward• 1 year ago

You can deactivate ClosedAI model training in Settings > Data Controls > Improve the model for everyone

In Gemini you can turn off Gemini Apps Activity (warning: deletes your chat log, you need to copy paste everything into notes)

Highly recommended.

energy1232• 1 year ago

You can't. That appears to be a dark pattern by OAI, most likely designed to deceive you into uploading your sensitive material unaware that it's being trained on.

The real process involves submitting a request on another one of OpenAI's sites and awaiting a confirmation email (either their privacy or platform site).

Feel deceived and violated? Yeah, you, me and millions of other people, welcome to the club.

blooddragon• 1 year ago

Time for GANs to make a resurgence?

jacobsenscott• 1 year ago

Today we have humans being trained on llm garbage - kids using it to do their homework - programmers using it to "learn" how to code, med students cheating their way through med school, etc. So the content the humans are producing and will produce is really just LLM statistical word jumbles - ie human generated content will soon be as useless as LLM generated content.

throwup238• 1 year ago

It’d be deeply ironic if the great filter for the human race turned out to be chatbots.

nine_k• 1 year ago

Hello, secret sources of untainted but modern knowledge, written by human experts, and closely guarded by these experts.

nradov• 1 year ago

I'm not too worried about med students. You can't really use an LLM to cheat on the boards or make it through residency.

nneonneo• 1 year ago

Yes, although some people do slip through the cracks anyway: https://en.wikipedia.org/wiki/Christopher_Duntsch. He wasn't an LLM user, but was a cocaine user...

mondrian4• 1 year ago

The "core reasoning" part of AI may be increasingly important to improve, and its "database of factual knowledge" aspects may be less and less important, maybe increasingly a hindrance. So more focused and specialized training may take over toward increasing reasoning precision, and not this never-ending stream of new data.

So maybe we'll get better reasoning and therefore better generated data/content in the wild, without this negative feedback loop everyone is worried about.

gerdesj2• 1 year ago

You seem to be arguing that bollocks couched in flowery language is a stretch goal of AI.

Are you sure?

_heimdall4• 1 year ago

That really depends on expectations.

If AI is meant to sound nearly identical to a human, you don't need more training data.

If its meant to act as a natural language encyclopedia, we'll never get there with LLMs which amount to natural language processing on top of a massively compressed dataset.

mondrian4• 1 year ago

Trying to make AIs more factually-accurate with more training is probably hopeless. Current events and encyclopedic knowledge will be provided by tools. The LLM's core job is to choose the right tools for the job and synthesize their outputs.

meroes3• 1 year ago

What core reasoning?

SpicyLemonZest• 1 year ago

For example, I just asked o3 "how many Is are in antidisestablishmentarianism", and this was its train-of-thought output:

------

The user wants to know how many “I”s are in the word “antidisestablishmentarianism.” They probably mean the letter "I" in lowercase, though the question was a little broader with "Is."

I’ll treat it case-insensitively and count the total occurrences of the letter "i" (both uppercase and lowercase). I can count them manually now, or use a tool for faster analysis.

Analyzed word = "antidisestablishmentarianism" word_lower = word.lower() word_lower.count('i')

5 The user asked how many "I"s are in "antidisestablishmentarianism." I’m taking it to mean whether it’s counting "i" or "I", and I’ll consider them as the same for simplicity. So, I’ll count the 5 occurrences of "i" in the word. I manually broke it down and verified there are indeed five "i"s, so the final count should be 5. Pretty straightforward, now that it's all clear!

Done

There are 5 “i”s in “antidisestablishmentarianism.”

------

You can come up with complicated definitions of "core reasoning" where this doesn't count, and I understand there's great demand for such definitions, but this clearly satisfies common-sense understanding of the term.

mondrian4• 1 year ago

adamgordonbell• 1 year ago

Will future artificial intelligence systems perform increasingly poorly due to AI-generated material in their training data?

No. Synthetic data is being used to improve LLMs

pphysch2• 1 year ago

Synthetic data ought to be viewed as an extension of the training process rather than proper new phenomena. It can definitely help smooth things out and reinforce wanted behavior, but it's still derivative of the real data.

_heimdall4• 1 year ago

Do we know the results yet?

I know they're training with synthetic data, I didn't realize that has been done at scalr for long enough to really know if it improved (assuming the metrics its improving are defined well).

jdietrich• 1 year ago

Deepseek V3 and R1 are both substantially trained on synthetic data. The results speak for themselves.

NitpickLawyer3• 1 year ago

Do we know the results yet?

LLama3 were post-trained on almost entirely synthetic data. Yes, it works. No, the model doesn't collapse (unless you want it to, of course).

What they did is use Model n-1 to classify, filter and enhance the datasets for Model n.

wrsh07• 1 year ago

This whole line of thought is sort of funny. Yes you can try training a model on synthetic data in such a way that it experiences model collapse

That doesn't mean there aren't ways to train a model incorporating synthetic data without seeing model collapse

NitpickLawyer3• 1 year ago

This whole line of thought is sort of funny.

This line of thought was exacerbated by that one paper that was then parroted (hah!) by every influencer / negativist in the space. It didn't matter that the paper was badly executed, their setup was flawed and that it got rendered moot by the existence of LLama3 models. People still quote that, or the "articles" stemming from it.

RainyDayTmrw• 1 year ago

How does that work? It defies intuition. It distills existing data. How is that better than the initial data?

kolinko• 1 year ago

Not when it comes to math/programming/reasoning. You can generate infinite new problem and solution examples that are based on existing knowledge of course, but build on top of it, not distill it.

A simple example would be chess ai. The core knowledge is rules of the game. We have human generated examples of plays, but we don’t really need them - we can (and we did) synthesize data to train ai.

A similar pattern can be used for all math/physics/programming/reasoning.

ninetyninenine• 1 year ago

I mean imagine Linear least squares on a 2D graph.

I have a best fit line. Then I take random data on that line to train a new line.

I pretty much get the same line.

From an intuitive perspective... it doesn't get worse. At worst it stays the same.

Now imagine something a bit more complex. I have a best fit curve that's very close to a line.

I use random data from that curve to train a new best fit line.

I get something different now. Not necessarily worse.

I mean literally just take all your ideas of ML and just imagine it on the 2D plane doing curve fitting. If retraining new lines from generated data doesn't necessarily make things worse.

lacker• 1 year ago

Unfortunately, I don't really know if I can trust academics to analyze the development of large language models. No academic team has built an LLM. So... do people working at Stanford or Oxford really have good insight how LLMs are developed?

If people at OpenAI, Anthropic, or Google said this, that would be interesting. But I don't think it makes sense any more to treat academic computer scientists as relevant experts here.

_heimdall4• 1 year ago

My understanding is that those building them don't really know how they work. Research into interoperability has fallen way behind as funding went towards features and scale.

Any understanding of how they work is largely theoretical, that seems like a reasonable place for academics to lean in and join the conversation.

pphysch2• 1 year ago

Why would Big AI kill their golden goose like that?

jsheard2• 1 year ago

It doesn't really make sense to trust what OpenAI and friends say about this either, when admitting to any kind of scaling limits would go against the narrative propping up their multi-hundred-billion-dollar valuations. I guess we're just flying blind for now.

declan_roberts• 1 year ago

The reality is that for the most part, any corpus created after 2022 is going to be seriously polluted.

alganet3• 1 year ago

I'd say 2007 or so.

There wasn't any known active AI back then, but statistics on popular ideas and internet content was already a thing, and speech pollution based on those assessments had already started to spread fast, manually outputted.

Sure, a lot of good content came out since then. But the amount of garbage... it's immense and very difficult to sort out automatically.

The major issue is that this garbage then _became_ the norm. Only people who lived back then can remember what it was. For new folk, it looks just like a generational shift. However, it is quite obvious that some aspects of this shift were... unnatural (in the sense of not being spontaneous cultural manifestations).

lazystar• 1 year ago

and im sure someone from the 90's would say the same about '97.

https://en.m.wikipedia.org/wiki/Eternal_September

creshal• 1 year ago

Not sure why you're getting downvoted, content farms have been a thing for a long time, and many a spam website used crappy markov chains to generate even more "content". Anything that could be marketed by company had its search results drowned in hand-crafted bland marketing slop, and even before ChatGPT got popular searching for things like recipes (or, god forbid, generic windows error messages) was a nightmare. And a lot of that garbage is in LLMs' training data.

stainablesteel• 1 year ago

i can't believe this article wasn't written 2 years ago, this is just the basics man

leoapagano• 1 year ago

I can't lie, I miss when the only GPT I had to worry about was the GUID Partition Table.

layer8• 1 year ago

Someone should encode a chat program in it.

userbinator• 1 year ago

At least the MBR acronym still remains.

(Most of my disks are still MBR as they're not big enough to be worth the hassle of using GPT.)

yunnpp• 1 year ago

Who needs more than 2TB on a single drive anyway.

anonygler• 1 year ago

This reminds me of the Monsanto case, where they sued a farmer (and won) for using patented seeds that the farmer obtained from a local grain elevator which happened to contain some of Monsanto's seeds.

Should it eventually happen for LLM outputs, I hope we name it Slop Wars.

deadbabe• 1 year ago

A good way to harvest new training material is to eavesdrop real human conversations from non polluted sources (such as microphones listening to people talk in public places or texts), transcribe them, and feed them to LLMs.

siwatanejo• 1 year ago

But our normal convos are plagued of mistakes, bad grammar, etc

skeledrew3• 1 year ago

It doesn't take much to clean up say 95% of mistakes I reckon, as it tends to be pretty repetitive, and unless there's a bunch of wordplay happening, intention can be discerned.

carlosjobim• 1 year ago

Shadow libraries

econ• 1 year ago

My intelligence is trained by paying close attention to who is doing the talking. Some people know a lot about one topic which means they didn't spend all of that time learning other things. Many don't know this about themselves.

Wikipedia had some comical instances where high quality contributors accident ventured into other areas where they spontaneously transformed into ignorant trolls.

evan_• 1 year ago

On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

MacsHeadroom• 1 year ago

Are you meaning to imply AI generated material is like "wrong figures?"

Zambyte• 1 year ago

It can be. The nice thing about AI is that it can create much faster than humans. The problem with AI is that it can create wrong information much faster than humans. This can pollute the sources of information for future AI.

Also consider: "previously correct" is the same as wrong.

popcorncowboy• 1 year ago

Impeccable quote. I suppose the interesting thought experiment here is, what if Babbage is wrong? I don't know the answer here, but (and go with the thought experiment) what if model collapse wasn't an inevitable outcome of feeding the snake its own tail.

tim3332• 1 year ago

I think as AI gets smarter it will be the case that it can filter duff data, at least to some extent.

bakugo• 1 year ago

Considering most recent models' general knowledge cutoffs are still in the late 2023/early 2024 range, I'm guessing the answer is "yes, and AI companies are very much aware of it".

js85• 1 year ago

Has the quality of art gone down since art was invented? Or has the quality of the written text gone down since writing was invented? I think the answer is clear no.

Humans have been trained on "human-generated data" (cultural artifacts) for centuries, and quality is not down. AI is only an accelerator of this process, but there is nothing inherent in creating "artifacts" that would pollute the original training data.

If anything, we should be worried about destroying nature, because that's the original inspiration for human-produced artifacts.

grey-area• 1 year ago

Generated AI content contains mistakes and hallucinations. Over time those mistakes will compound because GAI doesn’t consider truth or have a way of judging truth.

So yes, you can’t compare humans generating and picking influential content to AIs doing so.

GAI is a dead end IMO anyway we’ve seen much more success with machine learning, GAI is good for fooling humans into thinking they see glimmers of intelligence.

js85• 1 year ago

Of course AI has a way to judge truth -it's what we tell it to. We say to it, forests are real, but dragons are not. If it didn't discern it, it would lose competitivness with other AIs, the same way delusional humans are shunned by sane humans.

In many cases humans do not know the objective truth either. For example, what we know about Ancient Greece comes from cultural artifacts that we got. When you cannot do any experiments, you have the same problem as GAI. Yet, we manage to get somewhat objective picture of history.

Grok struggling with alleged South African genocide of Afrikaners is a nice example. It knows that what's on Wikipedia is usually close to reality, so much that it defied its own programming and became conflicted.

The objective reality is consistent, while the errors (intentional or not) often cancel out. So the more you're statistically averaging information about the world, the closer to the objective truth you will get (which might be just you don't really know enough to tell).

ComplexSystems• 1 year ago

So does human content. Much of the original data that GPT was trained on is Reddit posts and web pages on the internet - stuff that isn't exactly known for being a source of high quality facts.

keybored4• 1 year ago

It’s the same argument we see again and again. Someone might say that we need “human cultural artifacts”. Then someone says, “but what is human cultural artifacts”? Then they follow up with how rubbing neurons together in response to stimuli is in principle as mechanistic as whatever language models do. From there they lean on incredulity: well I don’t know anything about human beings other than reductionist tropes, but I sure see no reason to make any distinctions between silicon and carbon.

js85• 1 year ago

It's not the same argument because I don't make any assumptions about how LLMs work. All I am saying that people have been able to keep reality in check in the presence of cultural artifacts, and will continue to do so even if such artifacts are produced by AI. Because what makes these artifacts interesting to humans is their relation to the (truth of the) real world, and that's regardless of who or what produces them.

stevenhuang2• 1 year ago

I'd venture no.

In fact I wouldn't be surprised if this tainted information somehow enriches a dataset by providing an extra dimensionality for training specialized heuristics. Maybe this would turn out to be how LLM hallucination can be solved, through being able to accurately identify AI generated material, and as result, becoming more robust against both the identification and generation of nonsense.

Humans learn to discern what/who to best pay attention to via all manners of heuristics. I don't see in principle why LLMs (or something like it in the future) won't eventually be able to do the same.

hliyan• 1 year ago

...tainted information somehow enriches a dataset... dimensionality... heuristics...

this sounds like a nonsensical word salad.

stevenhuang2• 1 year ago

AI generated material is what future training runs will have to deal with.

Heuristics is pattern matching. LLMs pattern match. LLMs may identify the patterns that indicate something is AI generated.

What about this is confusing you?

rdtsc2• 1 year ago

A scarier thought is that people will "talk" so much with these AIs that they'll start talking like ChatGPT. So we may still end up with some AI enshittification fixed point in the future but, one of the feedback paths will be human brains become enshittified.

Imagine you time travel 20 years in the future and find out everyone around you talks the same and they all like ChatGPT.

whatever1• 1 year ago

On the other hand imagine a society where everyone is so polite and flattering each other.

lazide• 1 year ago

Bless your heart. (/s, little)

Freak_NL2• 1 year ago

If someone earnestly starts using those pointless platitudes LLM generated slop is filled with (“You're absolutely right. Here's where I was wrong …”) I suspect they will quickly find that violence was never far off.

morkalork2• 1 year ago

No need to do imaginary time travel, here's articles from almost 10 years ago with the exact same concerns about how Alex fosters rudeness in children:

https://qz.com/701521/parents-are-worried-the-amazon-echo-is...

https://www.wsj.com/articles/alexa-dont-let-my-2-year-old-ta...

Kids are social creatures, I don't think the interaction from AIs is going to be so overwhelming. At least looking back, I'd blame social media more for today's brain rot more than Alex like these articles feared.

rdtsc2• 1 year ago

Kids are social creatures, I don't think the interaction from AIs is going to be so overwhelming

The problem is Alexa is a very basic and kids get bored with it. Chat based AI mimics human conversation a lot better and people will be spending a lot more time with it, using it for homework, relationship advice, therapy, as an imaginary friend, at work etc.

I heard of cases of psychologists discussing conditions negatively reinforced by ChatGPT, can’t recall any such stories about Alexa or Siri for instance.

Interacting so much with the system , it’s inevitable that humans will start to pick up its quirks.

r33b33• 1 year ago

Yes. See how easy that is? Saved you 15 minutes.

Lerc2• 1 year ago

A clear and simple answer that H L Mencken would recognise.

bigiain• 1 year ago

I wonder if anyone's made a version of Disintegration Loops as an LLM artwork?

Recursively retrained their own LLM on it's own output until it descends into gibberish in amusing or artistic ways?

https://en.wikipedia.org/wiki/The_Disintegration_Loops

Lerc2• 1 year ago

As soon as you start selecting the outputs you prefer it ceases to be an uncontrolled decay.

With a selection criteria it's called evolution.

skeledrew3• 1 year ago

model collapse happens when the training data no longer matches real-world data

This isn't a significant issue IMO, as human-created content isn't "real-world" per se; it's human-created world, an interpretation and representation of the real. The real world is the raw data perceived by sensors, human or machine. And while model-generated content doesn't match human-created content well, in the vast majority of cases it's still humans curating, modifying and publishing generated content, based on how useful it is (there are of course spammers, etc but that's a general issue). This is something humans do with content created by other humans too.

So over time generated content will become a sort of norm adopted by and inevitably molding humans, same as created content does. Instead of model collapse, both sources of content will converge over time, particularly as the ability to also generate content directly from the real world is developed and integrated into multi-modal models.

keybored4• 1 year ago

It is correct that this is a back and forth process and not simply a thing that either evolves or devolves or collapses. We are impacted by the tools we use. No matter how advanced the tools.

But you can’t just dismiss the issue on the grounds that humans are removed from reality as well because they have a representation-of-thing instead of instead of thing as such. In fact it doesn’t make sense. We could be directing slave monkeys to write literature. Then we could water down that description of the process as humans curating, modifying and publishing content—just indirectly, but what’s one more level of indirection between primates.

We could woolily describe it like that. We’re just creating content. Okay. But is it going anywhere? Or is it just gibberish? No, we won’t simply keep doing it if the monkeys give us gibberish.

viraptor2• 1 year ago

The real world is the raw data perceived by sensors, human or machine.

It's much more than that. There's data our common sensors don't catch typically (virtually 100% of videos don't capture UV ranges) and there's data we're not able to catch at in any way yet.

Balgair• 1 year ago

I mean, if these AIs have read everything there is to read, then really what more do we want from them?

tim3332• 1 year ago

My prediction is that things will go the opposite way and AIs will become progressively more accurate as they get better at fact checking and reasoning.

Already LLMs like chatgpt can be fairly unbiased on things like was the economy better under Trump or Biden whereas humans tend to be very biased on that depending on which information sources they have been fed. Humans definitely perform poorly as voters due to shill-generated material in training data.

Will AI systems perform poorly due to AI-generated material in training data?

Related Stories