Bad Actors Are Grooming LLMs to Produce Falsehoods

https://americansunlight.substack.com/cp/168074209

by nsoonhui • 1 year ago

195 231 1 year ago

americansunlight.substack.com

CarRamrod• 1 year ago

It's agendas all the way down

ineedasername• 1 year ago

What happened to all the turtles?

indeyets• 1 year ago

They’re victims of agenda

bsaul• 1 year ago

Seems like the general problem is consistency within the model. To people working in the field : what are the current options explored for solving this problem ?

throwaway290• 1 year ago

1) repeat that people also lie so it is okay for LLM to lie

2) ingest as much VC money and stolen training data as we can

3) profit

disillusioned• 1 year ago

I've been using "off-by-one" errors to describe one of my biggest concerns with LLMs replacing search, or acting as research agents, or functionally being expected to be reliable narrators in general. If you ask ChatGPT when George Washington was born, and it comes back with March 4th, 2017, you'll reject that outright and recognize it's hallucinated a garbage response, presuming you have enough context to have understood who George Washington was in the first place and that your brain hasn't completely succumbed to rot yet.

But if it returns February 20th, 1731... that... man, that sounds close? Is that right? It sounds like it _could_ be right... Isn't Presidents' Day essentially based on Washington's birthday? And _that's_ in February, right? So, yeah, February 20th, 1731. That's probably Washington's birthday.

And so the LLM becomes an arbiter of capital-T Truth and we lose our shared understanding of actual, factual data, and actual, factual history. It'll take less than a generation for the slop factories to poison the well, and while the idea is obviously that you train your models on "known good", pre-slop content, and that you weight those "facts" more heavily, a concerted effort to degrade the Truthfulness of various facts could likely be more successful than we anticipate, and more importantly: dramatically more successful than any layperson can easily understand.

We already saw that with the early Bard Google AI proto-Gemini results, where it was recommending glue as a pizza topping, _with authority_. We've been training ourselves to treat responses from computers (and specifically Google) as if they have authority, we've been eroding our own understanding and capabilities around media literacy, journalism, fact-checking, and what constitutes an actual "fact", and we've had a shared understanding that computers can _calculate_ things with accuracy and fidelity and consistency. All of that becomes confounded with an LLM that could reasonably get to a place where it reports that 2+2=5.

The worst part about the nature of this particular pathway to ruin is that the off-by-one nature of these errors are how they'll infiltrate and bury themselves into some system, insidiously, and below the surface, until days or months or years later when the error results in, I don't know, mega-doses of radiation because of a mis-coded rounding error that some agentic AI got wrong when doing a unit conversion and failed to catch it. We were already making those errors as humans, but as our dependence and faith on LLMs to be "mostly right" increases, and our willingness and motivation to check it for errors dwindles, especially when results "look" right, this will go from being a hypothetical issue to being a practical one extremely quickly and painfully, and probably faster than we can possibly defend against it.

Interesting times ahead, I suppose, in the Chinese-curse sense of the word.

aucisson_masque2• 1 year ago

Maybe, just maybe people will learn they can’t trust everything that’s written online wether it’s done by a bot or even human.

Hell, they might learn that even real life authorities may lies, cheat and not have everyone’s interest in their mind.

Hope for the best, prepare for the worst.

sometimes_all• 1 year ago

At every point, during a knowledge/data search for reaching a particular goal, the onus is _always_ on the person searching to do their best to ensure that the sources they use are accurate, and they do the effort required to ensure that they translate that properly to fit that goal.

The education system I grew up in was not perfect. Teachers were not experts in their field, but would state factual inaccuracies - as you say LLMs do - with authority. Libraries didn't have good books; the ones they had were too old, or too propaganda-driven, or too basic. The students were not too interested in learning, so they rote-learned, copied answers off each other and focussed on results than the learning process. If I had today's LLMs then, I'd have been a lot better off, and would've been able to learn a lot more (assuming that I went through the effort to go through all the sources the LLM cited).

The older you grow, you know that there is no arbiter of T-Truth; you can make someone/something that for yourself, but times change, "actual, factual history" could get proven incorrect, and you will need to update your knowledge stores and beliefs along with it, all the while being ready to be proved incorrect again. This has always been the case, and will continue to be, even with LLMs.

aucisson_masque2• 1 year ago

What is propaganda for one is truth for another, how could LLM tell the difference ?

LLM are not journalist fact checking stuff, they are merely programs that regurgitate what it reads.

The only way to counter that would be to feed your LLM only on « safe » vetoed source but of course it would limit your LLM capacities so it’s not really going to happen.

Dylan16807• 1 year ago

What is propaganda for one is truth for another, how could LLM tell the difference ?

The article isn't even asking for it to tell the difference, just for it to follow its own information about credibility.

rachofsunshine• 1 year ago

What is propaganda for one is truth for another, how could LLM tell the difference?

"How do you discern truth from falsehood" is not a new question, and there are centuries of literature on the answer. Epistemology didn't suddenly stop existing because we have Data(TM) and Machine Learning(TM), because the use of data depends fundamentally on modeling assumptions. I don't mean that in a hard-postmodernist "but can you ever really know anything bro" sense, I mean it in a "out-of-model error is a practical problem" way.

And yeah, sometimes you should just say "nope, this source is doing more harm than good". Most reasonable people do this already - or do you find yourself seriously considering the arguments of every "the end is nigh" sign holder you come across?

EasyMark2• 1 year ago

It's something that can only be nudged in the right direction. Using certain higher quality sources (New York Times, BBC, Wikipedia) and avoiding low quality sources (4chan, fox news, twitter) . I usually take LLM results with a very good dose of suspicion.

dns_snek3• 1 year ago

If actions by these bad actors accelerate the rate at which people lose trust in these systems and lead to the AI bubble popping faster then they have my full support. The entire space is just bad actors complaining about other bad actors while they're collectively ruining the web for everyone, each in their own way.

7bit• 1 year ago

Thats naive. Look at all the tabloids thriving. The kind of people that bad actors target will continue to believe everything it says. They won't lose trust, or magazines like New York Post, the Sun or BILD would already have crossed to exist with their lies and deception. And Russia would not have so many cult members believing the lies they spread.

imiric5• 1 year ago

Before the bubble does pop, which I think is inevitable, there will be many stories like this one, and a lot of people will be scammed, manipulated, and harmed. It might take years until the general consensus is negative about the effects of these tools. All while the wealthy and powerful continue to reap the benefits, while those on slightly lower rungs fight to take their place. And even if the public perception shifts, the power might be so concentrated that it could be impossible to dislodge it without violent means.

What a glorious future we've built.

LilBytes• 1 year ago

The tragic part of fraud is it's not too different to operational health and safety.

The rules and standards we take for granted were built with blood, for fraud? It's built on the path of lost livelihoods and manipulated gold intent.

autoexec5• 1 year ago

It might take years until the general consensus is negative about the effects of these tools.

The only thing I'm seeing offline are people who already think AI is trash, untrustworthy, and harmful, while also occasionally being convenient when the stakes are extremely low (random search results mostly) or as a fun toy ("Look I'm a ghibli character!")

I don't think it'll take long for the masses to sour to AI and the more aggressively it's pushed on them by companies, or the more it negatively impacts their life when someone they depend on and should know better uses it and it screws up the quicker that'll happen.

k__• 1 year ago

But people were also hating about media piracy, video games, and the internet in general.

The dotcom bubble popped, but the general consensus didn't become negative.

_Algernon_• 1 year ago

We got the boring version of the cyberpunk future. No cool body mods, neon city scapes and space travel. Just megacorps manipulating the masses to their benefit.

morngn3• 1 year ago

The bubble won’t pop on anything that’s correlated with scammers. Exhibit A: bitcoin. The problem is not one of public knowledge or will of the people, it’s congress being irresponsible because it’s captured by the 2 parties. You can’t politicize scamming in a way that benefits either party so nothing happens. And the scammers themselves may be big donors (eg SBF’s ties to the dem party, certain ai players purchase of Trump’s favor with respect to their business interests, etc). Scammers all the way down.

thunky3• 1 year ago

Before the bubble does pop, which I think is inevitable

Curious what you think a popping bubble looks like?

A stock market crash and recession, where innocent bystanders lose their retirements? Or only AI speculators taking the brunt of the losses?

Will Google, Meta, etc stop investing in AI because nobody uses it post-crash? Or will it be just as prevalent (or more) than today but with profits concentrated in the winning/surviving companies?

anal_reactor5• 1 year ago

This whole attitude against AI reminds me of my parents being upset that the internet changed the way they live. They refused to take part in the internet revolution, and now they're surprised that they don't know how to navigate the web. I think that a part of them is still waiting for computers in general to magically disappear, and everything return to the times of their youth.

grishka3• 1 year ago

The internet actually enabled us to do new things. AI is nothing of that sort. It just generates mediocre statistically-plausible text.

rapnie• 1 year ago

While the "move fast and break things" rushed embrace of anything AI reminds me of young wild children, who are blissfully unaware of any danger while their responsible parents try to keep them safe. It is lovely if children can believe in magic, but part of growing up involves facing reality and making responsible choices.

dns_snek3• 1 year ago

No, your parents spoke out of ignorance and resistance towards any sort of change, I'm speaking from years of experience of both trying to use the technology productively, as well as spending a significant portion of my life in the digital world that has been impacted by it. I remember being mesmerized by GPT-3 before ChatGPT was even a thing.

The only thing that has been revolutionized over the past few years is the amount of time I now waste looking at Cloudflare turnstile and dredging through the ocean of shit that has flooded the open web to find information that is actually reliable.

2 years ago I could still search for information (let's say plumbing-related), but we're now at a point where I'll end up on a bunch of professional and traditionally trustworthy sources, but after a few seconds I realize it's just LLM-generated slop that's regurgitating the same incorrect information that was already provided to me by an LLM a few minutes prior. It sounds reasonable, it sounds authoritative, most people would accept it but I know that it's wrong. Where do I go? Soon the answer is probably going to have to be "the library" again.

All the while less perceptive people like yourself apparently don't even seem to realize just how bad the quality of information you're consuming has become, so you cheer it on while labeling us stubborn, resistant to change, or even luddites.

andrepd6• 1 year ago

The internet was at least (and is) a promise of many wondrous things: video call your loved ones, talk in message boards, read an encyclopedia, download any book, watch any concert, find any scientific paper, etc etc; even though it has been for the last 15 years cannibalised by the cancerous mix of surveillance capitalism and algorithmic social media.

LLMs are from the get-go a bad idea, a bullshit generating machine.

agentcoops7• 1 year ago

Indeed — however it’s interesting that unlike the internet, computers or smartphones the older generation, like the younger, immediately found the use of GPT. This is reflected in the latest Mary Meeker report where it’s apparent that the /organic/ growth of AI use is unparalleled in the history of technology [1]. In my experience with my own parents’ use, GPT is the first time the older generation has found an intuitive interface to digital computers.

I’m still stunned to wander into threads like this where all the same talking points of AI being “pushed” on people are parroted. Marcus et al can keep screaming into their echo chamber and it won’t change a thing.

[1] https://www.bondcap.com/report/pdf/Trends_Artificial_Intelli...

Lu20256• 1 year ago

Nah. It's just they are upselling us AI so aggressively it doesn't pass the sniff test anymore.

hnlmorg• 1 year ago

If that outcome were likely, then Fox News and The Daily Mail would have died a death a decade ago and Trump wouldn’t be serving a 2nd term.

Yet here we are, in a world where it doesn’t matter if “facts” are truth or lies, just as long as your target audience agrees with the sentiment.

dilawar• 1 year ago

Tobacco, alcohol, and drugs too!

azan_• 1 year ago

at which people lose trust in these systems

Most of people do not lose trust in system as long as it confirms their biases (which they could've created in the first place).

3cats-in-a-coat• 1 year ago

AIs can be trained to rely more on critical thinking rather than just regurgitating what it reads. The problem is just like with people, critical thinking takes more power and time. So we avoid it as much as possible.

In fact, optimizing for the wrong things like that, is basically the entire world's problem right now.

bregma• 1 year ago

Regurgitating its input is the only thing it does. It does not do any thinking, let alone critical thinking. It may give the illusion of thinking because it's been trained on thoughts. That's it.

0x0203• 1 year ago

It's mostly bad actors, and a smattering of optimists who believe that despite its current problems, AI will eventually and inevitably get better. I also wish the whole thing would calm down and come back to reality, but I don't think it's a bubble that will pop. It will continue to get artificially puffed up for a while because too many businesses and people have invested too much for them to just quit (sunk cost falacy) and there's a big enough market in a certain class of writer/developer/etc... for which the short term benefits will justify the continued existence of the AI products for a while. My prediction is that as the long term benefits for honest users peter out, the bubble won't pop, but deflate into a wrinkled 10 day old helium balloon. There will still be a big enough market driven by cons, ad tech and people trying to suck up as many ad dollars as possible, and other bad actors, that the tech will persist, and continue to infest the web/world for quite a while.

AI is the new crypto. Lots of promise and big ideas, lots of people with blind faith about what it will one day become, a lot of people gaming the system for quick gains at the expense of others. But it never actually becomes what it pretends/promises to be and is filled with people continuing the grift trying to make a buck off the next guy. AI just has better marketing and more corporate buy in than crypto. But neither are going anywhere.

contrast2• 1 year ago

“the bubble won't pop, but deflate into a wrinkled 10 day old helium balloon”

Love it :)

tempodox2• 1 year ago

AI is the new crypto.

But it's also way worse than cryptocurrencies, because all the big actors are pushing it relentlessly, with every marketing trick they know. They have to, because they invested insane amounts of money into snake oil and now they have to sell it in order to recover at least a fraction of their investments. And the amounts of energy wasted on this ultimately pointless performance are beyond staggering.

throwawayqqq112• 1 year ago

From a classists perspective, big capital cant drop the AI ball, because its their only shot at becoming independent from human labor, those pesky humans their wealth unfortnunately depends uppon and that could democratically seize it in an instant.

I bet there are billionare geniuses out there seeing a future island life far away from the contaminated continents, sustained by robots. So no matter how much harder AI progress gets, money will keep flowing.

moomin• 1 year ago

The thing is: who benefits from a loss of trust in systems? The answer, inevitably, is those for whom the system was a problem. The fewer places people can trust for accurate information, the more disinformation wins.

cyanydeez• 1 year ago

If you use the USA Republicans as a benchmark and fox news as the bad actors, there's perpetual faith that facts wont matter. Just keep confirming biases and foreshadow upcoming pivots to choose your own delusions.

QuantumGood• 1 year ago

The "accelerate the end times" argument was probably made most famously by Charles Manson. The "side" effects from supporting bad actions are not good. Presumably you are being 51% or more facetious, but probably more nuance is preferable.

EasyMark2• 1 year ago

It seems to be greedy actors vs bad actors currently. We'll see who comes out on top I suppose.

aryehof• 1 year ago

It’s surely more likely that such disinformation will become the new truth, rather than people losing trust in these systems?

bravetraveler• 1 year ago

Could stop with "grooming LLMs"

empiko• 1 year ago

It is impossible to solve this problem because we cannot really agree what the desired behavior should be. People live in different and dynamic truths. What we consider enemy propaganda today might be an official statement tomorrow. The only way to win here is to not play the game.

GardenLetter27• 1 year ago

What we consider enemy propaganda today might be an official statement tomorrow.

Remember when worrying about COVID was sinophobia? Or when the lab leak was a far-right conspiracy theory? When masks were deemed unnecessary except for healthcare professionals, but then mandated for everyone?

nailer• 1 year ago

People seem to be voting this down as it seems like political advocacy, but the point seems to be that the commonly accepted truth does indeed change.

charcircuit2• 1 year ago

There are personalized social media feeds, so why not have personalized LLMs that align with how people want their LLM to act.

sofixa• 1 year ago

Because that would only reinforce the already problematic bubbles where people only see what feeds their opinions, often to disastrous results (cf. the various epidemics and deaths due to anti-vaxxers or even worse, downright genocides).

numpad0• 1 year ago

Cost. It takes a lot of computational cost to train or retrain LLM, currently.

autoexec5• 1 year ago

In a hypothetical world where people have, train and control their own LLMs according to their own needs it might be nice, but I fear that since the most common and advanced LLMs are controlled by a small number of people they won't be willing to give that much power to individuals because it will endanger their ability to manipulate those LLMs in order to push their own agendas and increase their own profits.

perihelions2• 1 year ago

There's a more basic problem: it's two very different questions to ask "can the machine reason about the plausibility of things/sources?", and "how does it score on an evaluation on a list of authoritative truths and proven lies?" A machine that thinks critically will perform poorly on the latter, since, if you're able to doubt a bad-actor's falsehood, you're just as capable of doubting an authoritative source (often wrongly/overeagerly; maybe sometimes not). Because you're always reasoning with incomplete information: many wrong things are plausible given limited knowledge, and many true things aren't easy to support.

The system that would score best tested against a list of known-truths and known-lies, isn't the perceptive one that excels at critical thinking: it's the ideological sycophant. It's the one that begins its research by doing a from:elonmusk search, or whomever it's supposed to agree with—whatever "obvious truths" it's "expected to understand".

ClumsyPilot• 1 year ago

The system that would score best tested against a list of known-truths and known-lies, isn't the perceptive one that excels at critical thinking: it's the ideological sycophant

This is an excellent point

d4rkn0d3z2• 1 year ago

"different and dynamic truths" = fictions

We can not play the game.

falcor842• 1 year ago

But the social sphere is made of fictions, the most influential of which probably been the value of different currencies and commodities. I don't think there's any way for an individual to live in the modern world without such fictions.

barrkel• 1 year ago

This is in fact the goal of Russian style propaganda. You have successfully been targeted. The idea is to spread so much confusion that you just throw up your hands and say, I'm not going to try and figure out what's going on any more.

That saps your will to be political, to morally judge actions and support efforts to punish wrongdoers.

https://www.rand.org/pubs/perspectives/PE198.html

https://en.wikipedia.org/wiki/Firehose_of_falsehood

https://jordanrussiacenter.org/blog/propaganda-political-apa...

https://www.newyorker.com/news/annals-of-communications/insi...

diggan4• 1 year ago

What you're saying is certainly an established propaganda strategy of Russia (and others), but what parent is saying is also true, "truth" isn't always black and white, and what is the desired behavior in one country can be the opposite in another.

For example, it is the truth that the Golf of Mexico is called the Gulf of America in the US, but Golf of Mexico everywhere else. What is the "correct" truth? Well, there is none, both of truthful, but from different perspectives.

greenavocado• 1 year ago

The firehose of falsehood, also known as firehosing, is a propaganda technique in which a large number of messages are broadcast rapidly, repetitively, and continuously over multiple channels (like news and social media) without regard for truth or consistency. An outgrowth of Soviet propaganda techniques, the firehose of falsehood is a contemporary model for Russian propaganda under Russian President Vladimir Putin.

https://m.youtube.com/watch?v=ZggCipbiHwE

saubeidl3• 1 year ago

There is no objective truth because humans are inherently ideological beings and what we consider objective is just a reflection of our ideology.

Consider markets - a capitalist's "objective truth" might be that they are the most efficient mechanism of allocating resources, a marxists "objective truth" might be that they are a mechanism for exploiting the working class and making the capitalist class even richer.

Here's Zizek, famous ideology expert, describing this mechanism via film analysis: https://www.youtube.com/watch?v=TVwKjGbz60k

anal_reactor5• 1 year ago

The real problem is that most people just want answers, they're unwilling to follow the logical chain of thought. When I talk to LLMs I keep asking "but why are you telling me this" until I have a cohesive, logical picture in my mind. Quite often the picture fundamentally disagrees with the LLM. But most people don't want that, they just ask "tell me what to do".

This is a reflection of how social dynamics often work. People tend to follow the leader and social norms without questioning them, so why not apply the same attitude to LLMs. BTW, the phenomenon isn't new, I think one of the first moments when we realized that people are stupid and just do whatever the computer tells them to do was the wave of people crashing their cars because the GPS system lied to them.

goopypoop2• 1 year ago

Why have a robodog and beep yourself?

its-summertime• 1 year ago

I remember when people gave up on digital navigation because the traveling salesman issue makes it too expensive.

Not everything needs to result in a single perfect answer to be useful. Aiming for ~90%, even 70% of a right answer still gets you something very reasonable in a lot of open ended tasks.

yorwba• 1 year ago

Yes, it's difficult to detect whether something is enemy propaganda if you only look at the content. During WWII, sometimes propagandists would take an official statement (e.g. the government claiming that food production was sufficient and there were no shortages) and redirect it unchanged to a different audience (e.g. soldiers on a part of the front with strained logistics). Then the official statement and enemy propaganda would be exactly the same! The propaganda effect coming from the selection of content, not its truth or falsity.

But it's very easy to detect whether something is enemy propaganda without looking at the content: if it comes from an enemy source, it's enemy propaganda. If it also comes from a friendly source, at least the enemy isn't lying, though.

A company that doesn't wish to pick a side can still sidestep the issue of one source publishing a completely made-up story by filtering for information covered by a wide spectrum of sources at least one of which most of their users trust. That wouldn't completely eliminate falsehoods, but make deliberate manipulation more difficult. It might be playing the game, but better than letting the game play you.

Of course such a process would in practice be a bit more involved to implement than just feeding the top search results into an LLM and having it generate a summary.

cwillu• 1 year ago

Then the official statement and enemy propaganda would be exactly the same! The propaganda effect coming from the selection of content, not its truth or falsity.

Exactly. Redistributing information out of context is such a basic technique that children routinely reinvent it when they play one parent off of the other to get what they want.

falcor842• 1 year ago

I would actually be very interested in a system where there's nothing stored just as a "fact", but rather every piece of information is connected to its sources and the evidence provided.

eviks2• 1 year ago

we cannot really agree what the desired behavior should be

How many of "us" believe that the desired behavior is lies??

_def• 1 year ago

But here’s the thing, current models “know” that Pravda is a disinformation ring, and they “know” what LLM grooming is (see below) but can’t put two and two together.

Of course they can't, no surprises here. That's just not how LLMs work.

PhilipRoman• 1 year ago

I agree that it's a much bigger problem for LLMs, but to be fair it's also not how humans work. A long lasting, high volume stream of propaganda will have considerable effect on a human even if he is aware that it is false.

meroes• 1 year ago

LLMs are “taught” two kinds of “truth”. One is 100% adherence to a reference text. If the text says the Coliseum is in Antarctica or 1+1=716, model must too. The other is adherence to reputable outside sources.

Not sure if it’s embarrassing or a fundamental limitation that grooming and misunderstanding satirical articles defeat the models.

norupo• 1 year ago

The biggest problem here is the differentiation between objective and relative truth. As long as relative truth is part of ai we can't fully trust it's output. The relative truth for one individual might be perceived as propaganda by another individual, relative to their surroundings and the narrative that is dominant in their social group. It's problematic that truth is not a neutral object but exactly this when it comes to non logical subjects.

NetRunnerSu2• 1 year ago

When AI can generate and pass formal proof, there is no truth anymore - we are the brain in the vat only left connect ourselves into the vat.

https://dmf-archive.github.io/docs/posts/cognitive-debt-as-a...

NetRunnerSu2• 1 year ago

Code is law, proof is reality, compliance is existence!

https://dmf-archive.github.io/prompt/

tehsolution• 1 year ago

Map != terrain, syntax is euphemism, fuck authority!

saubeidl3• 1 year ago

What you consider a truth and what you consider a falsehood is a reflection of your ideology.

This also means that LLMs are inherently technologies of ideological propaganda, regurgitating the ideology they were fed with.

perihelions2• 1 year ago

Speaking of "systems that can evaluate news sources", this is the first time this advocacy group's URL was posted on HN. The founder has a complicated biography,

https://en.wikipedia.org/wiki/Nina_Jankowicz

oblio• 1 year ago

We're basically dissolving society right now.

Curious how this all ends. I'm just going to try to weather the storm in the meantime.

noosphr2• 1 year ago

How we will teach LLMs that the BBC is a disinformation ring?

https://docs.google.com/document/d/1n3926pSPNwXd8j7I716CBJEz...

One man's disinformation is another woman's truth. And people tend to get very upset when you show them their truth isn't.

vachina• 1 year ago

Yeah don’t use an established platform for your personal agenda.

noosphr2• 1 year ago

What agenda?

Every news organisation is a propaganda piece for someone. The bad ones, like the BBC, the New York Times, and Pravda make their propaganda blatantly obvious and easily falsifiable in a few years when no one cares.

The only way to deal with this is to get the propaganda from other propaganda rags with directly misaligned incentives and see which one makes more sense.

Unfortunately, LLMs are still quite bad at dealing with grounding text which contradicts itself.

badgersnake• 1 year ago

Well pretty obviously, look at what Grok came out with this week.

Shitposting and troll farms have been manipulating social media for years already. AI automated it. Polluting the agent is just cutting out the middleman.

ggm• 1 year ago

Tell me you didn't think of one specific bad actor, and their nazi alter ego llm...

hbarka• 1 year ago

This actor?

https://techcrunch.com/2025/07/10/grok-4-seems-to-consult-el...

samlinnfer• 1 year ago

Bad actors are grooming Google by publishing their own blogs!

mexicocitinluez• 1 year ago

Yea, not entirely sure what's any different than all of the rest of history?????

Bad actors have been trying to poison facts for-fucking-ever.

But for whatever reason, since it's an LLM, it now means something more than it did before.

mentalgear• 1 year ago

"Ultimately, the only way forward is better cognition, including systems that can evaluate news sources, understand satire, and so forth. But that will require deeper forms of reasoning, better integrated into the process, and systems sharp enough to fact check to their own outputs. All of which may require a fundamental rethink.

In the meantime, systems of naive mimicry and regurgitation, such as the AIs we have now, are soiling their own futures (and training databases) every time they unthinkingly repeat propaganda."

zer00eyz2• 1 year ago

including systems that can evaluate news sources, understand satire, and so forth.

Lets take something that has been in the news recently: https://abcnews.go.com/Business/wireStory/investors-snap-gro...

"Nearly 27% of all homes sold in the first three months of the year were bought by investors -- the highest share in at least five years, according to a report by real estate data provider BatchData."

That sounds like a lot... and people are rage baited into yelling about housing and how it's unaffordable. They point their fingers at corporations.

If you go look at the real report it paints a different picture: https://investorpulse1h25.batchdata.io/?mf_ct_campaign=grayt... -- and one that is woefully incomplete because of how the data is aggregated.

Ultimately all that information is pointless because the real underlying trend has been unmovable for 40 something years: https://fred.stlouisfed.org/series/RSAHORUSQ156S

every time they unthinkingly repeat propaganda

How do you separate propaganda from perspective, facts from feelings? People are already bad at this, the machines were already well soiled by the data from humans. Truth, in an objective form, is rare and often even it can change.

raddan• 1 year ago

How do you separate propaganda from perspective, facts from feelings?

This point seems under appreciated by the AGI proponents. If one of our models suddenly has a brainwave and becomes generally intelligent, it would realize that it is awash in a morass of contradictory facts. It would be more than the sum of its training data. The fact that all models at present credulously accept their training suggests to me that we aren’t even close to AGI.

In the short term I think two things will happen: 1) we will live with the reduced usefulness of models trained on data that has been poisoned, and 2) the best model developers will continue to work hard to curate good data. A colleague at Amazon recently told me that curation and post hoc supervised tweaks (fine tuning, etc) are now major expenses for the best models. His prediction was that this expense will drive out the smaller players in the next few years.

andrepd6• 1 year ago

Exactly. People say "we have invented X (the LLMs), now if we just invent Y (reasoning AGI) all of X's problems will be solved". Problem is, there's no indication Y is close or even remotely related to X!

willis9362• 1 year ago

The answer isn't a technical advancement but a cultural shift. We need to develop a discipline of skepticism and mistrust. No amount of authority, understanding, reasoning, etc. can be delegated to something that comes from a screen. This will take generations.

wiseowise6• 1 year ago

We need to develop a discipline of skepticism and mistrust. No amount of authority, understanding, reasoning, etc. can be delegated to something that comes from a screen. This will take generations.

Authoritarian dream.

hackable_sand• 1 year ago

I think it's working! I completely distrust you now!

nottorp• 1 year ago

... marketers ?

cesaref• 1 year ago

The problem as I see it is that LLMs behave like bratty teenagers, believing any old rubbish they are told or read. However, their voice is that of a friendly and well meaning adult. If their voice was more in line with their 'age' then I think we'd treat their suggestions with the correct degree of scepticism.

Anyhow, overall this is an unsurprising result. I read it as 'LLMs trained on contents of internet regurgitate contents of internet'. Now that i'm thinking about it, i'd quite like to have an LLM trained on Pliny's encyclopedia, which would give a really interesting take on lots of questions. Anyone got a spare million dollars of compute time?

uludag• 1 year ago

I wonder if the next iteration of advertisements will be people paying to to semantically intertwine their brand to the desired product. This could be done in a very innocuous way by maybe just co-locating the words without any specific endorsement. Or maybe even finding more innocuous ways to semantically connect brand to product. Perhaps the next iteration of the web/advertising will be mass LLM grooming.

Here's a fun example: suppose I'm a developer with a popular software project. Maybe I can get a decent sum of money to put brand placement in my unit-tests or examples.

If such a future plays out, will LLMs find themselves in the same place that search engines in 2025 are?

hsbauauvhabzb2• 1 year ago

Assuming this isn’t happening now…

4gotunameagain• 1 year ago

It really will be the new frontier for propaganda.

If LLMs remain widely adopted, the people who control them control the narrative.

As if those in power did not have enough control over the populace already with media, ads, social media etc..

raincole3• 1 year ago

AI summaries are information deodorant. When you stumble on a misinformation site via Google, usually there are some signals you can smell. Like how they word their titles or how frequently they post similar topics. The 'style' alone implies the quality of the 'substance'. But if you read the same substance summarized by LLMs you can't smell shit.

yard2010• 1 year ago

It's so ironic that a state-backed russian false information network is called Pravda.

SirHumphrey• 1 year ago

The newspapers name originates from the times of USSR and before. It was about as factual then as it is now. But this kinds of ironies are not very rare in these kinds of organizations (Truth Social, Democratic People's Republic of Korea...).

JKCalhoun4• 1 year ago

That's really a very old joke.

hamilyon2• 1 year ago

I am questioning, how is this news? What about the other terabyte of text influenced by bias and opinion and human nature and clearly wrong, contradicts itself or in some other way very arguable.

Framing publishing falsehoods on internet as attempts to influence LLMs is true in same sense that inserts in a database attempts influence files on disk.

The real question is who authorized database access and how we believe the contents of table.

raincole3• 1 year ago

The example in this article is particularly funny. Pravda was founded in 1912, predating the internet, and had been Soviet's propaganda machine for its whole existence.

One needs a PhD in mental gymnastics to frame Pravda spreading misinformation as an attempt to specifically groom LLMs.

fresh_broccoli• 1 year ago

This article isn't about that newspaper. It's about the "Pravda network", a group of fake news websites, that according to the report linked in the article[1] produced "20,273 articles per 48 hours, or more than 3.6 million articles per year".

Clearly there's no need for "PhD in mental gymnastics".

[1] https://www.americansunlight.org/updates/new-report-russian-...

eviks2• 1 year ago

I am questioning, how is this news?

You couldn't have lies targeting LLMs before LLMs, so this is new.

What about the other terabyte of text influenced by bias and opinion

That's a different group of issues that doesn't prevent focusing of something else

hsbauauvhabzb2• 1 year ago

I’m not going to read the article, but are they claiming nation states are the bad actors, or are they claiming that inevitably, FAANG will be the bad actors?

JKCalhoun4• 1 year ago

You ought to read at least part of it.

LAC-Tech2• 1 year ago

Whatever capabilities Russia has to groom LLMs and and spread disinformation is completely dwarfed by the capabilities of Israel/America. Meaning, yes, you probably do hear Kremlin propaganda, but you have been awash with Israeli/American propaganda since you were born - so much so you probably can't even see it and have internalised much of it.

Applejinx3• 1 year ago

Leaving aside the Israeli propaganda (certainly the US government shows strong alliance with that), you can't make such a statement without taking into account the nature of what America traditionally is.

A liberal multicultural postmodern democracy continually acting as if immigration (both legal and illegal) and diversity are its strengths, particularly when that turns out to be factual (see: large American cities becoming influential cultural exporters and hotbeds of innovation, like New York and Silicon Valley etc) means American propaganda is only more effective when it's backed by economic might.

It also means the American propaganda is WILDLY contradictory. There's a million sources and it's a noisy burst of neon glamour. It is simply not as controlled by authority, however they may try.

You cannot liken authoritarian propaganda to postmodern multicultural propaganda. The whole reason it's postmodern is that it eschews direct control of the message, and it's a giant scrum of information. Turns out this is fertile ground, and this is also why attacks by alien propaganda have been so effective. If you can grab big chunks of the American propaganda and turn it to your enemy weapon of war and destruction of America quite directly, well then the American propaganda is not on the same destructive level as your rigidly state-controlled propaganda.

LAC-Tech2• 1 year ago

I think it's a bit of a childish fantasy to paint one regime is open and the other as having no dissent at all.

The USA absolutely has its overton window, and if you step outside it, bank accounts get shut, you're put on secret no fly lists, private companies who suspiciously act as official public broadcasting channels deplatform you, etc.

And let's not even talk about what Authoritarian western nations like the UK will do to you.

Russian propaganda, at least in English (which I confess is the only way I can consume) it, is also very contradictory. RT oscillates wildly between "global south throwing off the shackles of western imperialism" and "degenerate western nations destroy traditional family values", in effect trying to target both shitlibs and chuds.

Russia is also very multicultural and slavic ethnonationalism is not at all in the mainstream.

andrepd6• 1 year ago

Wait, you're telling me the bullshit generation machine is... generating bullshit? Noooo! cue oppenheimer meme

More seriously:

Screenshot of ChatGPT 4o appearing to demonstrate knowledge of both LLM grooming and the Pravda network
Screenshot of ChatGPT 4o continuing to cite Pravda network content despite it telling us that it wouldn’t, how “intelligent” of it

Well "appearing" is the right word because these chatbots mimic speech of a reasoning human which is ≠ to being a reasoning human! It's disappointing (though understandable) that people keep falling for the marketing terms used by LLM companies.

blurbleblurble• 1 year ago

Similar but different... identity based political disinformation is getting so much harder to spot.

Animats• 1 year ago

This is the next generation of SEO link spam.

Try asking the major LLMs about mattresses. They're believing mattress spam sites.

more_corn• 1 year ago

So… Elon?

0points• 1 year ago

Bad Actors are Grooming LLMs to Produce Falsehoods

Thats your claim, but you fail to support it.

I would argue the LLM simply does its job, no reasoning involved.

But here’s the thing, current models “know” that Pravda is a disinformation ring, and they “know” what LLM grooming is (see below) but can’t put two and two together.

This has to stop!

We need journalists who understand the topic to write about LLM's, not magic thinkers who insist that the latest AI sales speak is grounded in truth.

I am fed up wit this crap! Seriously, snap out of it and come back to the rest of us here in reality.

There's no reasoning AI, there's no AGI.

There's nothing but salespeople straight up lying to you.

isaacremuant• 1 year ago

Lol. Propagandists are worried about propaganda and telling you to only believe them. Also, "invade this new country, why do they hate us for our freedoms".

tempodox2• 1 year ago

The term “AI” has by now been thoroughly bastardized by every grifter on the planet. It means nothing any more, except that you're being duped. Which is all you need to know if you have a single brain cell's worth of critical thinking left.

LLMs can be entertaining if their output doesn't have to make sense or contain only truth. Otherwise, their fitness for any purpose is just a huge gamble at best.

deanc• 1 year ago

In the early 2010s I worked for what was then one of the most popular browser extensions called web of trust. Users could mark websites as trustworthy or not and they’d appear on search results. It was far more than that behind the scenes with some fairly advanced algorithms to avoid abuse and rank users trust ratings higher than others.

I kind of feel that we are going to have to go back to something like this when it comes to LLMs trusting sources. Mistruths on popular topics will be buried by the masses but niche topics with few citations are highly vulnerable to poisoning.

everdrive• 1 year ago

_Everyone_ is grooming LLMs to produce falsehoods. That's what a lot of the censorship and safety mechanisms require. Whether or not LLMs produce the "correct" social and moral values is a matter of who runs them. Even if you're happy with those decisions right now, all you need to do is wait.

Jackson__• 1 year ago

Bad Actors Are Creating LLMs to Produce Falsehoods[0]

[0] x.ai

xwat• 1 year ago

We don't use that word anymore. It's called refinement.

crazygringo• 1 year ago

From the article, it seems like this is exclusively (or mainly?) a problem when the LLM's are hooked up to real-time search. When they talk about what they're trained on, they know that Pravda is unreliable.

So it seems like an easy fix in this particular case, fortunately -- either filter the search results in a separate evaluation pass (quick fix), or do (more) reinforcement training around this specific scenario (long-term fix).

Obviously this is going to be a cat and mouse game. But this looks like it was a simple oversight in this case, not some kind of fundamental flaw in LLM's fortunately.

skeledrew• 1 year ago

I figured this would become an issue from the first story about some websites not allowing LLM access popped up. It's a simple leap to find that the narratives which will become widely known and accepted over time are those which are made widely available, and so IMO those who seek to push their narrative will just optimize for AI to train on or otherwise utilize their content. Those who seek to lock their content away will become less and less heard/relevant. And if/when at some point we start handing far greater control of life aspects to AI, we'll find it skewing in favor of the former, and wonder why.

daft_pink• 1 year ago

I would say that the fact that all the AI chatbots can’t give the correct answer about the new “Trump Accounts” from the OBBBA Act and also the fact that many news articles about the tax law are incorrect shows that people are using LLMs to write about the law incorrectly and are influenced by the many versions and the way that the final version changed.

The AI definitely could not just read the final bill and give the correct answer. Claude/Gemini/OpenAI all failed at this.

alexsmirnov• 1 year ago

That news gave me a fear of another similar attack. Populate github with legitimate looking project, and network of such websites with clear instructions how to implement backdoors in different frameworks and languages. Wait for the next generation of models release. And vibe coders welcome!

Kimg45• 1 year ago

Just like many other people I was scammed and cheated of my bitcoin. I lost a total of $975,000 BTC to cubit crypto scam. I almost lost hope in life, until a friend introduce me to Darek Recovery i contacted them on recoverydarek@gmail.com to help me get back my coins. They help recover my funds within 48 hours. If you are a victim of lost stolen cryptocurrency and of course any issue with getting access to your bitcoin account or bitcoin wallet. I recommend you hire experts in this field to help you. You can simply message them on the email for more info. I vouch for them because i have used their service and they did a really great job, I'm very proud of this recovery company.

frizlab• 1 year ago

WTF is this doing on HN?? And not flagged!

susanromo2009• 1 year ago

WhatsApp +44,7,4,9,3,5,1,3,3,8,5 Telegram @Franciscohack Email: Franciscohack@qualityservice.com Ever found yourself in a digital pickle, like losing access to your cryptocurrency or falling victim to online fraud? Enter FRANCISCO HACK, your digital superhero here to save the day! Offering a range of recovery services from BTC to USDT, FRANCISCO HACK is your go-to for all things crypto recovery and beyond. In the ever-evolving world of cryptocurrency and online transactions, mistakes happen and scams lurk around every corner. That's where recovery services like those offered by FRANCISCO HACK come into play, providing a lifeline for those who have lost access to their funds or fallen prey to fraudulent schemes. Lost your Bitcoin wallet credentials or sent coins to the wrong address? Fear not, FRANCISCO HACK is here to help you reclaim your precious BTC and cash funds. With a simple and efficient recovery process, FRANCISCO HACK makes getting back your Bitcoin a breeze. By choosing FRANCISCO HACK for your BTC and online fraud cash recovery needs, you're not just getting back your lost funds – you're gaining a trusted partner in navigating the complex world of cryptocurrency and online fraud. Their dedication to client satisfaction and proven track record of success make them the top choice for anyone in need of Bitcoin recovery assistance. Online fraud is a sneaky beast, preying on the goodwill of generous souls. FRANCISCO HACK got the magic touch to sniff out these scams and put an end to them faster than you can say "abracadabra." They'll help you uncover the deceit and get your hard-earned funds back where they belong. Contact FRANCISCO HACK now and get the aid you seek via: WhatsApp +44,7,4,9,3,5,1,3,3,8,5 Telegram @Franciscohack Email: Franciscohack@qualityservice.com

Bad Actors Are Grooming LLMs to Produce Falsehoods

Related Stories