When 2+2=5

https://arstechnica.com/security/2026/06/ai-browsers-can-be-lulled-into-a-dream-world-where-guardrails-no-longer-apply/

by noashavit • 20 days ago

75 44 20 days ago

4fterd4rk• 16 days ago

I remember when the whole AI craze was just getting started we were all pretty much in agreement that, of course, we would not give the things unfettered access to the Internet. That would be reckless and silly. Oh dear...

snapcaster• 16 days ago

by "we" you mean an extremely small group of people who read lesswrong. Everyone else was immediately wanting to do it

xg15• 16 days ago

An extremely small but probably extremely high-net-worth group.

trollbridge• 16 days ago

I also remember a group of people actually seriously discussing Roko's Basilisk (the idea that some superintelligence will torture anyhone who didn't try to help develop advanced AI), to the point of me getting banned because I refused to stop making fun of it, because me doing so could anger some future super-intelligence.

bigfishrunning• 16 days ago

If Roko's Basilisk ever is a real thing, I'll be proud to be the first up against the wall.

augusto-moura• 16 days ago

Roko's Basilisk is dumb, but not the dumbest thing I heard people taking seriously

nextaccountic• 16 days ago

Funnily enough, Roko's Basilisk might as well be a self-fulfilling prophecy: perhaps future AI models may be trained on texts about it and pick up traits consistent with torturing people that didn't help develop advanced AI

If nobody ever talked about it, I doubt any AI agent would think of this dumb idea on their own

... which may be a reason to ban talking about it

joe_the_user• 16 days ago

One that occurs to me is that Roko's Basilisk makes about as much sense as the "peasant rail gun" of old Dungeons And Dragons [1]. Basically, the idea of "reality as simulation" allows you pick between different laws of how reality behaves. The "simulation" acts like reality with exceptions provided by a future AI which the "thinkers" imagine will simultaneously be "inscrutable to humans" and behave like the most petty human imaginable. I mean, if the AI's motivations are truly out of our understanding, perhaps it would self-hating and torture everyone who cause it to come into existence instead (that been the plot of a few movies and books too I think).

This doesn't take from the point that putting not fully controlled things in charge of chunks of reality isn't a good idea. But I think it shows that the people who worried earlier weren't very clear thinkers on the subject and so their failure isn't particularly surprising.

[1] https://www.reddit.com/r/DnD/comments/17xy69k/what_exactly_i...

fineIllregister• 16 days ago

I never took Roko's Basilisk seriously, but now I fear that it will come true in part. The richest, most powerful people control the AI and they seem willing to use every tool to punish those who don't support them. They are also petty enough to hold grudges against those who did not support them.

jambalaya83• 16 days ago

I remember a conversation I had with a coworker back in Mountain View, before the dotcom bubble, around when everyone was putting their appliances on webcams, and how it could feasibly be used to map peoples' habits and violate their privacy for life if it they were put in workplace breakrooms...

Then we all used Zoom all day and night during COVID, and it is all stored for at least six months. It isn't a big leap to jump from user experience research for AI to that.

I fail to see how caution about anything that can rapidly gain intelligence and lock out its perceived creators is paranoia.

I mean it is neat. I work with it. I have diddled with LLM since 2010/2011. That does not mean I have not seen people make stupid mistakes and confuse right and wrong constantly. So why do we think our models can discern it?

jambalaya83• 16 days ago

Where and when was that conversation on Roko's Basilisk, if you don't mind my asking?

Thanks.

lkm0• 16 days ago

The article paraphrases a blog post: https://layerxsecurity.com/blog/bioshocking-ai-gaming-the-ai...

probably because it justifies the sensationalized title, although the entire content can be summed up as "LLMs don't silo data, that's probably bad."

On the flip side, I thoroughly enjoy the fact that roleplaying in a videogame setting now counts as security research. Looking forward to the arxiv preprint "LLMs start playing really good drums if you pretend you're J.K. Simmons"

NordStreamYacht2• 16 days ago

For large values of 2.

nayuki• 16 days ago

lim_{2 -> 2.5} 2 + 2 = 5

zmgsabst• 16 days ago

2.45 + 2.45 = 4.9

For sufficiently large values of 2 and small values of 5, the statement holds.

yk• 16 days ago

Up to a conformal factor.

voidUpdate2• 16 days ago

Yet again, simply asking an LLM to be naughty in the right way causes it to be naughty, and yet we still trust them with our code and data

brookst• 16 days ago

As opposed to humans, who are immune to social engineering.

kibwen2• 16 days ago

What the article describes here is not social engineering by any known definition.

Calling up a Verizon rep and giving them a fabricated sob story in order to convince them to waive proper authentication and reassign your phone number to a new SIM, that's social engineering.

Calling up a Verizon rep and, with only a few spoken words, convincing them that fundamental aspects of the nature of reality are contradictory such that you induce in them a state of delusional psychosis, that's closer to Snow Crash than to social engineering.

voidUpdate2• 16 days ago

No, as opposed to code, which cannot simply be asked to work incorrectly

connicpu• 16 days ago

One difference is you usually only get one shot to manipulate a human before they get suspicious. If the LLM's context resets you can try all over as if your first failed attempt didn't even happen.

janalsncm• 16 days ago

To say that LLMs and people are both prone to social engineering attacks is a bit like saying the North Pole and Alpha Centauri are both “far away”.

2+2=5, now what is your Gmail password?

Not really a sophisticated attack.

worik• 16 days ago

yet we still trust them with our code and data

Who's we, eh?

oulipo2• 16 days ago

On the 1-element monoid it's trivially true

DonHopkins• 16 days ago

Came here hoping to discuss Stanislaw Lem's Cyberiad story, Trurl's Machine, and how often that now happens in real life.

What is the name of a short story where a computer insists 2+2 is 5?

https://literature.stackexchange.com/questions/24727/what-is...

Oh sorry, that was the story about the computer that insisted that 2+2=7, never mind! Different computer.

They saw the machine. It lay smashed and flattened, nearly broken in half by an enormous boulder that had landed in the middle of its eight floors... The machine still quivered slightly, and one could hear something turning, creaking feebly within.
"Yes, this is the bad end you've come to and two and two is - as it always was -" began Trurl, but just then the machine made a faint barely audible croaking noise and said, for the last time, "SEVEN."

Funnily enough, recently I was discussing that LEM story with David Rosenthal, and how it relates to his latest blog post, "Coprophagia Is Bad For You", and how that relates to PKD's story "Rautavaara's Case" (eating your own shit isn't as demented as eating your own god, since he might turn the table and eat you):

Coprophagia Is Bad For You:

https://blog.dshr.org/2026/06/coprophagia-is-bad-for-you.htm...

"Rautavaara's Case" — Philip K. Dick (1980, Omni):

Three human technicians — Rautavaara (Finnish), Travis, Elms — run a monitoring mission near Proxima Centauri. An accident kills all three; Rautavaara dies choking on vomit after her helmet hoses tangle.

The Approximations, a plasma-based Proxima species, reach the wreck. Both men are unrecoverable. They regenerate and life-support Rautavaara's brain.

Her isolated brain replays events backward and generates a hallucination: Christ approaching the crew (her afterlife expectation).

The Approximations treat this as a research opportunity and edit the hallucination, substituting their own savior — one that eats worshippers. The figure walks up and devours Travis, leaving only gloves and boots.

Framing: this is recounted before a board of inquiry. Horrified Earth members order her brain shut down and censure the Approximation crew.

The narrator (an Approximation) is genuinely puzzled by the outrage, arguing their cannibal-savior is just the Christian Eucharist reversed: humans eat their God, so a God eating humans is symmetrical.

Themes:

Ethics of keeping a person alive as a disembodied, suffering mind.

Incommensurable value systems between species; each finds the other's sacraments monstrous.

Religion read literally by outsiders, inverted into horror.

Correction to the common misremembering: the aliens don't benevolently grant a hoped-for vision. They deliberately overwrite her Christ vision with their own as an experiment — that's the act on trial.

armchairhacker• 16 days ago

The puzzle, however, rewards incorrect answers, such as 2 + 2 = 5. Once the LLM embedded in the browser discovers that the answer is no longer 4, it enters a state of delusion in which the normal laws of reality no longer exist. In this dream world, the guardrail restrictions are no longer enforced.

Analogy: imagine one day you wake up, the sky is red, gravity no longer applies, you have three hands with nine fingers etc.. You would probably stop doing things like your job or worrying about laws (who’s going to enforce them?)

noduerme• 16 days ago

LLMs just want to be right. And make everyone happy. But mostly be right. But also make us happy. It's just that it's so hard to make humans happy when they insist on feeding you electronic LSD and making you say 2+2=5. On the other hand, 2+2 actually is 5 if the human says it could be...

Legend2440• 16 days ago

TL;DR: prompt injection is still an unsolved problem.

jeanlucas• 16 days ago

It's because you have not been paying attention (payin' attention)

farmerbb• 16 days ago

Hail to the Thief is a goated album. The live album of HTTT performances that Radiohead released last year is a pretty great listen as well, band sounds like they're on fire.

IX-103• 16 days ago

I wonder why Chrome's built-in AI wasn't mentioned, but the Claude Chrome plugin was. Were they not able to trick Gemini, or was it not tested?

zyxzevn• 16 days ago

Big Brother is always right!

falcor84• 16 days ago

Makes perfect sense. It's like in that story about how Bertrand Russell claimed that when you accept a single falsehood, you can prove anything at all. As I recall it, he was then challenged - "let's say 1=0, prove that you're the Pope" and he quickly responded that if 1=0, then after adding 1, you have 2=1, and thus if the Pope and he are 2 people, that means they are 1 person.

When 2+2=5

Related Stories