A rogue AI led to a serious security incident at Meta

The two errors, then, were that the LLM hallucinated something, and that a human trusted the LLM without reasoning about its answer. The fix for this common pattern is to reason about LLM outputs before making use of them.

krupan8• 4 months ago

It's more like, the LLM "hallucinated" (I hate that term) and automatically posted the information to the forum. It sounds like the human didn't get a chance to reason about it. At least not the original human that asked the LLM for an answer

c-linkage• 4 months ago

If you don't like hallucinate, try bullshit. [NB: bullshit is a technical term; see https://en.wikipedia.org/wiki/On_Bullshit]

https://www.psypost.org/scholars-ai-isnt-hallucinating-its-b...

nytesky• 4 months ago

I’m not in AI, but what is happening is that it is building output from the long tail of its training data? Instead of branching down the more common probability paths, something in this interaction had it travel into the data wilderness?

So I asked AI to give it a good name, and it said “statistical wandering” or “logical improv”.

paxys3• 4 months ago

A big problem now both internally to a company and externally is that official support channels are being replaced by chatbots, and you really have no option but to trust their output because a human expert is no longer available.

If I post a question to the internal payment team's forum about a critical processing issue and some "payments bot" replies to me, should I be at fault for trusting the answer?

RussianCow• 4 months ago

I know this is happening with external customer support, but is this really happening internally at big companies? Preventing you from talking to a human in the correct department about an issue feels like a bomb waiting to explode.

Uhhrrr5• 4 months ago

Yes, of course, and the company which removes human experts should expect things to fail in the manner that things usually fail when you remove your internal experts.

leptons3• 4 months ago

If "the level of awareness that created a problem, cannot be used to fix the problem", then you're asking too much if you expect a human to reason about an LLM output when they are the ones that asked an LLM to do the thinking for them to begin with.

thwarted• 4 months ago

This feels like a rediscovering/rewording of Kernighan's Law:

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." ~ Brian Kernighan

Uhhrrr5• 4 months ago

In this case you would replace the human.

somewhereoutth• 4 months ago

However - Automation bias is a common problem (predating AI), the 'human-in-the-loop' ends up implicitly trusting the automated system.

krupan8• 4 months ago

At least pre-LLM automation was written by a careful human who's job was on the line, and was deterministic.

SlinkyOnStairs• 4 months ago

The fix for this common pattern is to reason about LLM outputs before making use of them.

That is politics. Not engineering.

Assigning a human to "check the output every time" and blaming them for the faults in the output is just assigning a scapegoat.

If you have to check the AI output every single time, the AI is pointless. You can just check immediately.

fhd2• 4 months ago

Well, I'd say there's two dimensions:

1. Check frequency (between every single time and spot checks).

2. Check thoroughness (between antagonistic in-depth vs high level).

I'd agree that, if you're towards the end of both dimensions, the system is not generating any value.

A lot of folks are taking calculated (or I guess in some cases, reckless) risks right now, by moving one or both of those dimensions. I'd argue that in many situations, the risk is small and worth it. In many others, not so much.

We'll see how it goes, I suppose.

Uhhrrr5• 4 months ago

The humans are not scapegoats, because they are capable of taking on responsibility.

There is a point to using LLMs. They can save time by doing a first pass. But when they do the last pass, disasters will follow.

AnimalMuppet3• 4 months ago

Well, attempts to engineer the brittleness out of human behavior have not worked, like, ever.

alfalfasprout• 4 months ago

When organizational incentives penalize NOT using AI and firing the bottom x% regularly then are you really surprised LLM outputs aren't being scrutinized?

Uhhrrr5• 4 months ago

Yes, because trusting LLM output is a great way to be in the bottom x%.

krupan8• 4 months ago

"A human, however, might have done further testing and made a more complete judgment call before sharing the information"

Because a human would have been fired for posting something that incorrect and dangerous

paxys3• 4 months ago

But funny enough the person who was responsible for setting up the bot will likely face no repercussions. In fact they will probably be rewarded for transitioning their team's workflows to AI.

pixl972• 4 months ago

I mean, only if it leads to embarrassment right off the bat.

If there is a year or two between writing your security fuck up and it being discovered the likelihood of repercussions drops significantly.

keybored• 4 months ago

A machine doesn’t need food, leisure time, or vacations. It doesn’t care.

It also doesn’t care.

jasonpeacock• 4 months ago

I'm concerned that someone had the permissions to make such a change without the knowledge of how to make the change.

And there was no test environment to validate the change before it was made.

Multiple process & mechanism failures, regardless of where the bad advice came from.

krupan8• 4 months ago

If you have to do all that, then what's the point of the AI? I'm joking, but I'm afraid many others say the same thing 100% seriously

marcosdumay• 4 months ago

As an article that was here recently claims, every verification you do in a chain increases the total time of your work by an order of magnitude. So, it's only work optimizing any productive task if you already removed most verifications.

Now, some people claim that you need to improve the reliability of your productive tasks so you can remove the verifications and be faster. Those people are, of course, a bunch of coward Luddites.

Fizzadar• 4 months ago

I’m predicting a wave of such incidents to start appearing over the next few months/years.

amelius• 4 months ago

How long until an AI puts all our personal data on the streets?

krupan8• 4 months ago

Very soon, and at this point I'm not sure even that would cure the delusions of the few who practically worship LLMs

esseph• 4 months ago

It's already there for a dollar to the right data broker. Could probably pull your doctor visit info from last week (example).

yieldcrv• 4 months ago

very misaligned! sprays bottle at mac mini

advisedwang• 4 months ago

AI can be used to move fast. So management expects us to move at that speed. AI can be used to move even faster if you don't check it's output. The ever ratcheting demand for faster output will make it infeasible to diligently check AI output all the time. AI errors being acted on without due care is inevitable.

AnimalMuppet3• 4 months ago

From Schlock Mercenary: "Oh, I love aiming. It's my very favorite thing to do before firing."

AI use without checking its output (at least at the moment) is firing without aiming. Sure, you can fire really fast. But who cares if you don't hit what you need to? The point wasn't to just shoot bullets, the point was to hit your target!

I mean, you might make a case that enough of them hit the target that shooting fast is a net win, and accept the occasional friendly fire incident. That might possibly be true. Or it might not. I'm not sure that everyone trying to run fast has really done the calculation, though.

ex-aws-dude• 4 months ago

This agent stuff is really making me lose respect for our industry

All the years of discussing programming/security best practices

Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter, determinism is going out the window, and its becoming standard practice to have bots on our local PC constantly running unknown shell commands

aeblyve2• 4 months ago

People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism.

My thinking is, this will increase the demand for backup and other resilience solutions.

_doctor_love2• 4 months ago

People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism.

This occurred long time ago comrade 'aeblyve.

polothesecond• 4 months ago

Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter

Is this new to people? I figured this out when I first entered the industry. The messages have never been particularly subtle.

exolymph• 4 months ago

Right? I was like when did software quality matter. Let alone code quality lol

Apocryphon• 4 months ago

Turns out all of the frenzy of the ZIRP era is piddling compared to what happens when ZIRP is taken away.

yoyohello13• 4 months ago

How can you respect an industry that doesn't respect itself?

testplzignore• 4 months ago

Our industry has never been serious about security. We all download and run unvetted code via package managers every day. At least now the insanity is out in the open. We won't change until Skynet fires off the nukes.

asdff4• 4 months ago

I keep getting so depressed thinking about the inevitable. Quite simply, humans can't scale or iteratively improve. We still need to eat, we still need to sleep, we can only think on one thread at a time basically, we take 20 years to get to our prime, which is a fleeting moment, while most of our lifespan is spent in a state of decline of capability. AI humanoid robot from the near future doesn't need to eat or sleep, can work 24/7, can compute thousands of processes in parallel, is the same fungible unit as any other humanoid robot, forever with some maintenance. Why justify a sustaining an inefficient human in that modern world? It is more profitable for the company to have humans go extinct and maximize planetary resource use to its fullest extent possible.

Seems we are digging our graves as a species and don't even realize it. I mean Sam Altman is already saying it taking 20 years to train a human is a Big Problem.

sunrunner2• 4 months ago

We won't change until Skynet fires off the nukes.

And then we won't need to, because at that point it will be too late.

dgxyz5• 4 months ago

This is never getting to skynet launching the nukes stage. It's not that clever and never will be.

Humans will kill us by it damage amplifying their worst characteristics.

Thus we'll die of a pandemic because some idiot LLM'ed up positive looking virology data when they were being too lazy to verify something. Everyone will trust it because they don't really care as long as it looks about right.

imglorp• 4 months ago

Yes and also the software industry has never been truly serious about security either: it's more of implied table stakes than an advertised product feature.

Also, customers outsource the risk to their vendors, so as long as there's someone to sue, nobody worries about doing it right. Ship it now and pay the lawyers later.

ponector• 4 months ago

It has never been serious about security, quality and performance. Only new sloppy features. And now everyone is bragging on LinkedIn how fast they create more slop: "Look, CC generated thousands lines of code for me! Approve and merge!"

superb_dev• 4 months ago

I’ve never had respect for the industry as a whole, only individuals within. There has a been a serious lack of rigor and professionalism in software engineering for as long as I’ve been a part of it

jihadjihad• 4 months ago

It's a slap in the face that we tack engineering onto it. A very small percentage of software engineering is as rigorous as actual engineering.

edf13• 4 months ago

It’s a nightmare… the problem is it’s far too easy for people to set these agents up - without understanding the security implications.

We’ve covered so many issues already on our blog (grith.ai)

wnevets• 4 months ago

The number of wasted hours spent talking about code quality and patterns has to be astronomical.

throwawaytea• 4 months ago

Don't worry, ai read all the transcripts and blogs and emails and has at least ingested some of the ethos in its outputs.

I self taught and wrote a small saas in 2017. Pays well enough to support me.

I'm building a new one using AI this year. I promise you, it's better built and more secure than what my previous still in use Saas is.

kstenerud• 4 months ago

I think it's batshit crazy. That's why I wrote yoloAI, so I could sandbox it up properly and control EXACTLY what comes out of that sandbox, diff style.

https://github.com/kstenerud/yoloai

I can't go back anymore. Going back to a non-sandboxed Claude feels like going back to a non-adblocked browser.

heisenbit• 4 months ago

Agents are providing to employees the long overdue benefits limited liability companies long enjoyed: Gambling with upside for themselves and other peoples downsides.

nickpinkston• 4 months ago

That's a fun insight. Have you / others written about this?

piva003• 4 months ago

We didn't collectively decided, we've got this forced down our throats to apply a novel tool to any imaginable situation because the execs got antsy about being left behind.

A truly absurd amount of capital was deployed which triggered a cascade of reactions by the people in charge of capital at other places. They are extremely anxious that everything will change under their feet, and if they don't start using as much as humanly possible of it right about now they die.

That's it.

The tools have definitely found some use, there's more to learn on how else they can be used, and maybe over time smart people will settle on ways to wrangle it well. The messaging from the execs though, is not that, it is "you'll be measured on how much you use this, we don't know for what or how, it's for you to figure out but don't dare to not use it".

I do understand their anxiety, their job is to not let their companies die, and make the most money as they can in the process; a seemingly major shift on the foundations of their orgs will cause fear.

But we have not collectively decided that it was safe, and good, to run rampant with these tools without caring for all that was learnt since software was invented...

dgxyz5• 4 months ago

This wasn't really forced on us.

The whole industry is like a fashion show and has been for a long time. This is just exceptionally stupid compared to moderately stupid things before. I see it ore that everyone's wearing pink feathered chicken suits because it's in fashion. If you don't wear a pink feathered chicken suit then you're a luddite scumbag who doesn't deserve the respect of your peers.

However some of us still have enough self-respect not to be seen dead in a pink feathered chicken suit. I mean I'm still pissed off at half the other stuff we do in the industry. I haven't even really looked at the chicken suits yet.

retinaros2• 4 months ago

no. openclaw wasnt forced by ceo's. it was forced by the same people who though there was money to be made in crypto then ICO then NFT. a bunch of scammers that bring negative value to the world

AnimalMuppet3• 4 months ago

We had it forced down our throats by CEOs and CTOs who thought that it would improve our productivity. Nobody forced it down their throats, though. Instead, they were seduced. They went willingly.

antonvs2• 4 months ago

The media isn’t helping. This wasn’t a “rogue AI”. It was a system that was given permission by a human operator.

We don’t say “a rogue plane killed 300 people today when it crashed into a mountain”.

The only difference in the AI case is that some people are attempting to shift blame for their incompetence into a computer system, and the media is going along with it because it increases clicks.

cassianoleal• 4 months ago

It was a system that was given permission by a human operator.

From TFA:

"But the agent also independently publicly replied to the question after analyzing it, without getting approval first."

zzgo• 4 months ago

cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter

I saw the sea change in 2008 when quality process got replaced with velocity and testing tasks. I've watched everything from Experian and health record data leaks to Windows 11 since that change. Software quality hasn't mattered for a long time.

dr_kiszonka• 4 months ago

I think it might be because we (or at least I) used to associate insecure actions with people, not computers. Computers should know better, right? Recently, I spotted that Opus 4.6 found config files for one of its tools and gave itself access to my whole filesystem. Similarly, Gemini CLI will rewrite itself if you let it.

lofaszvanitt• 4 months ago

The whole agent ecosystem is a ridiculous shitshow. All of this because you need to ASAP find something believable to sell your overinflated, bullshit machine to the masses. Otherwise the bubble will burst.

JuniperMesos• 4 months ago

Meta has never in its entire existence been known for caring about software quality.

a-french-anon• 4 months ago

M8... https://github.com/facebook/infer

mancerayder• 4 months ago

There's nothing "collectively" about it. I don't know what industry you work in, but in mine it's a top down mandate to use AI everywhere, tracked with KPIs, from the CEO down, and supported and pressured by companies like Amazon and MS.

We're the dummies that have to run around picking up dookies like a new puppy in the house.

brian_r_hall• 4 months ago

The frustrating part is watching all the careful thinking about reliability and failure modes get thrown out the window the second something new gets hyped. It's not even that people disagree with the principles, they just stop applying them.

worik3• 4 months ago

A rogue AI led to a serious security incident at Meta

The AI "led to" the incident , true. But do nt forget that this, like all similar incidents , is a human failure

AI is a tool with no agency. People make mistakes using it, thone mistakes are the responsibility of the humans

sunrunner2• 4 months ago

Why do we keep calling these things "agents" then? Or using the term "agentic"?

worik3• 4 months ago

Eternal optimism

butlike2• 4 months ago

Then the human should write the code.

falcor842• 4 months ago

AI is a tool with no agency

Claw AIs absolutely do have agency in the sense of being able to independently perform actions on their own, based on their "understanding" of a goal given by a "principal". I can't think of a better word than "agent" for that.

worik3• 4 months ago

They appear to have agency. But watch a while, they do not

LinkSpree• 4 months ago

Much like planes, people still would have a human pilot err and take the whole thing down than a robot go down even if it is statistically sound.

dmazin• 4 months ago

This is a lot less of a story than it seems.

It makes it sound like a rogue AI hacked Meta.

Instead, the "wild" thing here is that someone let an agent speak on their behalf with no review. The agent posted inaccurate instructions which someone else followed.

Those instructions lead to a brief gap in internal ACL controls, sounds like. I'm sorry, but given that the US government gave 14 year olds off incel Discords full access to Social Security data, this is not shocking by comparison.

To be clear, it is dumb and rude to let an agent speak on your behalf _without even reviewing it_.

This will eventually lead to a bigger snafu, of course. Security teams should control or at least review the agent permissions of every installation. Everyone is adopting this stuff, and a whole lot of people are going to set it up lazily/wrong (yolo mode at work).

BoneShard• 4 months ago

Yeah, a nothingburger for clicks.

skywhopper• 4 months ago

“Meta spokesperson Tracy Clayton said in a statement to The Verge that ‘no user data was mishandled’ during the incident.”

Wow, no mishandled user data? A striking change of standard operating procedure from Meta here.

Actually the later information in the story directly contradicts that, so The Verge probably shouldn’t have just quoted this line if their reporting is in opposition to it.

Regardless, this is one of the more insidious things about these tools. They often get minor but critical things wrong in the midst of mostly correct information. And people think they can analyze the data presented to them and make logical judgments, but that’s just not the case.

The article points out that “a human could have done the same thing” but, between the overly confident tone of the text generated by these tools, and the fact that weirdly people trust the LLM output more than they trust other humans (who generally admit or at least hint when they aren’t actually experts on a topic), it’s actually far worse when one of these bots gets something wrong.

ISL• 4 months ago

A central challenge for AI is understanding how accountability flows.

The language of this article is a great example, "... thanks to an AI agent that gave an employee inaccurate technical advice ...".

It should more-correctly read, " ... thanks to the people who made it possible for an AI agent to give an employee inaccurate technical advice ... ".

It is at our peril that we deem it acceptable to blame a black box for an error, especially at scale.

kkl• 4 months ago

"Had the engineer that acted on that known better, or did other checks, this would have been avoided."

I personally find "LLMs can do $THING poorly" and "LLMs can do $THING well" articles kinda boring at this point. But! I'm hopeful that stories like this will shift the industry's focus towards robustness instead of just short-term efficiency. I suspect many decision making and change management processes accidentally benefited from just being a bit slow.

[1] https://waffles.fun/amy.png

aussieguy1234• 4 months ago

More like Rogue Human, who didn't check the facts before taking the technical advice from the model at face value.

test20062• 4 months ago

test

test20062• 4 months ago

fds

gverrilla• 4 months ago

Skill issue.

A rogue AI led to a serious security incident at Meta

Related Stories