U of T researchers demonstrate AI worm could target any online device

https://www.utoronto.ca/news/u-t-researchers-demonstrate-ai-worm-could-target-any-online-device

by shscs911 • 2 months ago

152 44 2 months ago

utoronto.ca

throwaway81523• 2 months ago

Straumli blight?

e40• 2 months ago

Wrong zone.

smokel2• 2 months ago

The academic paper is here: https://arxiv.org/abs/2606.03811

It's not fully described how things work exactly, but apparently it does not transfer entire LLMs as part of the worm. Now that would be interesting :)

hamburgererror2• 2 months ago

In the abstract, what does it mean "the attacker's marginal cost per new infection is zero"?

amoshebb• 2 months ago

If you infect a machine with GPU enough to run the localLLM needed to steal another machine, you can let it burn tokens all day for free because whoever you stole the first one from will pay the electric bill.

a1o• 2 months ago

I think an approach could be to use some engineered security issue or however people build botnets, and give it some AI llm that is small and minimal but comes with instructions to download models from hugging face, and some other minimal prompts and descriptions of tools. Then it could use this to grow in infected computers and try find more capable and vulnerable computers to run better capable models and also devise some minimal communication between the different points of the botnet. Perhaps set itself a goal to dominate the biggest amount of compute and have some other goal. Would be curious to see what happens.

tiborsaas• 2 months ago

The abstract says:

The worm parasitically uses compromised machines to run open-weight large language models (LLMs) to sustain its reasoning, or extend its reach for further attacks.

smokel2• 2 months ago

Thanks for pointing that out. I scanned the paper and found that in their main experiments, they use a shared GPU resource and do not copy LLMs to target machines. Apparently they did other experiments in the ablation study where they did copy LLMs.

So it's even worse than I expected. The intended worm can spread through my thermostat, and when it reaches a GPU host, it can spread even harder. Fun times ahead.

m3kw9• 2 months ago

When the worm makes someone's machine start to sound like a leaf blower, you are found out.

hamburgererror2• 2 months ago

"Hey Honey look, I created Skynet!"

IshKebab2• 2 months ago

Did people doubt that this was theoretically possible? Seems self-evident to me. The interesting thing will be seeing it in the real world rather than in a controlled environment where they deliberately made all devices on the network have a known vulnerability.

acdha2• 2 months ago

There’s a difference between speculation and measurement, especially since you’d have people making arguments like saying that open models aren’t powerful/fast enough to work. Demonstrating this is a useful warning to everyone (most of the industry) who’s been slacking on internal defenses because they don’t think a well-resourced attacker will target them.

pixl972• 2 months ago

Honestly with some of the denialists here a terminator could kick down their door with lazgun in hand and they'd still tell you that AI can't do that.

And for the people that think that alignment is stupid, not training your AI to think twice about writing self spreading worms is a recipe for disaster after someone gets a token stealing, resource grabbing worm going.

IshKebab2• 2 months ago

It's not really a realistic measurement though is it?

jameslk• 2 months ago

Ah sweet, AI-made horrors beyond my comprehension

pfdietz• 2 months ago

I'm reminded of the universal computer viruses of Steve Barnes' SF stories, which ended up infecting people too.

criddell• 2 months ago

Doesn't Neal Stephenson's Snow Crash have a similar idea? IIRC, a computer infects human brains via language and sound.

In the 2004 Battlestar Galactica series, the explanation for why the Galactica was the only ship that survived a massive Cylon attack seems more and more likely. The ship was old and wasn't fully connected to the human's command and control systems and so the Cylon virus couldn't reach it.

crumpled• 2 months ago

"Memes" they called them in Snow Crash (1992).

In a very real way there ARE malicious AI agents working tirelessly to create and spread memes via language and sound to alter our brain software.

K0balt2• 2 months ago

Next up:

Obvious pattern of using ai to replace human reasoning in a proven methodology of malware distribution, C&C, and network infiltration obviously possible, say researchers.

Researchers use AI to create the torment nexus using commodity hardware, demonstrating the very real threat that AI could enable attackers to create torment nexus nodes using commodity hardware. “It wasn’t even that hard !“ says one researcher. Firmware available to qualified researchers who pinky swear that it will not be leaked.

Researchers set fire to laboratory with gasoline, killing seven volunteer victims, demonstrating that laboratory fires are a real risk and can carry significant consequences, especially when gasoline is involved.

Just because you can, doesn’t mean you should.

dijksterhuis• 2 months ago

this is part of the pro-active security loop. gotta demonstrate how it can break to figure out how to defend it.

our other choice is to let someone else figure it out in relative secrecy. then theyre able to cause a bunch of damage to a wide range of systems. with no defences for it. everyone would be scrambling around figuring out how to deal with it while the damage is going on. not good.

K0balt2• 2 months ago

I’m totally onboard with (and an adamant user of) proactive security. But there are classes of threats that are obviously possible, and the -concept- does not need validation.

Now , a control anchored experiment with balanced and unbalanced attacker/defender LLMs, that would be instructive and useful.

The idea that an LLM can deploy other LLMs on a machine it has access to is not research. Neither is the idea that an LLM can autonomously infiltrate and expand its access over a network. I have already done both, and it’s literally just a couple of prompts and a pile of reference docs. I use LLMs to deploy LLMs on my infrastructure, and I use LLMs to analyze security vulnerabilities on my networks, including deployment of access ladders on vulnerable machines. That is SOP, not research.

If they had used a pair of identical experiments, one that was exposed to an infiltrator LLM, and the other occupied by a defensive LLM and then exposed to the same threat, that would be an actual experiment.

As it is they just threw a roadflare on a dry field, and yup, Dry fields burn. They at least could have done it with and without recent rain.

They published only the obvious and dangerous part, none of the hypothetical or potentially useful part. Low effort, rush to publish.

malfist• 2 months ago

ANY online device? Even assuming AI can find vulnerabilities in every operating system, there's no indication that this is actually true beyond a "here's how it could work"

This is the same nonsense that lead to article saying researchers had created a wormhole when all they had done was draw one.

I have a microcontroller with an ROM disk (i.e., physically read only). You're telling me that an AI can find a way around the physics of not being able to mutate ROM and exploit it?

pixl972• 2 months ago

I mean, if it's online it has a network/wireless card and a TCP stack along with at least some amount of RAM, so yea, in theory unless the programming is perfect it could be exploited. Now, it's not going to be used to run AI, but could very well get used in a DDOS or something like that.

tosti• 2 months ago

I'm relieved my entire network consists of old computers and small, low power network devices. The only dedicated GPUs you'll find is discarded old junk in a cardboard junk box.

Being poor can have advantages :)

lisnake• 2 months ago

RAM-only viruses exist.

RetroTechie• 2 months ago

That's for when the uC is able to execute code from RAM. Some uCs can only execute code from a separate address space (Harvard architecture), that may not even be readable (let alone writeable) by the program itself.

On such a uC the "inject random payload" option would be out. Still possible might be denial-of-service attacks, altering or exfiltrating data etc.

But most stuff online are regular von Neumann machines where -in theory- anything is possible.

pbrum• 2 months ago

You cannot possibly be a full-time academic and your last name be "Papernot"!

moi2388• 2 months ago

Unless your field of academia is digital. Perhaps this is why he wanted to attack printers on the network.

_pdp_• 2 months ago

I made a tiny ai bug hunting harness (<4MB) that has everything (except the model obviously). It was designed for pentesting purposes where the tiny size matters to make it more portable between environments.

The intended purpose is not to be used as a worm but it does not take a genius to figure out that with small modifications such a thing could work relatively well - especially if it uses AI keys from compromised targets. Making the agent self-modifiable is relatively straightforward task and in fact I already did that in another project.

https://github.com/chatbotkit/rook

observationist• 2 months ago

Every Windows computer has a small rwkv model on it. Wouldn't be hard at all to get decent cpu performance from a tiny malicious harness, especially one that used the self-evolving skills features and open source models.

Malware is going to be crazy, people aren't ready for the revelation of how insecure and broken things are. Everything is held together by bubblegum, duct tape, and panicked engineers putting out fires.

mattvr• 2 months ago

Ah yes, viral AI gain-of-function research in a secure lab. What could go wrong?

xnorswap• 2 months ago

Yeah, lab leak is hard enough to contain with human viruses, but labs have well established protocols to prevent it happening.

Computing doesn't have good protocols except for air-gapping, we really just have lots of layers of best-effort detection, and billions of devices which mix data and instruction often in a careless fashion.

I used to not believe in the dangers of AI or the risk of internet-collapse from "rogue AI", but a genuine self-mutating virus could genuinely take down the internet and need an entirely new separate net. ( Or we'd discover if the current backbone actually has the power to break encryption to stop it. )

And this time, you can bet any new internet would be corporation captured. CompuServe and AOL failed because of the open internet, but we're a very different world now, governments would support the corporation led locked-down approaches for "safety".

I don't for a second believe the capability is actually there yet, but it's no longer unthinkable that such a thing could be created in a lab within a decade. Once out in the wild, there's a lot of idle compute out there to harness for self-improvement and spreading.

rtnplan• 2 months ago

In the paper they say that the worm uses either existing vulnerabilities that it has been trained on or new published vulnerabilities that it scrapes. 44% claimed success.

The paper is a bit silent on why a such a worm would need an LLM. It seems that brute forcing all known vulnerabilities, script kiddie style on each new machine is about the same.

But apparently that info is too dangerous to release ...

Retr0id• 2 months ago

AV/EDRs are kinda lame but "brute force all known vulnerabilities" is definitely something they can detect.

alentodorov• 2 months ago

sorry, but i had to do this…

is this papernot’s first paper?

computerphage• 2 months ago

Palisade Research demonstrated this capability, including fully copying model weights, on May 7th, 2026

https://palisaderesearch.org/blog/self-replication

https://arxiv.org/abs/2605.06760

arm32• 2 months ago

Of course this is possible, but to use a botnet for intelligence compute is going to be slow. That's one thing we have going for us—it's going to take a long time for this bug to fumble through your network. But rest assured, eventually, it will pwn you.

hulitu• 2 months ago

could

Come back when it does.

U of T researchers demonstrate AI worm could target any online device

Related Stories