Zenbleed

https://lock.cmpxchg8b.com/zenbleed.html

by loeg • 3 years ago

1283 361 3 years ago

lock.cmpxchg8b.com

dtx13• 3 years ago

AMD have released an microcode update for affected processors. Your BIOS or Operating System vendor may already have an update available that includes it.

Yes, I love flashing BIOS...

edit nvm, Microcode can get updated via system updates.

Night_Thastus4• 3 years ago

To be fair, flashing the bios isn't nearly as bad on most modern systems.

Put the file on a USB drive, plug it in, restart and go into the bios, look for the flashing utility, select the file, done. As long as the machine is on a UPS in case of disaster, everything's accounted for.

dtx13• 3 years ago

From my experience: Better have a > 32gig USB Flash drive, everything else doesn't work (MSI) and I don't have an UPS so it's always quite an exciting experience. Especially since Motherboard Manifactures save almost a whole dollar by not having a display ouputting anything. So it's blinkenlights and hope for the best

baq• 3 years ago

Sometimes there's even a backup BIOS die available, so yeah, bricking is now much harder than in the past.

yyyk• 3 years ago

Often the BIOS will allow reading the update file from EFI partition, so there's no need for the USB drive.

ls6122• 3 years ago

My new computer takes a while to POST (z690 with ddr5 smh) so it’s basically been continuously either on or in sleep since I built it 18 months ago and I’ve had an unexpected shutdown due to power loss once in that time according to the Event log. I think the risk of losing power while flashing the bios is very small in real life unless you are stuck in a place with third world electricity infrastructure.

astrange2• 3 years ago

Mine loses its settings when you update the BIOS, so your fan curves go away.

deaddodo• 3 years ago

Microcode updates haven't been managed in-BIOS for over a decade now. If you use Linux, you'll usually see them released as some package like "intel-microcode" or "amd-microcode".

Even EFI updates rarely are very intrusive or dangerous, and can also be handled by the Operating System via an update.

mrpippy2• 3 years ago

They are managed both ways. I think updating in BIOS is preferable to ensure no CPU parameters change while (some part) of the kernel has already initialized.

But of course BIOS updates have many downsides and often stop after a few years.

causi• 3 years ago

AMD Ryzen 5000 Series Processors with Radeon Graphics

Does this mean Ryzen CPUs without integrated graphics are fine?

gruez• 3 years ago

No, it's all Zen 2 CPUs, which include both desktop CPUs (with or without integrated graphics, laptop CPUs, and server CPUs. The reason why the product list is so confusing is that AMD reuses architectures across generations. You'd think that all ryzen 5000 series CPUs have the same microarchitecture, but they don't). It's much easier to consult this list instead: https://en.wikipedia.org/wiki/Zen_2#Products

paulmd5• 3 years ago

FYI this list isn't exhaustive. And I went to recommend the wikichips link and it's not exhaustive either.

https://en.wikichip.org/wiki/amd/microarchitectures/zen_2#Al...

Both of them are missing the newer 7000-family products with Zen2 like 7520U etc.

https://www.amd.com/en/products/apu/amd-ryzen-5-7520u

https://www.amd.com/en/products/apu/amd-ryzen-3-7320u

https://www.amd.com/en/products/apu/amd-athlon-gold-7220u

formerly_proven2• 3 years ago

No this means AMD's numbering scheme is intentionally obtuse. This has nothing to do with graphics, but with the CPU core, Zen 2.

lgl• 3 years ago

The only series 5000 cpu's that are still using Zen2 architecture are apparently the 5300U, 5500U and 5700U, which all use socket FP6 (mobile/embedded).

So I'm guessing it shouldn't affect any of the more recent and very popular Zen3 cpus like the 5600, 5700 etc. I personally own a 5600, which are a great bang for buck.

paulmd5• 3 years ago

Lucienne (5700U/5500U/5300U) are the only Zen2s in the 5000 series at present (afaik), but AMD continues to re-use the Zen2 architecture in the 7000 series (7520U, etc), as well as many semicustom products like Steam Deck.

It's in rather a sweet-spot as far as performance-power-area, so this isn't entirely a bad thing. Zen3's main innovation was unifying the CCXs/caches, but if you only have a 4C, or you want to be able to power-gate a CCX (and its attendant IF links/caches) down entirely, Zen2 does that better, and it's slightly smaller. We'll be seeing Zen2 products for years to come, most likely.

lopkeny12ko• 3 years ago

Relevant snippet:

This technique is CVE-2023-20593 and it works on all Zen 2 class processors, which includes at least the following products:

    AMD Ryzen 3000 Series Processors
    AMD Ryzen PRO 3000 Series Processors
    AMD Ryzen Threadripper 3000 Series Processors
    AMD Ryzen 4000 Series Processors with Radeon Graphics
    AMD Ryzen PRO 4000 Series Processors
    AMD Ryzen 5000 Series Processors with Radeon Graphics
    AMD Ryzen 7020 Series Processors with Radeon Graphics
    AMD EPYC “Rome” Processors

kevin_thibedeau• 3 years ago

FYI, Ryzen 3000 APUs aren't Zen 2.

neogodless2• 3 years ago

AMD Ryzen 3000 Series Processors

The above are desktop. If they meant APUs, it would list "Ryzen 3000 Series Processors with Radeon Graphics."

timw4mail• 3 years ago

They are Zen+, aren't they?

tremon5• 3 years ago

Do they mean "only confirmed on Zen2", or is the problem definitely confined to only this architecture?

Is it likely that this same technique (or similar) also works on earlier (Zen/Zen+) or later (Zen3) cores, but they just haven't been able to demonstrate it yet?

paulmd5• 3 years ago

It's Tavis Ormandy, and he reported it to AMD, so one would assume they tried it on related hardware and it's not working.

rincebrain• 3 years ago

At least the stock exploit code he provided said "nope I can't get shit to leak" on my 5900X.

Arnavion• 3 years ago

Doesn't repro on 2920x (Zen+).

zacmps• 3 years ago

I tested on a Zen 3 Epyc and wasn't able to get the POC to work, so I think it probably is just Zen 2.

justinclift• 3 years ago

Whew, my 5600X looks like it avoided this one too. :)

ye-olde-sysrq2• 3 years ago

So are Ryzen 5000's without Radeon not vulnerable? I guess said processors are zen 3?

I have an "AMD Ryzen 9 5950x Desktop Processor" which appears to be Zen 3. I think I'm good?

(Not that I'm running untrusted workloads, but yknow, fortune favors the prepared)

Tuna-Fish2• 3 years ago

You are likely frequently running untrusted workloads. As javascript in a browser. I don't know about this one, but at least meltdown was fully exploitable from js.

But yes, you are fine, 5950x is Zen3.

eugene3306• 3 years ago

and how about playstation 5 ?

and also xbox and that thing from valve?

javajosh3• 3 years ago

I mean, the PS5 is running a Zen 2 processor [0] so I would assume it's vulnerable. In general I would assume that AAA games are safe. Websites and smaller games made by malefactors will be the issue. (Note that AAA game makers have little interest in antagonizing the audience, OTOH they also will push limits to install anti-cheat mechanisms. On balance I'd trust them.)

0 - https://blog.playstation.com/2020/03/18/unveiling-new-detail...

winrid2• 3 years ago

Looks like my 2700x narrowly misses this one, assuming 7020 series is affected and not 7000 series.

_flux2• 3 years ago

The wording "at least" suggests the list might not be exhaustive.

loeg14• 3 years ago

Yeah -- Ryzen 2700x is Zen+, not Zen 2. Current understanding is that Zen+ is not affected.

mrpippy2• 3 years ago

It feels like not-a-coincidence that OpenBSD added AMD microcode loading in the last 3 days.

https://news.ycombinator.com/item?id=36838511

hammock• 3 years ago

Explain that like I’m 5?

laverya• 3 years ago

The patch for this exploit is to load AMD's updated microcode.

dralley• 3 years ago

This may or may not also be relevant (I actually have no idea): https://www.phoronix.com/news/Fedora-Server-Alert-FW-Updates

sounds4• 3 years ago

The site is getting hugged to death. https://web.archive.org/web/20230724143835/https://lock.cmpx...

AdmiralAsshat2• 3 years ago

And now we've hugged the archive to death. Nice job!

loeg14• 3 years ago

The original still loads (eventually) for me. YMMV.

nevi-me• 3 years ago

XMMV or ZMMV could also apply

ksec• 3 years ago

It is a simple static HTML page, how is it possible in 2023 a static site could be hugged to death. In most cases HN traffic barely hits 100 page view per second.

jedberg• 3 years ago

It's a security writeup so it's probably run by a security expert who is not an expert at running high traffic websites. Most likely there is something on the page that causes a database hit. Possibly the page content itself.

taviso4• 3 years ago

welp, that's unfortunate indeed.

It's a single-core 128 MB VPS, which seemed fine for my boring static html articles. I guess I underestimated the interest.

marcus0x62• 3 years ago

I imagine they are also getting traffic from sources other than HN.

winrid2• 3 years ago

100rps for most articles. I bet this is at least double that, and he's using apache which by default I think is still thread per connection.

ComputerGuru2• 3 years ago

Faster link: https://archive.is/QAwvQ

ItsTotallyOn3• 3 years ago

What does this allow the attacker to do? Steal data? The post isnt very clear.

sounds4• 3 years ago

Huh. The very first line seems pretty clear:

   > If you remove the first word from the string "hello world", what should
   > the result be? This is the story of how we discovered that the answer
   > could be your root password!

Can you please expand on your question?

bananapub• 3 years ago

I assume they meant "what does this do in normal vulnerability discussion terms", I don't know why tavis didn't just say "arbitrary memory read across processes" or whatever.

ItsTotallyOn3• 3 years ago

does it require physical access to the machine?

rkrzr• 3 years ago

It allows the attacker to steal data like e.g. your (root) password.

tremon5• 3 years ago

Only while it's stored unencrypted in memory, right?

beebmam• 3 years ago

It is very clear, you just didn't read it.

We now know that basic operations like strlen, memcpy and strcmp will use the vector registers - so we can effectively spy on those operations happening anywhere on the system! It doesn’t matter if they’re happening in other virtual machines, sandboxes, containers, processes, whatever!
This works because the register file is shared by everything on the same physical core. In fact, two hyperthreads even share the same physical register file.
It turns out that mispredicting on purpose is difficult to optimize! It took a bit of work, but I found a variant that can leak about 30 kb per core, per second.
This is fast enough to monitor encryption keys and passwords as users login!

hinkley3• 3 years ago

Literally the intro says it might contain the root password.

TLDR: The vector registers this bug affects are used for string functions like strcmp, so anything could get loaded into them, including passwords.

timmaxw• 3 years ago

It allows the attacker to eavesdrop on the data going through operations like strcmp(), memcpy(), and strlen(). (These are the standard functions in C for working with strings; and many higher-level languages use them under the hood.) It works on any function that uses the XMM/YMM/ZMM registers.

It's stochastic; the attacker randomly gets data from whatever happens to be using the XMM/YMM/ZMM registers at the time. So if the attacker could eavesdrop in the background constantly, they might eventually see a password. Or they might be able to trigger some system code that processes your password, then eavesdrop for the next few milliseconds.

The attacker needs to run code on your machine. Unclear if running code in a web browser is sufficient or not. It requires an unusual sequence of machine instructions, which isn't necessarily possible in JS/WASM, but 'sounds' says they did it: https://news.ycombinator.com/item?id=36849767

hprotagonist• 3 years ago

ah, not the color theme. Hamming distance strikes again!

https://kippura.org/zenburnpage

heywhatupboys5• 3 years ago

I knew there were color schemes for the color blind.

Schemes for the blind are news to me though

0xbadcafebee2• 3 years ago

This is super cool. This exploit will be one of the canonical examples that just running something in a VM does not mean it's safe. We've always known about VM breakout, but this is a no-breakout massive exploit that is simple to execute and gives big payoffs.

Remember: just because this one bug gets fixed in microcode doesn't mean there's not another one of these waiting to be discovered. Many (most?) 0-days are known about by black-hats-for-hire well before they're made public.

CPU vulnerabilities found in the past few years:

  https://en.wikipedia.org/wiki/Meltdown_(security_vulnerability)
  https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)
  https://aepicleak.com/
  https://en.wikipedia.org/wiki/Software_Guard_Extensions#SGAxe
  https://en.wikipedia.org/wiki/Software_Guard_Extensions#LVI
  https://en.wikipedia.org/wiki/Software_Guard_Extensions#Plundervolt
  https://en.wikipedia.org/wiki/Software_Guard_Extensions#MicroScope_replay_attack
  https://en.wikipedia.org/wiki/Software_Guard_Extensions#Enclave_attack
  https://en.wikipedia.org/wiki/Software_Guard_Extensions#Prime+Probe_attack
  https://www.vusec.net/projects/crosstalk/
  https://en.wikipedia.org/wiki/Hertzbleed
  https://www.securityweek.com/amd-processors-expose-sensitive-data-new-squip-attack/

cmrdporcupine• 3 years ago

In the end, I'm thinking most of these are related to branch prediction?

It strikes me that it's either that branch prediction is so inherently complex enough it's always going to be vulnerable to this and/or it just so defies the way most of us intuitively think about code paths / instruction execution that it's hard to conceive of the edge cases until too late?

At what point does the complexity of CPU architectures become so difficult to reason about that we just accept the performance penalty of keeping it simpler?

c7DJTLrn3• 3 years ago

We demanded more performance and we got what we demanded. I doubt manufacturers are going to walk back on branch prediction no matter how flawed it is. They'll add some more mitigations and features which will be broken-on-arrival.

Tuna-Fish2• 3 years ago

At what point does the complexity of CPU architectures become so difficult to reason about that we just accept the performance penalty of keeping it simpler?

Never for branch prediction. It just gets you too much performance. If it becomes too much of a problem, the solution is greater isolation of workloads.

paulmd5• 3 years ago

More generally, most of them are related to speculative execution, where branch mis-prediction is a common gadget to induce speculative mis-execution.

Speculation is hard, it's sort of akin to the idea of introducing multithreading into a program, you are explicitly choosing to tilt at the windmill of pure technical correctness because in a highly concurrent application every error will occur fairly routinely. Speculation is great too, in combination with out-of-order execution it's a multithreading-like boon to overall performance, because now you can resolve several chunks of code in parallel instead of one at a time. It's just also a minefield of correctness issues, but the alternative would be losing something like the equivalent of 10 years of performance gains (going back to like ARM A53 performance).

The recent thing is that "observably correct" needs to include timings. If you can just guess at what the data might be, and the program runs faster if you're correct, that's basically the same thing as reading the data by another means. It's a timing oracle attack.

(in this case AMD just fucked up though, there's no timing attack, this is just implemented wrong and this instruction can speculate against changes that haven't propagated to other parts of the pipeline yet)

The cache is the other problem, modern processors are built with every tenant sharing this single big L3 cache and it turns out that it also needs to be proof against timing attacks for data present in the cache too.

loeg14• 3 years ago

Speculative execution, not branch prediction.

0cf8612b2e1e• 3 years ago

If you pin the VM to a different core/CPU, would that do anything to mitigate? Or are the OS affinity guarantees not that strong?

rcxdude• 3 years ago

At what point does the complexity of CPU architectures become so difficult to reason about that we just accept the performance penalty of keeping it simpler?

Basically never for anything that's at all CPU-bound, that growth in complexity is really the only thing that's been powering single-threaded CPU performance improvements since Dennard scaling stopped in about 2006 (and by that time they were already plenty complex: by the late 90s and early 2000's x86 CPUs were firmly superscalar, out-of-order, branch-predicting and speculative executing devices). If your workload can be made fast without needing that stuff (i.e. no branches and easily parallelised), you're probably using a GPU instead nowadays.

kiririn• 3 years ago

You can rent one of the Atom Kimsufi boxes (N2800) to experience first hand a cpu with no speculative execution. The performance is dire, but at least it hasn’t gotten worse over the years - they are immune to just about everything

vetrom3• 3 years ago

Theres VLIW/'preprediction'/some other technical name I forget for infrastructures which instead ask you to explicitly schedule instruction/data/branch prediction. If I remember, the two biggest examples I can think of were IA64 and Alpha. I wanna think HP-PA did the same but I'm not clear on that one.

For various reasons, all these infras eventually lost out in the market due to market pressure (and cost/watt/IPC, I guess).

c7DJTLrn3• 3 years ago

Running untrusted code whether in a sandbox, container, or VM, has not been safe since at least Rowhammer, maybe before. I believe a lot of these exploits are down to software and hardware people not talking. Software people make assumptions about the isolation guarantees, hardware people don't speak up when said assumptions are made.

saagarjha7• 3 years ago

That is not true in this case. It's just a CPU bug; not even a side channel.

insanitybit5• 3 years ago

Hardware people are the ones making those promises, so I don't think that's right at all. And Rowhammer is a way overstated vulnerability - there are all sorts of practical issues with it, especially if you're on modern, patched hardware.

zamadatix7• 3 years ago

In the case of the VM won't registers be wiped when entering/exiting the VM?

loeg14• 3 years ago

The problem is the freed entries in the register file. A VM can, at least, use this bug to read registers from a non-VM thread running on the adjacent SMT/HT of a single physical core. I suspect a VM could also read registers from other processes scheduled on the same SMT/HT.

crote2• 3 years ago

The problem is that the logical registers don't have a 1:1 relation to the physical registers.

For example, let's imagine a toy architecture with two registers: r0 and r1. We can create a little assembly snippet using them: "r0 = load(addr1); r1 = load(addr2); r0 = r0 + r1; store(addr3, r0)". Pretty simple.

Now, what happens if we want to do that twice? Well, we get something like "r0 = load(addr1); r1 = load(addr2); r0 = r0 + r1; store(addr3, r0); r0 = load(addr4); r1 = load(addr5); r0 = r0 + r1; store(addr6, r0)". Because there is no overlap between the accessed memory sections, they are completely independent. In theory they could even execute at the same time - but that is impossible because they use the same registers.

This can be solved by adding more physical registers to the CPU, let's call them R0-R6. During execution the CPU can now analyze and rewrite the original assembly into "R1 = load(addr1); R4 = load(addr4); R2 = load(addr2); R5 = load(addr5); R3 = R1 + R2; R6 = R4 + R5; store(addr3, R3); store(addr6, R6)". This means we can now start the loads for the second addition before the first addition is done, which means we have to wait less time for the data to arrive when we finally want to actually do the second addition. To the user nothing has changed and the results are identical!

The issue here is that when entering/exiting a VM you can definitely clear the logical registers r0&r1, but there is no guarantee that you are actually clearing the physical registers. On a hardware level, "clearing a register" now means "mark logical register as empty". The CPU makes sure that any future use of that logical register results in it behaving as if it has been clear, but there is no need to touch the content of the physical register. It just gets marked as "free for use". The only way that physical register becomes available again is after a write, after all, and that write would by definition overwrite the stale content - so clearing it would be pointless. Unless your CPU misbehaves and you run into this new bug, of course.

Bluecobra2• 3 years ago

Yup! I worked at a few companies that would co-mingle Internet facing/DMZ VMs with internal VMs. When pointing this out and recommending we should airgap these VMs to it's own dedicated hypervisor it always fell on deaf ears. Jokes on them I guess.

Kwpolska• 3 years ago

I'm pretty sure AWS/Azure/GCP don’t assign separate boxes to every customer, and somehow they’re fine.

phendrenad2• 3 years ago

The problem is, VMs aren't really "Virtual Machines" anymore. You're not parsing opcodes in a big switch statement, you're running instructions on the actual CPU, with a few hardware flags that the CPU says will guarantee no data or instruction overlap. It promises! But that's a hard promise to make in reality.

msla• 3 years ago

This is because VM means two different things and has for a long time:

IBM's VM was and is a hypervisor. It dates to the mid 1960s, in the form of CP-40, and it didn't run opcodes in software, but in hardware.

https://en.wikipedia.org/wiki/IBM_CP-40

p-code machines, which interpret bytecode, date back almost as far, such as the O-code machine for BCPL.

https://en.wikipedia.org/wiki/BCPL

Getting people to distinguish between these concepts is probably a lost cause.

MuffinFlavored• 3 years ago

you're running instructions on the actual CPU

Just how many times is the average operating system workload (with or without a virtual machine also running a second average operating system workload) context switching a second?

Like... unless I'm wrong... the kernel is the main process, and then it slices up processes/threads, and each time those run, they have their own EAX/EBX/ECX/ESP/EBP/EIP/etc. (I know it's RAX, etc. for 64-bit now)

How many cycles is a thread/process given before it context switches to the next one? How is it managing all of the pushfd/popfd, etc. between them? Is this not how modern operating systems work, am I misunderstanding?

eru3• 3 years ago

The big switch statement wouldn't necessarily protect you either.

stingraycharles• 3 years ago

Isn’t the typical solution here to pin each VM to certain CPUs / cores?

tenebrisalietum• 3 years ago

What you're describing (switch statement) is emulation, not virtualization.

trebligdivad• 3 years ago

The comparison to Meltdown/Spectre are a bit misleading though - they were a whole new form of attack based on timing where the CPU did exactly what it should have done; This zenbleed case is a good old fashioned bug though - data in a register that shouldn't be.

stcredzero• 3 years ago

this is a no-breakout massive exploit that is simple to execute and gives big payoffs

Wouldn't we be able to avoid the "big payoffs" of no-breakout exploits if we had specialized hardware handle the secrets?

api• 3 years ago

I’m quite surprised there hasn’t been a cloud apocalypse yet where something just runs rampant through AWS or something.

jacquesm2• 3 years ago

It's still early days for the cloud. I'm pretty sure such a thing will happen sooner or later.

alecco• 3 years ago

Couldn't VMs zero all registers when switching? It shouldn't be much more latency than a typical context switch. Also purge CPU cache to be safe.

sgerenser• 3 years ago

It does zero registers when context switching, but only the "logical" registers, not the physical ones, as described in this comment: https://news.ycombinator.com/item?id=36855266

ComputerGuru2• 3 years ago

No details on the performance impact of the microcode update. Presumably it disables speculative execution of vzeroupper?

hinkley3• 3 years ago

Or adds a guard.

They mention perf issues for the workaround but they're notably absent from the microcode commentary.

I wonder what this is going to do to the new AMD hardware AWS is trying to roll out, which is supposed to be a substantial performance bump over the previous generation.

infinityio• 3 years ago

It looks like this is a Zen 2-only exploit, so it shouldn't have any impact - AWS are likely already running hardware that isn't vulnerable to this

jeffffff• 3 years ago

shouldn't have any effect, the new amd hardware is zen 4 and this only affects zen 2

wzdd• 3 years ago

Really lovely writeup. I liked the discussion of determining how can you tell if a randomly-generated program performed correctly. The obvious approach is to just run it on an "oracle" -- another processor or simulator -- and see if it behaves the same way. But if you're checking for microarchitectural effects with tight timing windows you can also write the same program with various stalls, fences, nops and so on -- things which shouldn't affect the output (for single-threaded code) but which will result in the CPU doing significantly different things microarchitecturally. That way the CPU can be its own oracle.

weinzierl• 3 years ago

This part was super interesting, especially the differences between fuzzing software and hardware. I also liked the chicken bit.

eric__cartman4• 3 years ago

This is incredibly scary. On my Zen 2 box (Ryzen 3600) logging the output of the exploit running as an unprivileged user while copying and pasting a string into a text editor in the background (I used Kate), resulted in pieces of the string being logged into the output of zenbleed. And this is after a few seconds of runtime mind you, not even a full minute.

Thankfully the exploit is highly dependent on a specific asm routine so exploiting it from JS or WASM in a browser should be extremely difficult. Otherwise a nefarious tab left open for hours in the background could exfiltrate without an issue.

I'm eagerly waiting for Fedora maintainers to push the new microcode so the kernel can update it during the boot process.

loeg14• 3 years ago

Thankfully the exploit is highly dependent on a specific asm routine so exploiting it from JS or WASM in a browser should be extremely difficult. Otherwise a nefarious tab left open for hours in the background could exfiltrate without an issue.

At least one commentor here claims to be able to reproduce this with javascript: https://news.ycombinator.com/item?id=36849767 .

IshKebab• 3 years ago

A very bold claim with zero evidence.

zekica• 3 years ago

I tried on my zen 2 box, and the same things works even when the exploit is run in a KVM.

kludge413• 3 years ago

How do you build the POC? I get "No such file or directory" and error 127 on Ubuntu.

eric__cartman4• 3 years ago

I had to run make on the uncompressed folder. Perhaps the build-essential package doesn't come with NASM in Ubuntu? I'll need a bit more info on the error if you want me to try and help you :)

slappy72• 3 years ago

Thankfully the exploit is highly dependent on a specific asm routine so exploiting it from JS or WASM in a browser should be extremely difficult.

I assume that once/if a method is found it will be applicable broadly though. At the same time, hopefully software patches in V8 and SpiderMonkey will be able to mitigate this further and sooner.

But a JS exploit would require some way to exfiltrate data and presumably doing that would be quite difficult to hide entirely.

darkclouds• 3 years ago

Nice catch!

If you can’t apply the update for some reason, there is a software workaround: you can set the chicken bit DE_CFG[9].

It reminds me of the compiler switches which can alter the way code at different levels (global, procedure, routine) can access variables declared at different levels and the change in scope that ensues.

Maybe some of this HW caching should be left to the coders.

artisanspam2• 3 years ago

Why does disabling SMT not fully prevent this? I don't know the details of Zen 2 architecture, but register files are usually implemented as SRAM on the CPU-die itself. So unless the core is running SMT, I don't understand how another thread could be accessing the register file to write a secret.

wbl2• 3 years ago

Because the context switch only affects architectural state not microarchitectural state.

artisanspam2• 3 years ago

Yes I understand that but I was struggling to think of a sequence of instructions that would cause this secret leaking on a single thread.

But a simple example is `vzeroupper` followed by anything that writes a secret to the same register file entry would be leaked on a subsequent flush.

adrian_b3• 3 years ago

Because unless you pin the threads to certain CPU cores (e.g. in Linux by using the taskset command, or in Windows by using the Set Affinity command in Task Manager), they are migrated very frequently between cores.

So even with SMT disabled, each core will execute sequentially many threads, switching every few milliseconds from one thread to another, and each context switch does not modify the hidden registers, it just restores the architecturally visible registers.

dontlaugh• 3 years ago

Pinning doesn’t help either, since there will always be more threads than cores. Scheduling all those threads and even blocking on IO will cause context switches.

jrmg3• 3 years ago

AMD have released an microcode update for affected processors. Your BIOS or Operating System vendor may already have an update available that includes it.

I don’t really understand how CPU microcode updates work. If I’m keeping Ubuntu up to date, will this just happen automatically?

naikrovek2• 3 years ago

no.

microcode changes are provided to the CPU at boot time and are only valid early in the boot process. the machine UEFI/BIOS must apply them.

tremon5• 3 years ago

Linux can (and does) apply microcode patches during kernel boot.

tremon5• 3 years ago

If you already have the package amd64-microcode installed (highly likely), then yes it will be updated automatically.

https://packages.ubuntu.com/search?keywords=amd64-microcode

jrmg3• 3 years ago

Great, thanks.

Sort of weirds me out that my OS can just silently update my CPU - I didn’t realize I was giving it that level of control… I guess it’s good vs the alternative of no-one actually updating for exploits like his though.

kzrdude7• 3 years ago

the security patch has now shown up in ubuntu and is visible in that packages listing, 3.20191218.1ubuntu2.1

sdht0• 3 years ago

https://www.cyberciti.biz/faq/install-update-intel-microcode...

cratermoon• 3 years ago

This link seems hugged to death, so here's an alternate source: AMD 'Zenbleed' Bug Allows Data Theft From Zen 2 Processors, Patches Coming: <https://www.tomshardware.com/news/zenbleed-bug-allows-data-t...>

ItsTotallyOn3• 3 years ago

This story has comments from AMD, too.

codedokode• 3 years ago

I don't understand how a microcode update could fix this. I assume microcode is used for slow operations like triginometric functions, and doesn't affect how registers are allocated or renamed. Or does the update simply disables some optimizations using "chicken bits"? And by the way, is there a list of such bits?

sebzim4500• 3 years ago

Everything a modern CPU runs is microcode. There are a few x86 instructions that translate to a single microcode instruction, but most are translated to several.

blibble• 3 years ago

this hasn't been the case since the 486

wolf550e3• 3 years ago

The designers leave themselves an ability to override any instruction using the microcode so they can patch any instruction. They don't use the microcode only to implement complex instructions that require loops.

kristjank• 3 years ago

At least it's fixed in microcode, unlike some recent exploits (Spectre and Meltdown come to mind)

gavinhoward2• 3 years ago

Off-topic question, but can some experts tell me why it is safe for `strlen()` and friends to use vector instructions when they can technically read out of bounds?

loeg14• 3 years ago

Essentially because memory mappings and RAM work at page granularity, rather than bytes. If a read from in-bounds in a page isn't going to fault, a read later in the same page isn't going to fault either (even if it is past the end of the particular object).

You can see this in glibc's implementation, which checks for crossing page boundaries: https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/x86... (line ~68)

gavinhoward2• 3 years ago

Ah, so that's why there is special code in Valgrind to handle glibc and friends!

nemetroid2• 3 years ago

The README in the tar file with the exploit (linked at "If you want to test the exploit, the code is available here") contains some more details, including a timeline:

- `2023-05-09` A component of our CPU validation pipeline generates an anomalous result.

- `2023-05-12` We successfully isolate and reproduce the issue. Investigation continues.

- `2023-05-14` We are now aware of the scope and severity of the issue.

- `2023-05-15` We draft a brief status report and share our findings with AMD PSIRT.

- `2023-05-17` AMD acknowledge our report and confirm they can reproduce the issue.

- `2023-05-17` We complete development of a reliable PoC and share it with AMD.

- `2023-05-19` We begin to notify major kernel and hypervisor vendors.

- `2023-05-23` We receive a beta microcode update for Rome from AMD.

- `2023-05-24` We confirm the update fixes the issue and notify AMD.

- `2023-05-30` AMD inform us they have sent a SN (security notice) to partners.

- `2023-06-12` Meeting with AMD to discuss status and details.

- `2023-07-20` AMD unexpectedly publish patches, earlier than an agreed embargo date.

- `2023-07-21` As the fix is now public, we propose privately notifying major distributions that they should begin preparing updated firmware packages.

- `2023-07-24` Public disclosure.

sedatk3• 3 years ago

AMD unexpectedly publish patches, earlier than an agreed embargo date.
As the fix is now public, we propose privately notifying major distributions that they should begin preparing updated firmware packages.

AMD had to drop the ball somewhere didn't it.

klyrs• 3 years ago

It's good that they published patches early, isn't it?

nextaccountic• 3 years ago

Something is a little unclear to me. Does https://archlinux.org/packages/core/any/amd-ucode/

amd-ucode 20230625.ee91452d-5

last updated 2023-07-25 11:48 UTC

Contains the microcode update that addresses this?

https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin... says that the fixed version is 2023-07-18, but the amd-ucode version in Arch is 20230625.. but it was last updated in 2023-07-25..

My guess is that this is still getting the 20230625 firmware, per the PKGBUILD at https://gitlab.archlinux.org/archlinux/packaging/packages/li...

Which contains those lines

_tag=20230625

source=("git+https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin...")

I suppose that it isn't up to date and thus Arch Linux is still vulnerable, right?

edit:

but actually there's two commits in the _backports array (which contains cherry-picked commits) that was last edited 20 hours ago

https://gitlab.archlinux.org/archlinux/packaging/packages/li...

Which is 0bc3126c9cfa0b8c761483215c25382f831a7c6f and b250b32ab1d044953af2dc5e790819a7703b7ee6

And b250b32ab1d044953af2dc5e790819a7703b7ee6 appears to be the commit I linked ealier at git.kernel.org so hopefully up-to-date Arch is not vulnerable to zenbleed

nemetroid2• 3 years ago

From what I can tell, 20230625 is the latest tagged release of of the linux-firmware repo: https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin...

Either way, as noted elsewhere in the comments, only the Rome CPU series has received updated microcode with fixes. All other Zen 2 users need the fix that was released as part of Linux 6.4.6: https://lwn.net/Articles/939102/

(which has been built and packaged for Arch)

sedatk3• 3 years ago

I didn't expect it to as it's Zen3, but still tried: doesn't repro on my 5950X.

href• 3 years ago

Can anyone explain the `wrmsr -a 0xc0011029 $(($(rdmsr -c 0xc0011029) | (1<<9)))`? It seems to help on my system, but I don't understand what it does, and I don't know how to unset it.

mmastrac• 3 years ago

This sets the chicken bit: https://www.phoronix.com/news/Linux-AMD-Spectral-Chicken

taviso4• 3 years ago

An msr is a "model specific register", a chicken bit can configure cpu features.

They don't persist across a reboot, so you can't break anything. You can undo what you just did without a reboot, just use `... & ~(1 << 9)` instead (unset the bit instead of set it).

mike_hearn• 3 years ago

CPU designers know that some features are risky. Much like how web apps may often have "feature flags" that can be flipped on and off by operators in case a feature goes wrong, CPUs have "chicken bits" that control various performance enhancing tricks and exotic instructions. By flipping that bit you disable the optimization.

dTP90pN2• 3 years ago

AMD have released an microcode update for affected processors.

I don't think that is correct. AMD has released a microcode update[0] for family 17h models 0x31 and 0xa0, which corresponds to Rome, Castle Peak and Mendocino as per WikiChip [1].

So far, there seems to be no microcode update for Renoir, Grey Hawk, Lucienne, Matisse and Van Gogh. Fortunately, the newly released kernels can and do simply set the chicken bit for those. [2]

[0] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin...

[1] https://en.wikichip.org/wiki/amd/cpuid#Family_23_.2817h.29

[2] https://github.com/torvalds/linux/commit/522b1d69219d8f08317...

dTP90pN2• 3 years ago

More details:

`good_revs` as per the kernel: https://github.com/torvalds/linux/commit/522b1d69219d8f08317...

Currently published revs ("Patch") (git HEAD):

https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin...

As of this writing, only two of the five `good_rev`s have been published.

anarazel3• 3 years ago

What does that chicken bit do?

userbinator• 3 years ago

and Mendocino

That's the same codename Intel used for Celerons 24 years ago, the ones famous for 50% overclocks:

https://ark.intel.com/content/www/us/en/ark/products/codenam...

LtdJorge3• 3 years ago

This is both as cool as it is scary. I managed to "exfiltrate" pieces of my Bitwarden password (could easily be reconstructed), ssh login password, and bank credentials in a minute of running from a 10MB sample.

akyuu4• 3 years ago

https://www.amd.com/en/resources/product-security/bulletin/a...

According to AMD's security bulletin, firmware updates for non-EPYC CPUs won't be released until the end of the year. What should users do until then, disable the chicken bit and take the performance hit?

stefan_• 3 years ago

Are they out of their mind? This is not a "medium".

qhwudbebd• 3 years ago

Presumably classified as severity 'medium' in an attempt to look marginally less negligent when announcing that they can't be bothered to issue microcode updates for most CPU models until Nov or Dec.

themoonisachees3• 3 years ago

Under what circumstances is this not a medium? The only case this applies is if you have public runners running completely untrusted code, and if you're doing that I hope you're doing it on EPYC, which is fixed. And if you're doing that, you're probably mining crypto for randoms.

blinkingled2• 3 years ago

On my Zen2 / Renoir based system the PoC exploit continues to work albeit slowly even after updating the microcode (linked from TFA) that has the fix for this issue. The wrmsr stops it fully in its track.

Edit: just realized it must have been that the initramfs image is not updated with the manually updated firmware in /lib/firmware.

Edit2: Updated the initramfs and even if the benchmark.sh fails, ./zenbleed -v2 still picks out and prints strings which doesn't happen with the wrmsr solution.

johnp_• 3 years ago

linux-firmware does not carry any microcode update for Renoir (yet). Or what do you mean by "TFA"?

The fixed Renoir microcode should have revision >= 0x0860010b as per the kernel: https://github.com/torvalds/linux/commit/522b1d69219d8f08317...

blinkingled2• 3 years ago

TFA == The Fine Article :)

Updated microcode shows 0x08600106 revision so I guess that explains it.

farhanhubble• 3 years ago

What a beautiful article! All my memories from the computer architecture class came rushing back.

rodgerd• 3 years ago

It's worth noting that the FSF consider that supporting microcode updates makes software "un-free". This is a great example of what an inordinately stupid and harmful position that is.

boomboomsubban• 3 years ago

The FSF would have no issues with this if the microcode was free software. I don't necessarily agree with the FSF position on microcode, but they aren't against the idea of updating microcode.

SamuelAdams• 3 years ago

To me this is the buried lede:

It was challenging to get the details right, but I used this to teach my fuzzer to find interesting instruction sequences. This allowed me to discover features like merge optimization automatically, without any input from me!

What happens when AI can start fuzzing software? Seems like a golden opportunity for opsec folks.

slappy72• 3 years ago

Anybody know how to disable the chicken bit on Windows?

They're not planning to fix this until fucking December...

attentive2• 3 years ago

That's atrocious. Looks like microsoft doesn't care about AMD CPU as much as AMD doesn't care about windows.

As far as I can see there is no easy way to use MSR on windows.

attentive2• 3 years ago

I may be missing something, but since it basically exploits AVX that is easy to disable in win10.

bcdedit /set xsavedisable 1

and then reboot.

At least it fails the zenbleed POC.

JDW1023• 3 years ago

It seems like there's an wrmsr (write MSR) tool for Windows [0]. I don't know whether it will work the same way as the linux verison.

[0] https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

badrabbit• 3 years ago

Nice writeup, very easy to read. This guy is a bonafide legend, the whole time, based on his writing style I thought it was someone new to the field explaining in detail because it was hard for him to get it as well, but I see his name at the end and recalled all the epic findings of his. Very humble/nice guy, even in person I hear (rare for well accomplished people in tech).

One thing I don't get though, if amd64 is affected, shouldn't the complementary instructions in X64/intel also be affected? Does intel move around RAT referenced values instead of just setting/unsetting the z-bit?

raymond_goo• 3 years ago

Summary of the blog post "Zenbleed" by Tavis Ormandy:

The blog post discusses the discovery of a vulnerability called "Zenbleed" in certain AMD Zen 2 processors. It revolves around the use of AVX2 and the vzeroupper instruction, which zeroes upper bits in vector registers (YMM) to avoid dependencies and stalls during superscalar execution.

The vulnerability arises from a misprediction involving vzeroupper, which can be exploited with precise scheduling and triggering the XMM Register Merge Optimization, leading to a use-after-free-like situation. This allows attackers to monitor operations using vector registers, potentially leaking sensitive information like encryption keys and passwords.

The author found the bug through fuzzing and developed a new approach called Oracle Serialization to detect CPU execution errors during testing. The vulnerability (CVE-2023-20593) affects various AMD Zen 2 processors, but AMD released a microcode update to address the issue. For systems unable to apply the update, a software workaround exists by setting the DE_CFG[9] "chicken bit."

The post concludes with acknowledgments to individuals who contributed to the discovery and analysis of the Zenbleed vulnerability.

pythonist42• 3 years ago

You mean I will have to change my processor???

xinayder2• 3 years ago

I read the article and I really liked that the author tried to make it seem as simple as possible that you don't need a degree or a deep understanding of how CPUs work to understand the issue.

However, one thing that bothers me is that the author claims it's possible to retrieve private keys or root passwords by triggering a faulty revert from the instruction that resets the upper bits of a register. Where is the demo results? All I see is a small-enough gif that looks like the Matrix terminal text scrolling through. Is there any way (other than running the exploit program myself) to check the results and see that it actually leaked the root password and other information?

ZeKK142• 3 years ago

The exploit will leak more or less random data (data which was accessed recently by the CPU). You cannot target a specific part of the memory, but you can keep fetching data until you get something interesting.

xinayder2• 3 years ago

so if I wanted to test if it leaks my root password, I should run the code and open a terminal and say, upgrade packages, or upgrade packages before running the exploit code?

eric__cartman4• 3 years ago

The only way would be to let the thing run and say, pipe the output to grep looking for your password or something else you're looking for. SIMD instructions are used very often for parsing text so I wouldn't be surprised if sensitive passwords eventually get loaded into an YMM register and the exploit just so happened to dump that.

bfrog• 3 years ago

VMs were and always have been a sort of faked out security blanket. The reality is there's shared hardware, shared memory, shared network interfaces, shared storage.

That and these days you can run so much on a single machine its really hard to understand the thinking that colo isn't and wasn't the best option. AWS isn't some mythical devops dream that never fails. It's a highly convoluted (in price as well as function) affair.

carapace• 3 years ago

Every time I think it's insane to contemplate making one's own chips something like this comes up, and I realize it's insane to let so much of our lives depend on these insanely complicated devices. It feels like walking on a tightrope above an unbounded fractal chasm.

Zenbleed

Related Stories