Multiple OS Vendors Release Security Patches After Misinterpreting Intel Docs

https://www.bleepingcomputer.com/news/security/multiple-os-vendors-release-security-patches-after-misinterpreting-intel-docs/

by ingve • 8 years ago

300 91 8 years ago

bleepingcomputer.com

darkerside• 8 years ago

I'm sure the initial reaction here is going to be lamentation about the state of documentation. People will correctly point out that, if multiple entities misread the documentation, it just have been unclear. And they are right. But that doesn't make this Intel's fault alone. Clear or unclear, the documentation described behavior that was understood at the Intel organization, and the shipped product worked as described.

Where was the security testing at the OS level? Why can't there be automated test suites that catch unauthorized access issues before ship (if not before merge commit)? If your vendor delivers an insecure product and you don't discover it, how much blame do you share?

sillysaurus3• 8 years ago

Why can't there be automated test suites that catch unauthorized access issues before ship (if not before merge commit)?

Usually the search space is too large.

rocqua• 8 years ago

Isn't that what fuzzing is for?

throwaway2048• 8 years ago

Your idea is akin to searching the space of unknown unknowns, By definition someone can not even begin to quantify the space of what you don't know you don't know.

empath75• 8 years ago

It’s as simple as determining if a given program will halt.

dboreham• 8 years ago

You forgot to include the sarcasm flag. I suppose "tag" these days, right?

simion314• 8 years ago

If you can't trust the CPU documentation how can you test that addition works? even if you could test all possible combinations of terms to add and verify the results there may be a hidden flag somewhere that when flipped will change how addition works.

Tobba_3• 8 years ago

On that subject, I'm curious whether there is any CPU out there that sets the overflow flag incorrectly when computing (-1) - n when n is the most negative number (which negates to itself, so implementing subtraction by simply negating the RHS and adding will screw up the flags).

hedora2• 8 years ago

This mentality is why I like OpenBSD.

If they can’t sign off on the security/correctness of a pile of code, they delete it, even if it means losing functionality.

In this case, they simply didn’t invoke the incomprehensible instruction of doom.

erric• 8 years ago

Wow, the article shows that many vendors mis-read the docs: Apple, Microsoft, FreeBSD, Red Hat, Ubuntu, SUSE Linux, and other Linux distros...as well as VMware and Xen.

This is going to be a busy day!

tpaksoy• 8 years ago

Maybe the docs were written badly? If everyone makes the same mistake, then there might be something wrong in the source.

paol• 8 years ago

At that point can it really be attributed to "mis-reading" the docs? If every single independent implementor understood it the same way, the docs were wrong.

Tobba_3• 8 years ago

To be fair, Intel docs are so consistently gibberish that it might as well be classified a separate language (similar to english, but only a quarter the information density).

In this case it seems they just didn't properly specify a piece of insane behaviour though. Hell, I'd consider it an outright CPU bug if I'm reading this right. Seemingly there's a "feature" where loading SS causes interrupts to be delayed until after the next instruction, even if the next instruction disables interrupts - so you can cause an interrupt to fire on the first instruction of the handler (where it should be impossible).

Sgt_Apone2• 8 years ago

Speaking as a technical writer, if that many vendors misinterpreted the documentation then it was the fault of the documentation.

Someone4• 8 years ago

It also could be innate complexity of the thing being documented. Looking at https://software.intel.com/en-us/articles/intel-sdm, the combined PDF has 4.844 pages.

samueloph• 8 years ago

I was really sad to see Debian omitted from that list, they patched it too but were described as "other Linux distros" :(

erikb• 8 years ago

And in Germany it's national holiday. Really great.

jacquesm• 8 years ago

Not just Germany, a very large chunk of Europe has a holiday today. Belgium, France, Netherlands, Portugal, the Nordics and Switzerland I'm sure about, there may be others.

hannob• 8 years ago

Apple, Microsoft, FreeBSD, Red Hat, Ubuntu, SUSE Linux, and other Linux distros.

Just to clarify, this is kernel code. Listing 3 different (+ "other") Linux distros as affected is kinda bogus, it's not that they all made the same mistake, they just all use the same kernel.

sxcurry• 8 years ago

Also, Apple and Microsoft are not an “OS”. This is a poorly written article unfortunately.

confounded• 8 years ago

Many of them use different versions of the same underlying Linux kernel, sometimes put together in different ways.

SmellyGeekBoy2• 8 years ago

Not necessarily, maybe they all contributed their own (incorrect) fixes?

ddtaylor• 8 years ago

Is there a PoC exploit source?

amluto4• 8 years ago

https://lkml.kernel.org/r/67e08b69817171da8026e0eb3af0214b06...

ams6110• 8 years ago

Interesting that some of the BSDs (Dragonfly, FreeBSD) are listed as Affected and others (NetBSD and OpenBSD) are listed as Not Affected.

mmillin• 8 years ago

There isn't one BSD kernel like there is one Linux kernel, there are several different ones which might borrow from each other, but are developed independently. A comparison can be found on Wikipedia: https://en.wikipedia.org/wiki/Comparison_of_operating_system...

barkingcat• 8 years ago

Dragonfly descends from FreeBSD so it makes sense that it's affected.

The other BSD's have different kernels and vastly different development histories as well.

amluto4• 8 years ago

Some BSDs never allowed debug register writes in the first place, so they were immune.

lallysingh• 8 years ago

It's been so long I had to look it up: SS is the stack segment.

0x0• 8 years ago

And furthermore, the current stack address is determined by the combination of two registers: ss for the segment and rsp/esp/sp for the stack pointer within the segment. I guess the strange behavior around modifying ss comes from the fact that you need to also modify sp immediately afterwards, because otherwise you are running with a wild stack address pointing to random memory. You also can't modify sp before modifying ss because then you are also running with a wild stack, and an interrupt could come in at any time and push things onto random memory.

twunde2• 8 years ago

I'm super happy to see that the BSDs were contacted (with the exception of HardenedBSD).

sattoshi• 8 years ago

HardenedBSD would just receive it downstream from OpenBSD, wouldn't it? It's like contacting the Linux Mint group while contacting Ubuntu is sufficient.

nbsd4life• 8 years ago

HardenedBSD is based on FreeBSD, not OpenBSD.

twunde2• 8 years ago

HardenedBSD is a separate OS forked from FreeBSD with its own kernel development. While the BSDs may share some code, they're essentially all different OS'. There is no upstream like Linux has. As a side-note after Spectre/Meltdown, Shawn Webb complained in a NYCBUG thread about getting not being able to get access to these embargoed vulnerabilities.

mrpippy• 8 years ago

Definitely, it appears this is thanks to the work/awareness of Nate Warfield at MSRC

https://twitter.com/dk_effect/status/993903701419999232

jgtrosh• 8 years ago

Fixing the bug and having synchronized patches out by yesterday was an industry-wide effort, one that deserves praises, compared to the jumbled Meltdown and Spectre patching process.

Is this a fair comparison? I feel like the patching techniques must have been easier to develop than for Meltdown/Spectre. Furthermore, if this affected the same kind of people in this community, maybe this time around benefitted from the communication channels of the previous exercises.

Maybe this isn't a comparison to try and badmouth the previous iteration, and instead just tried to show a general improvement in the industry—I just find it a bit unfair.

gregkh• 8 years ago

This is one of the first times that I know of that the Linux kernel and Windows kernel developers discussed a security issue together directly. So while the fix was much simpler than Meltdown/Spectre was (Linux was fixed with a patch that was written in 2015) overall, the communication between different OS kernel developers right now is very good.

And yes, it is all due to the horrible Meltdown/Spectre problem and how that was handled. We were not allowed to work together for that problem, and we do not want to that to happen again.

pritambaral• 8 years ago

We were not allowed to work together for that problem

How do you mean?

LeifCarrotson• 8 years ago

At first I thought this was an error in implementing new docs related to Meltdown/Spectre. But the original research paper:

https://everdox.net/popss.pdf

the researcher wrote:

Somewhere around the release of the 8086, Intel decided to add a special caveat to instructions loading the SS register...where loading SS with [`pop ss` or `move ss`] would force the processor to disable external interrupts, NMIs, and pending debug exceptions.

So it's a really, really old piece of documentation, dating from around 1980.

To call it a 'misinterpretation' rather than a vulnerability is extremely generous, given that most Intel engineers spent entire careers in the presence of code vulnerable to this 'misinterpretation' without calling the OS vendors out on their error.

ianopolous3• 8 years ago

On this particular thing I think the Intel docs are clear. This is where I implemented the same in JPC:

https://github.com/ianopolous/JPC/blob/master/src/org/jpc/em...

an_account_name• 8 years ago

They might be clear today. Were they always? There are literally decades of timeline to examine here.

caf• 8 years ago

The interrupt shadow itself is clear.

The specific implication of that for a mov ss ; syscall pair with a hardware breakpoint set on the first instruction is a lot more subtle.

feikname• 8 years ago

"Both Peterson and the CERT/CC team blamed the "unclear and perhaps even incomplete documentation"

Yet the article's title makes it seems like it was the OS implementors faults instead of Intel's.

I wonder for what reasons the website tries to shift/soften Intel's fault on this?

Seems like there's a trend in news articles to have incoherent titles in relation to content lately, it's really annoying...

hyperman15• 8 years ago

I'm trying to understand this one, even if most of my ASM knowledge is from the 8086 era. My guess:

* When an interrupt, debug exception, ... occurs, the CPU pushes stuff on the stack as part of the task switch.

* The stack is managed by 2 registers: SS and (e/r)SP. To change your stack, you have to change both registers. If an interrupt happens and you've changed only 1, stuff gets pushed on an invalid stack and you're toast.

* To fix this, the CPU has a wild card: When you change SS, you get exactly 1 instruction that will not be interrupted. The idea is you use that instruction to change (e/r)SP and make the stack valid again. If there is a need for an interrupt, it will be delayed for 1 instruction.

* Now this being a security problem, what would happen if you use this second instruction to switch to kernel mode ? It turns out the delayed interrupt happens before the first kernel mode instruction, but in the kernel.

* And you can trigger the right kind of interrupt with debug exceptions and single stepping.

* And if you do this, the kernel tells the debugger not about the debugged program but about the kernel. Oops.

So to fix this, I suppose the kernel checks the debug exception info from the CPU, and if it is debugging the kernel it fixes things up so you go back 1 instruction.

codedokode2• 8 years ago

To fix this, the CPU has a wild card: When you change SS, you get exactly 1 instruction that will not be interrupted. The idea is you use that instruction to change (e/r)SP and make the stack valid again. If there is a need for an interrupt, it will be delayed for 1 instruction.

I wonder, why could not they make a single instruction to change both SS and SP?

vardump• 8 years ago

They probably could, but it would have had a large silicon cost for 1980 without performance benefit. That and ISA incompatibility.

tych0• 8 years ago

You could, but it wouldn't allow them to relax the constraints here, since you'd have to recompile all the code in the world to use this new instruction before relaxing.

Someone4• 8 years ago

They could, but attackers would still use the approach that works for them.

And they can’t really remove the old instructions because of backwards compatibility.

Whitelisting a limited set of instructions that can follow setting SS and making all others trap might be an option, though. It still would break backwards compatibility, but if the effective impact would be negligible, they could deem it acceptable.

hyperman15• 8 years ago

There are. I suppose LSS SP,value is possible on i386+ and there is also the task state segment which might help. But this was a hack in the original 8086, which wasnt the best processor design ever to start with. It stays there because of backward compatibility.

himom• 8 years ago

In protected mode, SS is a descriptor offset into the GDT. It doesn’t make sense to change SS very often unless there are multiple stacks. Furthermore, it doesn’t make sense to overcomplicate an already over-complicated instruction set burdened with vestigal, legacy features. And, loading SS and SP atomtically would probably create a byte pattern that’s too long to encode.

monocasa2• 8 years ago

They use the same scheme other places as well.

   sti
   hlt

will hold off on enabling the interrupts until right after the hlt has started so that you don't end up in a race condition when you get an interrupt between sti and hlt.

Albeit, they could have made an stiandhlt instruction here as well.

ysleepy2• 8 years ago

The x86 ISA and its implementations are now in the spotlight of the whole security research community. There is probably a lot more to come since it accumulated a lot of cruft in the name of backwards compatibility.

I hope we learn a lot, and take the time to record the experience, for coming platforms like RISC-V and others.

Why is there no big CAVEATs document from intel detailing weird quirks. I strongly assume the intel arch engineers are well aware of many of those counter-intuitive behaviours in their products.

adrianratnapala• 8 years ago

I strongly assume the intel arch engineers are well aware of many of those counter-intuitive behaviours in their products.

But it is likely just kind of distributed, organic knowledge that is hard to condense into a single document. Writing and maintaining such a thing would be a significant project, and (I am speculating here) not the kind of thing that significantly burnishes anyone's performance review.

That said, the whole community of assembly-hackers has even broader knowledge of the topic, and could start such a document out in the open. And Intel engineers might likely contribute their own two cents. (Unless lawyers forbid it).

ysleepy2• 8 years ago

I stumbled on a blogpost by bunnie huang which describes a liability angle, which I found to be plausible: https://www.bunniestudios.com/blog/?p=5127 - worth a read.

Tobba_3• 8 years ago

As far as I understand, what's happening is:

* There's an old feature which causes POP SS/MOV SS instructions to delay all interrupts until the next instruction has executed, to safely allow changing both SS and SP without an interrupt firing inbetween on a bad stack.

* If such an instruction itself causes an interrupt (by triggering a memory breakpoint through the debug registers), it is delayed (as intended).

* The delayed interrupt will fire after the second instruction even if the second instruction disabled interrupts.

* By means of the above, a MOV SS instruction triggering a #DB followed by an INT n instruction will cause the #DB exception to fire before the first instruction of the interrupt handler, even though this should be impossible (as entering the handlers sets IF=0, disabling interrupts).

* The OS #DB handler assumes GS has been fixed up by the previous interrupt handler, which in now under user control.

dwighttk• 8 years ago

The vulnerability notes[1] say Apple patched this on May 8, but my last security update was May 3 and I don't currently show any available updates... I wonder if the May 3 patch fixed this, or if my computer might not be affected.

[1]https://www.kb.cert.org/vuls/byvendor?searchview&Query=FIELD...

amluto4• 8 years ago

The May 3 patch fixed it. The nature of the fix was such that Linux and Mac OS were able to patch it early without revealing much.

In fact, the Linux fix was a patch I wrote in 2015 (for unrelated reasons) and just never got around to upstreaming.

jey• 8 years ago

Link to referenced patch: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

tmd83• 8 years ago

Can someone explain the risk factor? It cannot be remotely exploited or though browser if I'm reading it right. But a malicious program with user level access can get kernel access. So exposure to malware running on a limited account can get higher access?

ComodoHacker• 8 years ago

Archived version: http://archive.is/DxUwA

std_throwaway2• 8 years ago

Is this also affecting code in 64 bit mode?

dfox• 8 years ago

As the code in the paper is written for amd64 I would assume that it is affecting it.

In the other hand it is somewhat surprising, because loading anything into SS is mostly pointless operation.

std_throwaway2• 8 years ago

This doesn't even make sense. Why would they keep a behavior that could be considered a bug when creating a new instruction set?

hyperman15• 8 years ago

I wonder what happens if you execute multiple POP SS instructions. In fact, you could set up a 64K v86 mode segment containing only copies of the POP SS instruction. jmp far into it. When IP reaches the last instruction it wraps around and starts again. Will it ever be interrupted by anything? If the stack usage bothers it, just do MOV SS,AX

Someone4• 8 years ago

Intel thought of that, but doesn’t seem to give us an answer. https://software.intel.com/sites/default/files/managed/7c/f1..., page 6-8:

”If a sequence of consecutive instructions each loads the SS register (using MOV or POP), only the first is guaranteed to inhibit or suppress events in this way.”

So, we still don’t know. For all I know, it may even depend on the exact cpu used or cpu state.

dooglius• 8 years ago

Tested it: you get an interrupt after only skipping one instruction

hyperman15• 8 years ago

Thanks, both dooglius and Someone. One learns something everyday.

_bxg1• 8 years ago

I felt a great disturbance in the Force, as if thousands of voices suddenly cried out, "Oops..."

emily-c• 8 years ago

A bit tangentially related, but I've always wondered why the syscall instruction doesn't use the TSS for stack switching like int does. I guess it does give you more flexibility to load rsp from gs during a cpl 3 -> cpl 0 transition rather than consulting the TSS to switch it automatically. Can anyone weigh in on this?

floatboth• 8 years ago

Was illumos not affected?

Multiple OS Vendors Release Security Patches After Misinterpreting Intel Docs

Related Stories