Linux Kernel Developer Criticizes Intel for Meltdown, Spectre Response

http://www.eweek.com/security/linux-kernel-developer-criticizes-intel-for-meltdown-spectre-response

by CrankyBear • 8 years ago

454 145 8 years ago

cthalupa5• 8 years ago

"The majority of the world runs Debian or they run their own kernel," Kroah-Hartman said. "Debian was not allowed to be part of the disclosure, so the majority of the world was caught with their pants down, and that's not good."

Is there any actual statistics to back this up? I feel like RHEL and to a lesser extent CentOS have a stranglehold on the big enterprise-y environments, and I see Ubuntu basically everywhere else, and Canonical does their own kernels.

Edit: To be clear, I am aware Ubunbtu is a Debian derivative, but since we're talking specifically about who was informed for kernel level mitigations, and Canonical does their own kernels, it seems weird to talk about how Debian wasn't informed and thus people were affected, when Ubuntu being updated wasn't reliant on Debian being updated.

sounds• 8 years ago

Perhaps Greg-KH was implicitly including Ubuntu under Debian?

2trill2spill6• 8 years ago

Yea I don't use Linux too often but how different is an Ubuntu kernel compared to a Debian kernel? I know Ubuntu is downstream from Debian or at least they were. So how much extra work/code does Canonical add to its kernels.

Twirrim• 8 years ago

As mentioned in the article, Canonical were brought in much earlier by Intel. They appear to have been among the "chosen few": RedHat, SuSE and Canonical.

stefan_• 8 years ago

The majority of Linux installations is probably Android smartphones, and well Google was informed alright, but still 95% of those phones were caught with their pants down because well their pants had fallen down a long long time ago given they last received security updates years ago.

joe_the_user• 8 years ago

How many android installations use Intel processors?

Edit: never mind, Android was apparently indeed effected.

https://www.androidcentral.com/meltdown-spectre

DCKing• 8 years ago

Spectre and Meltdown are a problem for Android but it's a smaller problem than on PCs. The vast majority of Android phones use processors that are in-order, and lack the out-of-order execution necessary to exploit.

High-end Android phones with out-of-order processors (the minority) are only vulnerable to Spectre, but it's a little complicated [1]. Then there is only one ARM core (Cortex A75) that is vulnerable to Meltdown, but that was released after the vulnerability was published, so software could ship with mitigations in place.

[1] https://github.com/lgeek/spec_poc_arm

dijit• 8 years ago

I can speak to the companies I've worked for: US companies (and, this includes the UK because apparently we love to be like america) use CentOS predominantly, but mainland europe (Sweden, Germany, Finland) seem to prefer Debian.

I can say that I've personally administered roughly a 1:1 ratio of CentOS:Debian despite coming from a country that's servers tend to be CentOS.

Of course, this is anecdotal, but don't undersell debian.

bluedino• 8 years ago

Is SuSE still popular in Europe? (Apologies if I got the capitalization wrong)

gaius• 8 years ago

Since 2008 working for US/UK companies I have only seen RHEL and CentOS (prior to that, mostly SunOS/Solaris). I prefer Debian personally but for the last few years I have used CentOS for my own projects too - too much effort to maintain two parallel sets of skills now I am not really into OS level stuff anymore.

xchaotic• 8 years ago

I'm sure he cares a lot about Debian and a lot of my early learning was on Debian derivatives but I feel that the production server side world has largely standardised on RedHat. Non RedHat/CentOS, non-Ubuntu debian distros are probably a distant fifth in terms of business critical production server so no wonder they were not included in the disclosure early on...

jessaustin• 8 years ago

Does CentOS get immediate updates like this? My overwhelming impression of that platform is that everything is years out of date... of course maybe that's just the fault of the organizations I've encountered that have used CentOS.

l0b0• 8 years ago

Translation: s/The majority of the world runs/My fellow Debian kernel developers run/

bogomipz2• 8 years ago

Ubuntu is a Debian derivative[1], I imagine that is what he means:

[1] https://www.debian.org/derivatives/

cthalupa5• 8 years ago

Yes, but the problem is we're talking specifically about kernel level mitigations, and Canonical rolls their own kernels. They do NOT use debian kernels.

pera2• 8 years ago

The Linux distribution can be positively identified in around 30% of cases, and of these 1.39 million Linux computers, just over half are running Ubuntu Linux, nearly a quarter are running CentOS, and around fifth are running Debian Linux.

https://news.netcraft.com/archives/2017/09/11/september-2017...

craftyguy3• 8 years ago

There are a massive number of systems running Linux that are not publicly facing (e.g. compute/service nodes in supercomputers, systems for providing internal infrastructure for R&D, manufacturing, etc).

krn5• 8 years ago

I think, what he meant by saying, that "the majority of the world runs Debian or they run their own kernel", – is that almost everyone else, who is not running a product by Red Hat, SUSE, or Canonical, is running a system based on a Debian kernel. And, therefore, Debian should have been informed.

rurban• 8 years ago

Desktop and personal PC has Ubuntu dominant, but servers are 70% CentOS. Nobody trusts Debian's security record and gcc/kernel abilities.

He is right with the affected number in this case because this time bugs are affected by simple webpage drive-by Javascript attacks, like a common virus. So it affects Debian and Android more than servers and cloud services. And those are the ones which are not regularly updated so the security impact is bigger than on a server.

curveship• 8 years ago

By "majority" he may have meant majority of Linux distributions, not majority of Linux users. There are a ton of distributions derived from Debian: https://wiki.debian.org/Derivatives/Census . All of them would have to update their distros to respond to Meltdown.

shakna• 8 years ago

Why would an off-the-record statement, which is what he said this was, need to be 100% without exaggeration or hyperbole?

cthalupa5• 8 years ago

The article doesn't mention it was off the record, and I wasn't there, so how would I know it's off the record?

But it seems likely that this isn't just exaggerated or hyperbolic, but just not close to accurate, in which case even if it's off the record it's silly.

jack12• 8 years ago

https://youtu.be/lQZzm9z8g_U?t=1565 seems to be the same talk delivered at another instance of this conference. He does claim one statistic that an unnamed "top 3" cloud provider told him that fewer than 10% of their customers install company-based kernels and over 90% install community-based kernels like debian or kernel.org code.

He is not limiting "the majority" to just Debian, though. He is comparing installs of distributions backed by companies (like Red Hat+Canonical+SuSE) as "the minority" vs. every possible non-corporate-backed distribution or source.

crankylinuxuser• 8 years ago

I run centos in production. That's because of the following:

     1. SELinux 
     2. OpenSCAP
     3. 99% fit with RedHat

None of the other Linux variants have those tools, or the time put in for compliance goodies. My time == $$$

2bluesc• 8 years ago

Any videos of the presentation?

mikeyouse• 8 years ago

No video as far as I know, but he did post the slides with some backup materials on his GitHub:

https://github.com/gregkh/presentation-spectre

One note from his talk though, his slides say that Foreshadow was fixed in April -- but it's clearly supposed to be August. He mentioned he was fixing the updates even on the flight out to OSSNA.

2trill2spill6• 8 years ago

"That's a long time, and we only heard rumors because another very large operating system vendor told Intel to get off their tails and tell us about it."

I wonder which operating system vendor pressured Intel to tell the Linux dev community, especially because it sounds like it was a non Linux OS vendor. Whomever it was, good job!

But it seems like Intel has angered the Linux community as well as the various BSD operating systems. You would think Intel would be doing whatever it can to please all operating system vendors especially now that AMD is getting competitive again.

judge2020• 8 years ago

My guess is Microsoft seeing how recently they're pouring a lot of resources into the Linux subsystem and how they're trying to seem more "developer-oriented" overall in recent months.

2trill2spill6• 8 years ago

That was my guess as well, also doesn't Microsoft run some Linux in their Azure cloud? So that adds yet another incentive for Microsoft to help Linux folks out.

Edit: I found a link confirming that Microsoft uses Linux for networking in their Azure data centers [1].

[1] https://www.datacenterdynamics.com/news/microsoft-runs-azure...

simcop23872• 8 years ago

Along with that I'd imagine that they're really invested in it with Azure and such too. If all their customers were vulnerable to this they could end up in a bad position. They could obviously work on their own patches but they'd have to get them upstream into distros without telling them what's going on because of the nda/silence desired by Intel. That would leave them in a terrible position.

cthalupa5• 8 years ago

Azure has a sizable customer base running Linux, so I imagine it was Microsoft.

snaky4• 8 years ago

BTW

Experts called for a new generation of secure-by-design computers at the Hot Chips conference here. In small steps in that direction, Microsoft and Google described their separate but similar hardware security architectures.

https://www.eetimes.com/document.asp?doc_id=1333616

y-c-o-m-b• 8 years ago

I wonder how many NSA back-doors will go into those.

snaky4• 8 years ago

The vast majority of regular users would voluntarely install a dozen of NSA backdoors on their computers if it guarantees them full protection from script kiddies, phishing, malware and ransomware.

jokoon• 8 years ago

As opposed to what? He said secure by design.

rrix2• 8 years ago

"Intel has gotten better at this," he said.

Someone should let the BSD folks know and see what they think...

2trill2spill6• 8 years ago

They are getting better, even if they didn't set a very high bar to begin with. I believe they let the FreeBSD folks know about the last round of Intel bugs as opposed to trying to keep them in the dark like the first few rounds of Intel bugs.

dshep• 8 years ago

I get the impression OpenBSD isn't so happy with Intel:

Disable SMT/Hyperthreading in all Intel BIOSes https://news.ycombinator.com/item?id=17829790

taborj• 8 years ago

And also:

"I'm going to spend my money at a more trustworthy vendor in the future." - Theo de Raadt

cperciva4• 8 years ago

My understanding is that they've gotten better. FreeBSD has had advance notice of some issues. Last I heard they offered to let OpenBSD in too, but hadn't found anyone willing to sign an NDA.

sverige• 8 years ago

I'm reasonably certain that OpenBSD has never agreed to any NDA from anyone as a matter of principle. It's one of the things that makes me love the project so much.

driverdan• 8 years ago

Why would anyone sign an NDA for vulnerabilities? The problem is on Intel's end, not the OS devs.

bigDeal• 8 years ago

Does anyone honestly buy into the idea that this is something other than a back door, carefully plotted by certain nation-state actors?

For real.

froogie• 8 years ago

Why would Intel be complicit in it?

Suppose they did this. Now, that the vulnerabilities are public, Intel takes the fall. Oopsie, huh?

cmurf• 8 years ago

Yes. But then, you're asking for an opinion. If you were asking the reader for evidence, you wouldn't have started out with a premise depending on paranoia and conspiracy.

bArray2• 8 years ago

The creators aren't always aware of the monsters they create. I think Intel was caught with their trousers down - AMD and ARM in many respects too. Hopefully this will lead to automated testing procedures for Intel, other processor manufacturers and security researchers.

The reason these flaws were found in the first place were because of security researchers, hopefully they can now make their cases to get large research grants and bring about a more secure future.

On the other hand, if you want to talk about tin-foil hat "CPUs working against users", only look so far as Intel ME and the equivalents on other processors. Closed-source and highly privileged code running all the time inside the CPU - it's not unthinkable that Intel would bend the knee to some state actors, especially if that leads to higher profits (i.e. a tax break or entry into a new market (cough China)). It's enough of a concern that a lot of reverse engineering effort has gone into preventing it from running.

acdha2• 8 years ago

Hopefully this will lead to automated testing procedures for Intel, other processor manufacturers and security researchers.

They have tons of testing already. The problem is that this is the kind of problem which is easy to miss with tests, especially automated ones, because everything worked correctly and no real code would ever have been affected by the side effects.

adiusmus3• 8 years ago

Plenty more security vulnerabilities in Intel ME to be found. It’s now a well known attack surface full of goodies to plunder. I’m sure AMD and ARM have similar excitement in store. This is in addition to what has already been found.

apetresc• 8 years ago

Yes, literally every rational person.

adiusmus3• 8 years ago

Applying Occam’s razor to Intel ME and similar security issues: conspiracy with some evil agency is possible. Maybe. But not the only explanation.

Following the money suggests Intel was chasing an enterprise market and didn’t bother to think through the consequences (unlikely) or made a car-manufacturing style decision where the costs of failure were deemed to be overshadowed by the profits on their balance sheets.

For seven years the Ford Motor Company sold Pinto cars in which it knew hundreds of people would needlessly burn to death. It had a better design but it would have cost production delays to implement it. Less money to have people die than fix the issue when it was noticed in testing.

While I’m not equating Intel’s actions to Ford’s in direct comparison, I would say the bean counter thought process is far more likely than conspiracy.

dschuetz• 8 years ago

Because Intel is that good in security and hardening, why not use Intel's newly minted own special secure Linux distribution?

https://01.org/blogs/imad/2018/letter-industry

This is almost like satire.

smhost• 8 years ago

Not sure what you mean. This just looks to me like they're admitting they're awful at security and they're pleading/threatening the global community to help secure their ecosystem or else you're all going to be in a ton of trouble because the robots/cars are going to kill you and your workers and what'll you do then?

benchaney3• 8 years ago

This just looks to me like they're admitting they're awful at security and they're pleading/threatening the global community to help secure their ecosystem

That post seems to make it pretty clear that they are trying to position themselves as the experts rather than the people asking for help. That is the part that is absurd.

hornetblack• 8 years ago

I think even a Intel contractor was indirectly responsible for GRSecurity going behind a paywall.

(tl;dr intel contractor were shipping an os that was using grsec tradmark in marketing for a non-grsec approved kernel)

docker_up• 8 years ago

What is the current status of this? The last time I heard, the OS fixes would impact CPU performance by 30%. Is this still the case? Will new iterations of Intel CPUs be immune to this, or is this an ongoing issue going forward because it's inherent with the architecture?

craftyguy3• 8 years ago

Meanwhile, on HN frontpage: https://news.ycombinator.com/item?id=17876476

simcop23872• 8 years ago

Based on my own experience with things and seeing other benchmarks, it looks like there's some synthetic benchmarks and very specific loads that will see up to 30% losses, but most things are really around 10-15% if they've got any significant losses at all. That's still a big impact but not nearly as "sky is falling" as it seemed like it might be. No idea how this'll be going forward with hardware changes since those haven't happened yet, and even when they do, how will we benchmark them properly?

hansendc5• 8 years ago

Disclaimer: I work on Linux at Intel.

Future hardware will be mitigated against side-channel issues. There's a nice table showing how things are mitigated on future processors here:

https://www.tomshardware.com/news/intel-cascade-lake-details...

But, not everything is mitigated in the silicon or microcode. The mitigation for Spectre / Variant 1 / Bounds Check Bypass, for instance, will continue to be in software.

tracker1• 8 years ago

Everything I've read has said real workload differences can be as little as 3% or as much as 30%, but mostly somewhere in between (closer to the 3% side). Depends on what you're doing I guess.

cperciva4• 8 years ago

While I agree that Intel's response was far from ideal, I find it a bit rich for Linux kernel developers to be criticizing them. Remember, the completely uncoordinated disclosure happened because Linux kernel developers started discussing the vulnerability -- while under NDA -- on a public mailing list.

NullPrefix• 8 years ago

You sure those devs were under NDA?

0xFFC2• 8 years ago

I was in the room. Greg specifically said “off the record”.

TillE• 8 years ago

That's not a magic phrase, it's something you negotiate with reporters in advance. You can't really make "off the record" comments in front of an audience at a conference.

0xFFC2• 8 years ago

Did I say it is a magic phrase? It used to say when you want something not reported. You should definitely get familiar with journalism terminology.

hansendc5• 8 years ago

I worked on the Meltdown mitigations, at Intel, during the embargo. I still work on Linux at Intel.

Intel told quite a few members of the Linux community about this well before late December. Some got told because they work for traditional distributions and others were contacted directly because they are community maintainers or subject matter experts.

Unfortunately, Greg was not one of those folks that was told early. It's pretty clear at this point that it would have been a lot better had he and folks like him been involved earlier. This is especially true for Greg since he plays such a crucial role in stable kernel maintenance, which is how a lot of the world consumes their kernels (including distros like Debian).

I'm glad Greg thinks "Intel has gotten better at this." I'd like to think so too.

2trill2spill6• 8 years ago

Shouldn't Greq and Linus be some of the very first people to be informed considering the amount of impact they both have on the entire Linux ecosystem? I find it alarming that they were not contacted immediately considering how much of the world/internet relies on Linux.

ibotty• 8 years ago

That does not address the stupid embargo on involved linux developers to speak about it to each other.

mjevans• 8 years ago

It shows that legal and marketing (PR spin) were prominent on the security decision tree rather than security minded individuals.

hansendc5• 8 years ago

Disclaimer: I work on Linux at Intel.

I'm biased, of course. But, I think things are much better now. We have ways of having relatively "normal" conversations about these things where a broad set of community folks are involved. It's not perfect, but it's way better than it was a year ago.

The embargo was really painful to work with, even for folks like myself who were inside Intel during the whole thing. I'm used to being able to ask folks for help and to lean on the incredible knowledge of the kernel community. For Meltdown, I was much more on my own than I was comfortable with. It wasn't fun.

geezerjay2• 8 years ago

Unfortunately, Greg was not one of those folks that was told early.

Why wasn't he one of the first persons informed by Intel?

hansendc5• 8 years ago

Disclaimer: I work on Linux at Intel.

I'll give you an engineer's perspective, and specifically about Meltdown since that's what I worked on.

I posted the first Meltdown mitigations in late October, publicly claiming to mitigate a less dangerous class of side-channel (https://lkml.org/lkml/2017/10/31/884). I figured that Linus would merge it (Linus knew the real reason by that point) and would "wink" at Greg and the other stable maintainers enough to get them to merge it too. At this point, we had months before the embargo ended.

In retrospect, that was a kinda silly assumption, but I was heads down trying to get the patch to work and didn't dwell on it much. I wish I would have made more noise about it, especially as we got closer to the embargo ending.

AdmiralAsshat• 8 years ago

Greg's response is understandably frustrated, though seemingly less so than the OpenBSD devs. Why are they being repeatedly left out of the loop?

amaranth• 8 years ago

OpenBSD has been accused of breaking embargoes in the past. They are pretty open about their policy of pushing their fix as soon as it's ready and not doing anything to obfuscate what they're fixing and why.

adamrt• 8 years ago

Can you provide some links for that?

I feel like this is a thing people repeat, specifically about KRACK and its been shown to be false multiple times (like here: https://lobste.rs/s/dwzplh/krack_attacks_breaking_wpa2#c_pbh...).

benchaney3• 8 years ago

They are pretty open about their policy of pushing their fix as soon as it's ready and not doing anything to obfuscate what they're fixing and why.

This isn't true at all. Please stop spreading misinformation.

mackal• 8 years ago

Its not that OpenBSD breaks embargoes, it's that they won't just be strung along delaying embargoes for ever. So they can't be trusted to extend an embargoes. They probably would be happy to if there was a good reason, but not for bad reasons.

protomyth• 8 years ago

Theo de Raadt answers that at https://youtu.be/UaQpvXSa4X8?t=4m7s

pmontra• 8 years ago

While there have been many patches made in Linux, he strongly advised users to update with Intel's microcode fixes as well, as they provide an additional layer of protection beyond what an operating system can provide.

I found this page explaining how to do it https://www.cyberciti.biz/faq/install-update-intel-microcode...

I checked and my microcode is from 2018-01-21. Being on a 2014 laptop it means it got updated by the package manager.

Linux Kernel Developer Criticizes Intel for Meltdown, Spectre Response

Related Stories