Intel vulnerabilities costing 25% CPU performance loss to a cloud provider

https://twitter.com/waxzce/status/1128711501206913028

by samber • 7 years ago

332 178 7 years ago

samber• 7 years ago

@waxzce: "FYI, as cloud provider we rawly loss around 25% of CPU performances the lasts 18 months due to different CVE and issues on CPU and mitigation limiting capacity using microcode, so we stuff more CPUs, but prices didn't go down at all... That's a kind of upselling. #IntelFail"

aiCeivi9• 7 years ago

Is it #IntelFail if Intel had supply issues in last year? It doesn't look like demand was affected by much.

Traster• 7 years ago

Any way you slice it it's an Intel Fail. They've basically had to lower the performance of their existing chips, they've failed on the road map for new chips, and they don't have capacity for more of the existing chips because they thought 14nm would be winding down by now rather than peak production.

walrus012• 7 years ago

Don't get married to Intel, if you need big beefy xen or kvm hypervisor machines, there's lots of good EPYC based motherboards.

gcells2• 7 years ago

I am planning on making a home-server. Any suggestions for EPYC boards?

cyphar3• 7 years ago

I'm setting up a home server and decided to just go with Ryzen. You get a bunch of server features (ECC RAM, lots of PCIE lanes, and lots of cores) for a lot less cash than the equivalent EPYC build.

Obviously EPYC has a place but for the home usecase you could use a Ryzen as a substitution for Intel Xeons because of the baseline features of Ryzen.

_JamesA_• 7 years ago

I just built my first AMD machine since the Athlon days with an ASRock Rack EPYCD8-2T [0] motherboard and an AMD EPYC 7281 CPU [1].

I upgraded a VMWare ESXi server hosting FreeNAS from a consumer motherboard with an Intel Core i7 3930K and it was completely painless.

[0] https://www.asrockrack.com/general/productdetail.asp?Model=E...

[1] https://www.amd.com/en/products/cpu/amd-epyc-7281

mizzack4• 7 years ago

Depending on what you're using it for, I would take a look at Threadripper too. There are some crazy deals right now. If you don't IPMI and registered ECC, it's a great option for a ton of cores and PCIe lanes.

kop3162• 7 years ago

If you are running only open source software, you may want to consider this:

https://www.raptorcs.com/

oldjokes• 7 years ago

Intel has been proven unreliable and untrustorthy time and time again, people have been complaining about how awful and unscalable x86 is since the 90s.

The future is more likely ARM- I know a few companies are experimenting with moving their internal servers to ARM, and AWS started offering that platform a few months ago.

But I'm not sure how long this takes, people have also been wrongly predicting the death of x86 since the 90s.

jeltz2• 7 years ago

If AMD was in a better shape, there is a real market momentup here.

What does this comment refer to? As far as I know AMD is in pretty good shape.

jasonvorhe• 7 years ago

They probably can't meet the demand for server-grade CPUs. But I'm just guessing.

ksec2• 7 years ago

Most of the current AMD offering, APU, GPU, and CPU is still being fabbed with GF 14nm, which I believe to be very limited in capacity. Once the next generation GPU ( Navi ) and CPU ( Zen 2 ) along with EPYC 2 moves to TSMC 7nm they should be able to flex their muscles a lot more.

I wished they have made the move earlier, like start of this year, but it is now all set for Q3.

Someone12342• 7 years ago

They're a cloud provider, so cloud offerings. AMD has had some consumer success but remain rare in the date center.

jeltz2• 7 years ago

Hetzner offers AMD processors in some of their dedicated servers, but I have no idea if they also run some of their VMs on AMD.

kllrnohj12• 7 years ago

AMD's Epyc itself is highly competitive, but there's the very slow turn around to get that into actual server offerings from the major companies. And of course just being competitive isn't going to get everyone to start adopting a different platform.

botto• 7 years ago

Is this a chance for ARM providers to move in and outcompete Intel, especially if they can provide similar tech but with security? Basically an ARMs race?

cyphar3• 7 years ago

I think it's more likely that AMD will pick up the slack.

AnIdiotOnTheNet• 7 years ago

You can't stop this class of vulnerabilities without sacrificing performance because the reason for the performance is also the reason for the vulnerabilities. That's the choice you have, and no brand affiliation will save you.

mfatica5• 7 years ago

It's not about brand affiliation, AMD and Intel have vastly different implementations

Symmetry3• 7 years ago

You can't stop all speculative leaking without sacrificing a large amount of performance. But you can stop all leaks across security barriers with a very modest performance cost. This still leaves web browsers with a hard task in terms of isolating Javascript but it solves the issue for, e.g., web hosting services like the OP.

dfrage7• 7 years ago

ARM is more less examined than less vulnerable. Most of their cores aren't speculative execution out-of-order, that's expensive in power and silicon real estate/cost, but they have 13 vulnerable to Spectre, including 1 known for the 2018 Rogue System Register Read which like this latest set for Intel is design specific. It would be wise to assume there are more vulnerabilities hiding, the name Spectre was chosen because we'll be haunted by it for a long time. Also one of their newest designs is vulnerable to Meltdown, see the whole list here: https://developer.arm.com/support/arm-security-updates/specu...

davrosthedalek• 7 years ago

If they want to compete with intel and AMD on performance, they will have to include the same speculative out-of-order architecture. These bugs are actually not bugs in the sense that the hardware does something it shouldn't do. It's 100% to spec. The problem is the spec. AMD has less problems, but I'm not so sure this is only because they are more secure. I think there are more eyes looking for exploitts for intel CPUs than for AMD.

adrianN2• 7 years ago

Why is nobody suing Intel? They sold defective chips.

orliesaurus2• 7 years ago

Not voluntarily?

adrianN2• 7 years ago

So? If I buy a car that is promised to have 200HP but it actually just has 150HP after a software update wouldn't I be able to sue the manufacturer at least for the lost resale value?

deweller• 7 years ago

Is there an End User License Agreement that users agree to before using these chips? If it is anything like software EULA's, then Intel is likely protected from lawsuits for defects.

ahartmetz• 7 years ago

Except unreasonable EULA terms are void in most (all?) of Europe.

gruez• 7 years ago

Is there an End User License Agreement that users agree to before using these chips?

I certainly didn't see one when booting up a new Intel computer.

SomeOldThrow• 7 years ago

I was under the impression EULAs are basically useless.

gnode• 7 years ago

This isn't a defect in the traditional sense. The device is behaving to specifications, but a new method of attacking it has been found.

Rowhammer is arguably more aptly described as a defect; in that case the RAM is not behaving to spec.

sureaboutthis2• 7 years ago

Perhaps this is the reason for Intel's recent push on their self created version of Clear Linux?

yjftsjthsd-h4• 7 years ago

How does that help? You still take the perf hit?

sureaboutthis2• 7 years ago

Tests on Phoronix show Clear having significant wins in a number of tests. They have done so by rewriting libraries to perform well with Intel hardware. Those who complain about a 25% performance hit elsewhere can now be shown pretty good performance with Clear that isn't attainable with other distros.

exabrial• 7 years ago

What are the possible pitfalls of _not_ runnning spectre et all mitigations, if you're _not_ hosting other people's code, but on your own hardware?

Filligree• 7 years ago

Not much. We've yet to see any live Spectre exploits (that I know of), and I'd be surprised if the first one is a remote exploit.

londons_explore• 7 years ago

Good point... I wonder why? The proof of concept code works so well for some variants.

Symmetry3• 7 years ago

I can see how a local Meltdown exploit would be useful but what would be the point of a local Spectre exploit?

idrae• 7 years ago

Not an expert, but if you are running absolutely no untrusted code (which is hard depending on your definition of untrusted), then the risks are low.

However, when a vulnerability is found, then spectre etc. make it easier to abuse that vulnerability to do something useful.

chmod775• 7 years ago

Just guessing here, but suppose an attacker is able to get unprivileged code to run on your machine (by taking over some process). He is now able to extract secrets from other processes on the machine that he ordinarily wouldn't have access to. Think SSL keys etc.

But I believe you're right in that those exploits are not as dangerous in a non-shared hosting scenario.

zorked• 7 years ago

Quick reminder that a browser with JavaScript is a way to run unprivileged code on your machine.

phire• 7 years ago

I'm not even sure code execution is strictly necessary for this style of attacks.

An attacker could carefully craft network packets to force the control flow of existing software to manipulate the CPU state. They could and then use the timing differences in network packets to read data out.

Would be painfully slow, but theoretically possible.

acdha4• 7 years ago

suppose an attacker is able to get unprivileged code to run on your machine (by taking over some process). He is now able to extract secrets from other processes on the machine that he ordinarily wouldn't have access to. Think SSL keys etc.

I think that's the catch: it's the set of cases where someone can get code execution without being able to directly retrieve the targeted data, which is already pretty low, and where the best mitigation technique is to harden against these attacks rather than use a dedicated service (e.g. if you're using an HSM for crypto or have a dedicated SSL-offload box, you don't have to worry about keys being compromised when someone finds a bug in the much greater exposed surface area of your application).

userbinator3• 7 years ago

None, really.

If you're not sharing your hardware with other code you don't trust, it's not a problem.

mtgx3• 7 years ago

So then we agree it's a problem.

What code do you actually trust? (to be incorruptible/unhackable, etc)

zamadatix• 7 years ago

Says the person posting from a web browser.

Shivetya• 7 years ago

According to Apple[1] Full mitigation requires using the Terminal app to enable an additional CPU instruction and disable hyper-threading processing technology.

So, if you have a system which you cannot expose it is a pretty big damper on performance.

[1] https://support.apple.com/en-us/HT210107

etaioinshrdlu• 7 years ago

And you also benefit from herd immunity. If 99.9% of people have mitigations, then the likelihood of your difficult timing attack working is quite low... so why bother trying?

userbinator3• 7 years ago

It's a little funny to see the "use AMD!" comments --- since from what I understand, Intel's optimisations that lead to these side-channels are specifically for performance, so using AMD instead of Intel might mean the same amount performance loss.

mfatica5• 7 years ago

AMD is in a better position here

scotty79• 7 years ago

Amd was walways better wit bang for buck. Intel just had the fastest CPUs for rich who nwed best single core performance. If Intel doesn't have the fastest CPUs then AMD is just hands down cheaper and Intel has no advantage.

Kye• 7 years ago

Alternate perspective: AMD not doing these risky optimizations reflects a better engineering culture, or at least less risky priorities.

Go with Intel and risk losing a huge chunk of your performance some day, or go with AMD and know what to expect. Different people and companies have different risk tolerances, so it's not an obvious choice one way or the other.

PedroBatista• 7 years ago

Current AMD offerings are generally beating Intel, and in the price-to-performance game it's a complete slaughter.

mda2• 7 years ago

They have completely different architectures, some optimizations can be done in safer ways.

dfrage7• 7 years ago

Isn't one of the "secrets" to AMD's post-486 success adopting the general approach the Pentium Pro took along with others in the 1990s, which in turn are based on IBM's 1960s Tomasulo out-of-order algorithm adding speculative execution? This general approach made all out-of-order speculative designs, including ARM, POWER and IBM Z vulnerable to Spectre, so I submit they're not "completely different".

Symmetry3• 7 years ago

Not at all. AMD does a hardware check for memory accesses in serial with the reading the data rather than in parrallel. This probably adds an FO4 or so of latency to the critical path of a read which could mean a lot of things from a design standpoint. Maybe you reduce your L1 cache size or associativity or such to make back that Fo4. Maybe you just say "All our stages are going to be 15 FO4s instead of 14" and you clock 5% slower but your pipeline gets shorter and you have more room for cleverness in other places. I'd be very surprised if the net performance impact was more than 1%.

By contrast having to deal with the issue in software or worse, by turning off SMT, is much more damaging and gets you the 10s of percents of performance impact that people are talking about.

snvzz2• 7 years ago

Might have been the case during AMD bulldozer. There, they were still doing well by surviving with a massive fab disadvantage.

From Zen thereon, they've been more or less on par in single threaded performance, and have the performance edge on multicore performance, while being cheaper and unaffected by most of these new CPU vulnerabilities, on top of that.

morrbo• 7 years ago

Does anyone actually have some benchmarks of the latest gen AMD vs the latest gen intel processors with all mitigations for spectre, meltdown, and the 10 other sidechannel/speculative execution vulnerabilities applied?

I'd genuinely be curious to find out what the eventual results are because as i understand it AMD is not too far away from Intel as a standalone processor, surely in a "real world" scenario they'd be significantly faster?

Someone12342• 7 years ago

It is a moving target. A lot of "day 1" patches have absolutely tanked performance only to regain performance later via smarter mitigations. Both Linux and Windows as well as microcode updates have all seen some of the performance loss regained.

And while it would be nice to call any point "the end" and measure then, as of two weeks ago they were still finding additional vulnerabilities and patching them. So if there is an end we aren't there yet.

That all being said, I am also curious on what the results would show. Just very hard to pin down.

rincebrain• 7 years ago

To be fair, at least Microsoft explicitly went with finding perf optimizations elsewhere to reduce the impact, not smarter mitigations (though they also eventually did that when they decided to go for retpoline).

noir_lord• 7 years ago

On multithreaded stuff AMD is killing it, they where slightly behind on IPC about 5% but the clocks are typically a bit lower.

I'd expect with both machines patched that clock for clock AMD are slightly ahead.

I've got a 2700X at home and it's a monster and I'd Zen2 is on the conservative end of the leaks/rumours it's going to be a total killer.

bitL• 7 years ago

Seems like Threadripper 3 might be delayed until 2020, so no 64c HEDT for a while...

dhx• 7 years ago

For Linux 5.0 and as of March this year, the performance impact of enabling Linux kernel mitigations for Spectre/Meltdown against various CPUs with the latest microcode (as of March) are:

Intel: -13% for the Core i9 7980XE, -17% for the 8086K.

AMD: -3% for the 2700X.

Reference: https://www.phoronix.com/scan.php?page=article&item=linux50-...

Phoronix is due to release new benchmarks tomorrow showing full impact from Spectre/Meltdown/L1TF/MDS. There are some initial benchmarks at https://www.phoronix.com/scan.php?page=news_item&px=MDS-Zomb...

adossi• 7 years ago

What about for the newer processors like the 9980XE, 9900K, etc.? I would have assumed that Intel's latest processors have some additional engineering in place to mitigate the spectre/meltdown performance impacts.

gnufx• 7 years ago

Surely the effect depends on what you're running, and you can't put a single number on it. There are actually somewhat contradictory results I've seen for HPC-type applications, and no useful analysis of them with low level profiling.

shereadsthenews6• 7 years ago

On realistic workload (branchy C++ server code, not SPEC and the other crap that Phoronuts like to report) the AMD CPUs are 50% slower ... Intel could pile on the mitigations and still be way ahead. The reason these mitigations cost Intel so much is because they have these speculation features and AMD just didn't ever have them. At least with the Intel parts you can decide for yourself whether to enable or disable them.

kllrnohj12• 7 years ago

Please provide a single benchmark of any kind to support that. It sure sounds like you're stuck in 2016, though, before Zen happened with its +52% IPC improvement over Excavator. Otherwise basically all clock for clock battles shows Zen in single-digit percentage range of Intel on IPC. Combined with Intel having a clock speed advantage that does give Intel +10-20% in pure single-thread performance, but that was pre-mitigations. If this vulnerability fix really does cost 25% that'd erase that advantage entirely.

And no, Intel is not unique in speculation capability.

mtgx3• 7 years ago

I bet Google isn't so giddy now about being FIRST!! [1] with Skylake in the data center a couple of years ago, or "going on all-in" with Intel on the Chromebooks (it didn't even give AMD a chance until very recently...), despite Chrome OS being one of the very few operating systems that are truly architecture agnostic.

Now it's paying dearly for that mistake, with up to 40% performance loss on Chromebooks due to the disabling of HT:

https://www.techrepublic.com/article/mds-vulnerabilities-lea...

Google broke one of the most basic business rules: never rely on a single supplier. You're always worse off in the end, even if the exclusivity deals seem very tempting in the short-term.

[1] https://cloud.google.com/blog/products/gcp/compute-engine-up...

yjftsjthsd-h4• 7 years ago

"going on all-in" with Intel on the Chromebooks

Weren't Chromebooks on ARM before x86?

Google broke one of the most basic business rules: never rely on a single supplier. You're always worse off in the end, even if the exclusivity deals seem very tempting in the short-term.

I don't think this is universally true, although I agree that it's probably prudent. Diverse options/suppliers are a risk mitigation, but they do have a cost.

qaq• 7 years ago

Intel vulnerabilities are providing additional 25% revenue to Cloud Providers.

bufferoverflow• 7 years ago

To Intel maybe. It's a loss to the cloud providers and their customers.

iopq• 7 years ago

I pay by the hour, if I need to provision more servers AWS doesn't take a hit, I do

kasey_junk• 7 years ago

Most cloud providers are trying to move up the value chain & provide higher margin differentiated services (managed db, queues, etc) instead of staying at the race to the bottom vm market.

These intel mitigations impact those services just like everyone else.

tln• 7 years ago

Does it? If customers can't run code on those servers, then these CPU level side channel attacks aren't an issue.

This assumes the servers for S3 etc are dedicated to the task.

IfOnlyYouKnew• 7 years ago

In the immediate short term: maybe. But fundamentally, these vulnerabilities are (hyper)threatening their business model, because not sharing the infrastructure suddenly becomes far more attractive.

So to the extend that you're suggesting that cloud providers are happy, or may even have had a hand in this: nahh.

Aissen• 7 years ago

Not really: https://twitter.com/waxzce/status/1129381076160454657

The performance loss is not visible for our customers, we have to manage the loss in ourselves. It’s a kind of hidden defect.

ithkuil• 7 years ago

other replies mentioned that the cost of the slowdown is absorbed by the cloud provider; let's just pretend it's fully passed down to the user and only nitpick at the math behind the joke:

Imagine a slowdown (performance penalty) of 50%. It would double the time to complete a task, thus doubling the cost (expressed in cost increase that would be a 100% increase!).

Generic formula: 1÷(1−x)−1

For x = 25% ---> 1÷(1−0.25)−1 = 33% duration (cost) increase

qes3• 7 years ago

I run a high traffic web service in the cloud. Well over a billion requests served per month from the origin. We run our CPU's around ~60%.

We've seen maybe a 2% hit from spectre & meltdown mitigations. It's hard to even tell because it's a small enough amount that it tends to get lost in the noise.

imhoguy• 7 years ago

Now how much this 25% is in terms of CO2 emissions?

the-dude• 7 years ago

About 25%.

kierank2• 7 years ago

This is an argument for organisations to go back to being on-prem, running their non-public facing workloads privately.

nixgeek• 7 years ago

How's that an improvement over using 'Dedicated Instances' or the 'Bare Metal' offerings which exist with cloud providers, where you're not colocated with other entities but still enjoy all the benefits of how the infrastructure is managed, flexibility in upscale/downscale (instead of acquiring capital assets which depreciate over 36-48 months).

kierank2• 7 years ago

External access to the bare metal is still possible and thus information can be leaked via Meltdown and Spectre

LinuxBender3• 7 years ago

We are doing perf testing of the AMD versions of HP/Dell servers. We may tech refresh about 40k servers with epyc-2 if performance is better than Intel. It's about time for some new servers anyway.

tgtweak• 7 years ago

It seems that they're on track to get epyc into OEMs and cloud providers in quantity. They've been successfully growing the ecosystem in the server space over the last 2 years (Supermicro, dell, HPE, Lenovo all pushing epyc sku's now) so they're very well poised to capitalize on it if they can keep up with supply. The power efficiency gains on 7nm will be pretty tempting to cloud providers and colocation clients alike. If you can put 256 cores (4 epyc2 sockets) into a 1U server and keep it under 1200W that's going to be a landslide win for density and running costs. Factor in pcie 4.0 and things are looking pretty sexy.

I've read that wafer yeilds on zen2 are 70%+ which gives a huge advantage on cost and production efficiency to AMD. I think Intel's skylake yield for an equivalent 28 core die is sub-40%. If anything the limiting factor is going to be TSMCs ability to give AMD capacity on it's very crowded and in-demand 7nm fab.

LinuxBender3• 7 years ago

Dell and HP already have them in quantity. We just have to perform due diligence to ensure all of our workloads will remain the same or better performance. Thus far, testing is looking good.

bArray• 7 years ago

Quick question: Why put all of your eggs in one basket? I know you want to maximize profits, but you also want to minimize risk? For all anybody knows the next big CPU CVE could be on AMD architecture, especially as people will now being taking the processor much more seriously.

LinuxBender3• 7 years ago

Our eggs are already all in one basket across two vendors and that did already bite us. Cost wise, it doesn't make sense to load balance across Intel and AMD. We have to pick one or spares cost are doubled and tooling has more overhead.

In a few years, if Intel have their act together and AMD start having more vulns, then we may swing back the other direction. Only time and vendor actions will tell. This assumes we even move to AMD.

shereadsthenews6• 7 years ago

Have fun replacing all of the CPUs after a few months. There hasn't been an AMD CPU since Opteron that worked correctly on the first stepping. EPYC has a huge number of errata ranging from bit flips to voltage dropouts to hangs and reboots, most of which remain unfixed. I mean "Load Operation May Receive Stale Data From Older Store Operation" is a pretty serious erratum. And AMD has a history of this going back to Barcelona which just flat out didn't work (workarounds for broken Barcelona B3 parts cost 50% performance or more). They don't have the resources to qualify new parts properly and they rely on customers to find all the bugs for them.

Intel vulnerabilities costing 25% CPU performance loss to a cloud provider

Related Stories