Report: Apple building its own servers to prevent snooping

http://9to5mac.com/2016/03/23/apple-cloud-infrastructure-servers-snooping/

by dankohn1 • 10 years ago

231 105 10 years ago

dang• 10 years ago

Url changed from http://daringfireball.net/linked/2016/03/23/apple-data-cente..., which points to this.

samstave4• 10 years ago

"Apple has long suspected that servers it ordered from the traditional supply chain were intercepted during shipping, with additional chips and firmware added to them by unknown third parties in order to make them vulnerable to infiltration"

Wow, so this is very interesting given, pretty much, everything that has been going on.

Will apple actually be the bastion of freedom (both in market and privacy) that the US supposedly stands for??

Google makes its own machines, as does facebook (actually more interested in FB's fiber switches/routers, but thats beside the point)....

But Apple has made "servers" for years... I guess they didnt consume them in their own DCs?? So I basically take it that they are effectively joining their take on Open Compute (mobos that can be mounted in controlled environments, where controlled now also means they can ID if anything was modded/changed in shipping?)

EDIT: I would really like to know how long "long" is from "apple long suspected"....

I was informed of NSA back-doors in Cisco gear in 1997 - WTF is Cisco's stance on any of this -- I haven't heard anything from them at all (or I missed anything they said)

seanp2k2• 10 years ago

Xserves were replaced a few years ago. The current "replacement" is the Mac Pro. I highly doubt that all of iCloud runs on racks filled with the little cylinders.

https://en.wikipedia.org/wiki/Xserve http://readwrite.com/2014/08/26/apple-icloud-amazon-web-serv...

protomyth2• 10 years ago

If they were really paranoid they could build a new Xserve powered by an offshoot of their A series ARM chips. To help defray costs, they could actually sell the thing.

ipsum2• 10 years ago

Maybe not Apple data centers, but it is possible to have a data center of Mac pros: http://photos.imgix.com/racking-mac-pros

jonathankoren• 10 years ago

Sounds like NSA's Tailored Access Operations[0], and other equivalent agencies. When you're a big enough target, state actors bring the full force of the state's espionage tools to bear.

[0] http://www.pbs.org/wgbh/frontline/article/how-the-nsas-secre...

samstave4• 10 years ago

Exactly!

How many machines have you purchased in your career? I've purchased THOUSANDS - which of these have been intercepted and compromised?

I have no clue.

iam-TJ2• 10 years ago

It occurs to me if the company is that big, or that big of a target, it can manage to collect its servers directly from the manufacturer/supplier rather than rely on a third unrelated party for transportation.

Spooky232• 10 years ago

Or the Chinese equivalent. Or both.

JustSomeNobody2• 10 years ago

Intercepted during shipping?

How long would an operation like that take? And can't a company with Apple's clout ask the supplier for a spec sheet that lists all the chips and a diagram showing the location of said chips?

samstave4• 10 years ago

Of course they can, which is why perhaps apple "long suspected"... maybe shipping times were fucked and apple noticed - but my question is WHEN did apple notice???

jkestner2• 10 years ago

You sound incredulous. It's a thing. http://www.pcworld.com/article/2083300/report-nsa-intercepts...

mc322• 10 years ago

This doesn't make much sense. Even if apple made their own, unless they will be shipping with their own fleet of delivery trucks with armed guards not on the take, how would this differ from shipping dell machines, or if they do go this route, why not use their own guarded shipping for Dells and netapp stuff?

Not only that, but they can confirm the firmware after receiving the equipment, unless they suspect the CPUs machine code, but then who will they get their CPUs from?

And besides, it's not like they'll be tearing out all the netapp, dells and Cisco --everyone even amazon have them all, from what I hear.

randall• 10 years ago

It's a lot easier to diff if you invented the originals.

mtgx• 10 years ago

They could do stuff like this for their own servers:

https://www.kickstarter.com/projects/designshift/orwl-the-fi...

But they may want to build their own chips, too, because they probably shouldn't trust Intel (they do some of that already, but they may need more powerful chips for their datacenters).

sna1l• 10 years ago

I have a buddy who was super high up in a telecom back in the late 90s/early 00s and he mentioned that all the big telecoms basically voluntarily were creating traffic backdoors for the government.

When he mentioned it, he didn't even blink. It was eerie.

samstave4• 10 years ago

Echelon.

It is real.

It has been around forever.

tetheno2• 10 years ago

I don't know how Apple's servers security fares now, but I took a quick look 4 years ago and it was notoriously bad. They were responsive and solved the exploitable bugs soon after I notified them. No bounties though.

If their software and network security is similar now.. then they should spend resources there rather than care too much about modified hardware by a governmental agency.

hrktb• 10 years ago

Wouldn't it be completely separate teams ? I can't imagine the hardware team's efforts taking away engineers from teams handling software security.

mc322• 10 years ago

An issue at very large companies like Apple is that while they can be superb in one area or in one core product does not translate into being adept in other areas under different teams with different management styles, demands budgets, etc.

So unless Apple are going to put their best and poach other industry bests, their result will likely not deliver on premise or promise.

tetheno2• 10 years ago

Certainly they are different teams.

My comment just meant that I needed 1 hour to find a way to get a shell in a couple of their servers. That is more worrisome to me than the NSA snooping around. And I'm not a great pentester, I probably wouldn't find a bug in Google even if I spent a couple weeks.

ryao18• 10 years ago

They are HP built Intel systems running Linux.

ryao18• 10 years ago

Apple might want to talk to IBM about OpenPOWER. The POWER8 chips and chipsets lack known internal micro controllers that have their own flash unlike Intel's chips that have the Intel management engine. Consequently, there is nothing there to flash with malware. Also, all of the system firmware is open source. That is the motivation behind the Talos workstation board, provided that they get enough interest for an initial run:

https://raptorengineeringinc.com/TALOS/prerelease.php

If Apple is concerned about tampering enroute, they could have the flash chips for the system firmware provided separately by a trusted party and transported like a bank shipment. Then flash and install them at the datacenter. That should thwart adversaries who cannot do their own manufacturing runs of modified versions of the chips, which is just about everyone. I guess the manufacturer/fabrication plant could do a custom compromised chip, but given that the costs involved are prohibitive, I doubt that would happen.

Apple could do the same with the firmware and flash for every other component in their datacenter that has a microprocessor such as the hard drives, the NICs, etcetera. They are large enough that part manufacturers would likely turn over the source code for their firmware in order to secure their business along with anything else that they need/want.

avn21093• 10 years ago

> "...transported like a bank shipment..."

If you think the NSA/state level adversaries have not penetrated bank shipments, you are dangerously naive. Instead, maybe transport it like the CIA moves the communication hardware for its quiet rooms, or like the Russians move the bugs destined for US embassies.

Apple is interesting because it's one of the few organizations that could plausibly do this.

ryao18• 10 years ago

I said bank shipment because I did not know of a more secure way of transporting a high value target. :/

hammock• 10 years ago

What is a quiet room?

PeCaN• 10 years ago

I think Apple would probably just use ARM chips; they already have the license and they have lots of experience designing their own.

OpenPOWER is awesome though. I'm hoping Talos will get enough interest; I'm ready to throw money at them if it succeeds. POWER chips are super good, I don't know how IBM has managed to keep up with Intel's R&D but they're definitely competitive in performance.

1. http://0pf.org/j-core.html

ryao18• 10 years ago

I doubt Apple would go so far as designing a low volume server processor for their datacenters. It is probably not cost effective. Apple reusing their iDevice SoCs in microservers could work though.

The A9 has PCI Express lanes for NVMe flash that could be used to add wired networking and storage while the secure boot and code signing would allow them to build a microserver that only runs their code. This arrangement would also be incredibly cost effective for them.

They would want to give their SoCs have properly protected internal data paths and the necessary logic for ECC RAM before using them in micro-servers if those chip features are not already there though. If they are, I doubt that the ECC is being used in their iDevices.

bogomipz2• 10 years ago

Doesn't the management engine exist as part of the BIOS on the same NAND? Couldn't Apple simply have Intel give them a custom BIOS that is signed?

pgeorgi3• 10 years ago

Groups more powerful than Apple tried to convince Intel to give them the opportunity to run only their own custom code on those Intel-signed areas of the processors they buy. What seems possible is running code on top of and in addition to the existing system, see CompuTrace.

This isn't "simple" except perhaps for some US (and US only, by virtue of Intel being head-quartered there) agency with huge procurement budgets and persuasive legal instruments, and I'd expect push-back even then.

ryao18• 10 years ago

I have been told by the CoreBoot developers that it has its own separate flash inside the chipset. There is definitely something loaded by the BIOS though.

adrianN• 10 years ago

This is an interesting talk about Intel's management engine:

https://media.ccc.de/v/32c3-7352-towards_reasonably_trustwor...

RRRA• 10 years ago

When will we get secure PC architecture for the consumer market?

jen729w• 10 years ago

When it's in someone's financial interest to build it. Which is to say, never.

meddlepal• 10 years ago

Never.

ryao18• 10 years ago

When consumers start paying for things like Talos:

https://raptorengineeringinc.com/TALOS/prerelease.php

beeboop• 10 years ago

https://www.crowdsupply.com/sutajio-kosagi/novena as close as we have, at least in terms of verifiabily uncompromised.

ikeboy• 10 years ago

In theory, does a backdoored firmware run slower? If yes, then can you detect a backdoor by building one yourself and benchmarking?

Or are the margins of error on repeated benchmarks larger than any performance hit due to a backdoor, or can something be backdoored without any performance hit?

What about power consumption?

strooper• 10 years ago

Practically, backdoored firmware doesn't necessarily run services and communicates to the mothership all the time, nor does it communicate using conventional protocols. It creates invisible hole in the system beyond OS's reach to detect and repair it. Furthermore, the hardware (or chip based) backdoor is way more sophisticated, typically have been in use in the servers of foreign governments of interest. This chips can be used to take full control of a server, and the whole network when places strategically.

In another note, hard RTOS are not used in servers, so performance cannot be compared with benchmarking tools.

IncRnd• 10 years ago

> In another note, hard RTOS are not used in servers, so performance cannot be compared with benchmarking tools.

It may seem surprising, but performance is easily determined by side channel attacks.

https://www.google.com/search?q=blackhat+side+channel+web+ti...

rdancer7• 10 years ago

It runs at hardware speeds, and only some of the time, so not really. You can perform timing attacks, and see if you're running under a modified hypervisor, for example. But if you have a sniffer chip on the PCI bus or Northbridge, you won't be able to see it. Power consumption has been done as well, but you really need to know what it is you're looking for, and if it is just dormant for most of the time, how are you going to test for that?

Animats2• 10 years ago

Is Apple saying anything about what they found? This is a big deal. Who put something in their servers? NSA? The PLA? Samsung? Did they have any strange chips analyzed? There are companies that can take an IC apart and see what's inside.[1][2] It's not cheap, but Apple could afford it.

[1] http://www.siliconinvestigations.com/ [2] http://www.istgroup.com/english/3_service/03_01_detail.php?M...

bogomipz2• 10 years ago

The PLA?

Animats2• 10 years ago

People's Liberation Army. China's military.

beefsack• 10 years ago

No company is static, what will Apple look like in 10 years? What would happen with my data they recorded today should company policy take a shift?

I'm quite uneasy regardless of who holds my data.

uptown• 10 years ago

I feel like the entire world has gone insane, and every boundary is being pushed to its limits ... and then pushed beyond those limits. Where does this end?

ryao18• 10 years ago

The world was that way long before you realized it.

rdancer7• 10 years ago

No, this is post-9/11. It only started with the PATRIOT Act. I vividly remember the Clipper chip debates in Congress. This is that on steroids, with no disclosure what-so-ever. In the communist block, that was different, of course. But in the West, you used to laugh and sneer at Stasi and Soviet "papers, please!".

update• 10 years ago

we were never this powerful, though

rsync• 10 years ago

I'm sorry to say it, but I read this and think "plausible deniability for future-Apple when they open the (back) doors".

onRoadAgain23• 10 years ago

If this is a trend I guess HP wishes it would never have so readily worked with the NSA bugging servers.

benmmurphy• 10 years ago

if the NSA is doing this in the US against a US company isn't this illegal?

abysmal_cynic2• 10 years ago

"Legality", in this context, went out the window long ago. Power relentlessly weaponises "legality". Power uses it to attack its enemies and to shield the unethical activities it wishes to protect.

siliconviking• 10 years ago

Why not just contribute a few designs or some open source software and join OCP?

http://opencompute.org/

Last time I checked, Apple was not in the business of making data center equipment, so it's not like they would give up IP that is central to their business model.

abysmal_cynic2• 10 years ago

Like the FBI vs Apple case, this looks like marketing hype to me. Apple is increasingly selling on "security" they can't deliver.

(I mean, I'm sure they'll try to action something, but we all know security is a compromise not a guarantee).

smoser• 10 years ago

“[23:06] <ATPtipster> we've gotten Cisco equipment, Supermicro servers, and Seagate hard drives that have been tampered with [by the NSA]”

sickbeard• 10 years ago

So Apple is supposed to safeguard our data now? If you really want privacy don't expect big corp to keep it for you.

massemphasis2• 10 years ago

This is one of the faultier parts of having to answer to share holders. Apple is finding it more difficult to justify having their own entire supply chain rather than relying on other companies (i.e. Intel, IBM, Samsung, etc...)

If they didn't make that horrible mistake of their recent multi-billion dollar purchases, they'd have extra billion atleast towards a fab plant.

I'm laughing at Apple right now. Idiots. You Apple board... are stupid.

draw_down• 10 years ago

What? Why would they care about a billion dollars? They're sitting on around $200 billion cash.

massemphasis2• 10 years ago

They are seen as throwing away cash, and throwing away 1% of cash holdings is very bad.

Report: Apple building its own servers to prevent snooping

Related Stories