Google reveals its servers all contain custom security silicon

http://www.theregister.co.uk/2017/01/16/google_reveals_its_servers_all_contain_custom_security_silicon/

by chris-at • 10 years ago

363 125 10 years ago

theregister.co.uk

Duobix• 10 years ago

I barely dug into the article when It came to me: Google just did a lockout chip, 1980s Nintendo style.

na85• 10 years ago

I suppose if you're looking for a sound bite, yes.

But whereas Nintendo's chip was DRM, this Google chip appears to be more about determinism in boots and server provisioning, allowing them to immediately cut out a server that appears malicious or that has been compromised.

I.e. pry open case to insert an implant, chip notices bios has been altered, sends the "don't trust me" message to the network.

digi_owl• 10 years ago

Makes me think of Intel's IME. It has legitimate uses on corporate desktops and servers. But when it makes its way to consumer desktops it runs face first into a massive conflict of interest.

woliveirajr• 10 years ago

Disks get the following treatment:
“We enable hardware encryption support in our hard drives and SSDs and meticulously track each drive through its lifecycle. Before a decommissioned encrypted storage device can physically leave our custody, it is cleaned using a multi-step process that includes two independent verifications. Devices that do not pass this wiping procedure are physically destroyed (e.g. shredded) on-premise.”

Interesting. There were discussions on the past on how to clean HDD, if multiple-passes were really necessary or not.

Then SDD become the problem, since there is a interface between what you see (from the OS) and where the data really is (inside those chips). Now Google not only encrypts data before saving (that should be enough, no?) but also tries to wipe using multiple passes and 2 verifications.

Wonder how many companies do that.

nom2• 10 years ago

'multi-step process' doesn't imply they are wiping more than once, only that their procedure consists of multiple steps (e.g. wipe + verification)

sphix• 10 years ago

The article specifies two independent verifications.

problems2• 10 years ago

If you use on-board crypto on most SSDs, there's a dedicated place for key storage and using the SSD's onboard wipe feature just changes the key and TRIMs the whole drive.

Most of these drives use cryptographic keys even if you don't use a password on the device. Think about it as an SSD manufacturer - what's the easiest way to wipe a drive? To actually go and zero out every cell on the disk or to overwrite a very small cryptographic key with a new one - effectively destroying the data without the need for any other write cycles to occur.

Pretty easy to verify - if you have an SSD with support for this, which most do now.

throwawayish• 10 years ago

Think about it as an SSD manufacturer - what's the easiest way to wipe a drive?

That's not the reason why encryption is always on. Flash endurance is; encrypting the data before FEC means that it will have a random distribution, which avoids pathological worst cases with certain workloads. You could also use a different (cheaper) scrambler than AES (like CPUs do [1]), but since encryption is a marketable feature...

[1] Which are also switching to using AES and offering memory encryption in current mainstream architectures.

tscs372• 10 years ago

I remember reading up on the BND (German Intelligence Agency) Guidelines on how they wipe their data.

They basically wipe the drive first and verify it appears to be wiped and then shred it. The highest level allows for only 0.5mm^2 sized particles with tolerance up to 1.5mm^2.

If data is encrypted, then in theory destroying the key should be sufficient given that the encryption is good (Chapoly or AES)

kevincox• 10 years ago

It also depends on how long you want the data to be safe. So if you are storing user data you probably don't want to release drives containing encrypted user data as you don't long they wanted that data to remain secret for.

Imagine that your user was Coca-cola and they uploaded their recipe. They wouldn't be happy if in 100 years the encryption was cracked.

Far fetched, maybe slightly but a real consideration.

dspillett• 10 years ago

* Devices that do not pass this wiping procedure are physically destroyed on-premise*

If you are going to go to that much effort why not physically destroy the drive anyway? You might still want to test them to flag up problems in your process, but if you have the facility locally why not use it for all drives instead of paying an external party to do some of them?

detaro• 10 years ago

They don't say that the drives are all destroyed by the external parties. And even if they are, I could imagine that proper recycling is easier if you have the full drives and not a shredded mixture of all the materials in it.

dx0343• 10 years ago

Thought they'd shred all disks? This reads as most drives (all passing the test) are sold to others.

Would be interesting to know where you can buy old Google disks? Should be rather high volume.

CuriousCosmic• 10 years ago

They likely recycle the disks rather than sell them. I'd imagine that reclaiming the materials is cheaper with intact disks compared to shredding.

jabl• 10 years ago

I guess at their scale and the business they're in, it's cheaper to dedicate a few engineer-hours/days/weeks to implement an overkill wiping procedure rather than arguing with potential customers that "no, it's not really feasible to extract usable data with an electron microscope despite what you have read on the interwebs". Or even worse, losing said customers if they're not persuaded by your arguments.

donavanm2• 10 years ago

Check out "Opal" and "SED" https://en.m.wikipedia.org/wiki/Opal_Storage_Specification. Many (most?) drives support it these days. When I briefly looked a year or two ago basically all drives had the physical capability, but some firmwares didnt expose it to the iser. As always key management is the hassle.

sergiosgc4• 10 years ago

This is another signal of an interesting development on the hardware front. What used to be decoupled, with some companies offering hardware, and different companies buying hardware, is now coupled and hidden within these mega-companies (Google, Amazon, FB).

Google is big enough to develop a trusted hardware solution for internal use only, it has no financial need to sell it. Worse, due to competitiveness in the cloud segment, it is dis-incentivized from selling the solution.

Amazon Glacier is another one. It's an interesting long-term storage solution, whose hardware implementation is unavailable to the market, since AMZN can better explore it as a service under AWS.

We are heading onto a more closed ecosystem than we are used to up until here. The cloud, which gave us the immense positive benefit of moving all capex to opex, is birthing this immense negative side effect of closing off hardware implementations in favour of exploring the added value in the form of services.

nom2• 10 years ago

"We are heading onto a more closed ecosystem than we are used to up until here. "

It wouldn't be more open even if they didn't spin their own hardware. You won't ever see it nor have access to it on a low level - it's 'the cloud'. The only thing it tells us is that they have reached a scale where custom hardware makes things cheaper, more reliable and more manageable for them.

deelowe• 10 years ago

Google joined open compute last year.

sergiosgc4• 10 years ago

It wouldn't be more open even if they didn't spin their own hardware.

Of course. If the alternative is non-development, we are better off. However, my text states that the alternative is de-coupling: One company develops and markets the innovative hardware solution, other companies build services on top of that solution. This would be more open, it would present a more liquid market.

_yosefk• 10 years ago

It's not the cloud - it's the sad downside of the democratization of hardware design, as in fabs like TSMC and IP companies like ARM making it relatively cheap to make your own chips with competitive functionality in a wide range of areas. There's a lot of custom hardware outside the cloud, say in embedded electronics, that's just as closed as the stuff in server farms - closed specs and no way to program the thing, increasingly often no ability to run binaries unsigned by someone in a small set of vendors.

Moreover, the GPUs and more so, DSPs and ISPs in your phone or PC are hidden from you in that they run code written by a very small number of people. You don't even have an idea how many small DSP cores are scattered throughout a desktop-class chip, let alone what they do or how to program them. Effectively it's for internal use of a very small number of hardware and software vendors, and the software is very much tied to the hardware.

The reason computing hardware used to be open is that very few could make it and they only stood to gain from making it usable in as many applications as possible, or at least so they thought. Once (almost-)cloning hardware products became increasingly cheap with less and less vertically intergated hardware vendors (from fab to design), vertical integration moved to design+software because that's how you fend off competition today.

sergiosgc4• 10 years ago

Very much true. I don't think the two reasons are exclusive; it is probably a factor mix. Lowered development costs, economies of scale, need for competitive edge in a market that slopes into commoditization.

ust• 10 years ago

It seems a bit counterintuitive that open hardware results in less choice, so I disagree. I think that hardware is getting more and more open, also drivers for it. With FPGAs it is (relatively) straightforward for one to create it's own crypto processor and integrate it in the system. Also PCBs are getting easier and cheaper to make. I hope that also there will be some open PCB designs that incorporate some kind of crypto chips and functionalities outside of CPUs, so everyone can start creating their own servers, if desired.

Didn't also Facebook started some open server hardware initiative? I don't remember what happened with that...

I do agree that the current status is not great, and that we could all benefit from more open hardware design. I think that it would also benefit large companies as well.

azernik• 10 years ago

Another good example that I have experience with - closed firmware blobs in everyone's wifi chipsets and cell phone basebands. Early-standard ARM processors are cheap enough to embed in peripherals these days, making it easy to hide your functionality in hard-to-extract embedded software.

rwmj• 10 years ago

Pretty sure this is nothing to do with Google manufacturing their own silicon. It's an open secret that standard Intel server chips contain "special silicon" with features which are only switched on for certain customers. I'm pretty sure this is what Google is referring to. Source: https://techreport.com/forums/viewtopic.php?t=118026

vgt• 10 years ago

It's a sign of changing times indeed, but for the consumer's benefit.

It is absolutely in Google's best interests to externalize security for its customers as a differentiator of Google Cloud. The parent article itself links to the white paper that outlines how this is done for Google Cloud.

I understand how one may consider this a "closed ecosystem" from one perspective. However, from a customer point of view any startup or mom-and-pop can leverage these very complex and expensive world-class security developments, whereas in the past this access has been reserved to the very select few that could afford it. When the barrier to entry is lowered and access is commoditized, customer wins.

(work at Google cloud)

pm902• 10 years ago

I understand how one may consider this a "closed ecosystem" from one perspective. However, from a customer point of view any startup or mom-and-pop can leverage these very complex and expensive world-class security developments, whereas in the past this access has been reserved to the very select few that could afford it. When the barrier to entry is lowered and access is commoditized, customer wins.

I don't understand why the customer can't get these benefits AND the ecosystem be open as well.

sergiosgc4• 10 years ago

However, from a customer point of view any startup or mom-and-pop can leverage these very complex and expensive world-class security developments, whereas in the past this access has been reserved to the very select few that could afford it.

I don't expect startups or mom-and-pop's to build internal clouds. I do expect medium to large companies to do so. The current market turns innovations such as these into competitive advantages of Google, instead of directly exposing the innovation to the market and allowing incorporation of its advantage into anyone's product. It's a less liquid market. Either you buy all of Google's solution as a package, or you buy none. You can't pick, sort and mash a solution of your own from disparate parts.

Note that Google here is just an example. All companies of similar size in IT are doing the same, and strategically it is the correct option (for them). I'm just stating that the overall result is sub-optimal through a collusion of disparate factors.

a3n• 10 years ago

any startup or mom-and-pop can leverage these very complex and expensive world-class security developments,

Until a random cosmic ray triggers Google to revoke access for mom and pop.

There's tradeoffs, and most people don't discover them until the random cosmic ray hits the fan. And, to be fair, most Google customers probably never encounter a RCR event.

ocdtrekkie3• 10 years ago

Let's be clear: The customer never wins when the product is closed. Google used to understand that: https://googleblog.blogspot.com/2009/12/meaning-of-open.html

In realty, mom-and-pop don't need these security developments, because mom-and-pop have much less attack surface on a server running in their back room. The cloud necessitates it be possible to manage a server over the Internet, but for many situations that isn't necessary. And in many cases, the limited needs a mom-and-pop company has doesn't require their infrastructure be public facing on the Internet at all.

I'd argue the only reason one needs these "world-class security developments" is because Google itself is a world-class target. The sort of threats you're defending against would almost never be necessary for a smaller business with an on-premises solution to be concerned about.

Many small business internally need little more than a shared network drive, rudimentary user management, etc. And you'd be stunned how many businesses today still operate off a single AOL mail account.

hedora2• 10 years ago

I've worked on both sides of the fence. My take on it is that the cloud is raising the security bar in many dimensions, but lowering it in otheres. Frankly, for-profit surveillance (and government conspirators) is at the top of my list of security concerns.

The good news is that the "evil" mega corps aren't so evil, and generally contribute back.

Also, the bar for custom hardware is dropping, and the recent moore's law stall means a good board design can live for 3-6 years instead of 1-2. In turn, this means that niche operating systems like BSD and opensource Solaris have increasingly stable hardware targets.

Taken together, this is great news.

pjc50• 10 years ago

immense positive benefit of moving all capex to opex

Hmm. Is that really a benefit or Enron-ing yourself? I can see how it allows efficient ramp-up of small companies which can't afford a whole sysadmin for ops, but the big downside is that the return on capital goes to the megacompanies, to whom you are paying rent.

I can sort of see why it's happened, because it's very hard to capture the added value of software by selling it. Especially in this area, the value of software is driven to zero. Whereas restructuring as a service and putting up barriers to entry brings back the margins for the developer.

We almost need an update to Coase's Theory of the Firm to account for the disruptive effects of IP-heavy organisation.

pm902• 10 years ago

It reduces the resources needed to bootstrap your own company. Of course once it reaches a certain size, it makes it more cost effective to host your resources internally. There are also many organizations who do not want to deal with a whole division dedicated to maintaining a reliable, distributed, secure computer network for their employees and would be more willing to pay for the cloud rather than the extra people required to maintain the in-house solution.

cookiecaper4• 10 years ago

Is that really a benefit or Enron-ing yourself?

I read it as sarcasm. ;)

GFK_of_xmaspast• 10 years ago

So basically what was assumed to be a linear trend is turning out to be a dialectic.

pdkl952• 10 years ago

More ground is lost in the cold[1] civil war[2] for control of the General Purpose Computer. I hope that everyone choosing to centralize computing power likes the future they are creating.

[1] https://www.youtube.com/watch?v=nT-TGvYOBpI#t=2824 (sec. 10 - http://geer.tinho.net/geer.blackhat.6viii14.txt )

[2] http://boingboing.net/2012/08/23/civilwar.html

pacala• 10 years ago

What is different about centralized compute power compared with centralized energy production?

waqf• 10 years ago

It's not a choice†, it's market forces. You'll never change the world effectively if you don't start by correctly diagnosing the problem.

†(as far as your objective is concerned. Yes, Google could choose not to have secure hardware, but that wouldn't change the end result that the market leaders in five years will have secure hardware — Google just wouldn't be among them.)

bsaul• 10 years ago

it's probably going to be just a cycle. Much like energy is very centralized with a lot of custom hardware ( think nuclear power plants) but tends to have decentralized alternatives ( solar panels ) for some advantages, you'll probably end up with the same cycle with computing power. Once a single affordable machine will be able to serve all your applications to all your customers with zero maintenance cost ( because innovation will keep going on), you'll probably switch away from the cloud.

it may be in a long time though..

amelius3• 10 years ago

I think we should limit companies to a maximum of N employees.

This will ensure more modularity in the market. And more competition as well, because barriers are lower.

gnu8• 10 years ago

I like this idea. Maybe we should limit governments to a population of N citizens for the same reasons.

cobbzilla• 10 years ago

interesting idea, impossible to enforce. we'd end up with even more contractors, and mega-conglomerates of hundreds of companies with ~N employees each.

true_religion• 10 years ago

I read a sci-fi story with this premise.

Companies could only be as large as X before starting to pay prohibitively large taxes in order to stay 'for profit entities' or become companies devoted to the public good.

So you'd end up with large telecoms who were non-profits dedicated to improving the level of global interconnectivity, and lots and lots of tiny 2-10 man companies that did research or sales.

matt_wulfeck• 10 years ago

The fact that you're reading an article about a paper Google just published suggests it's not as closed as your doomsaying might suggest. Also the paper notes that Google is one of the largest contributors of bugs and CVEs to KVM, which is a security tide that will raise a lot of boats.

idlewords• 10 years ago

I don't think your use of "doomsaying" is fair, and detracts from a useful comment.

bitJericho• 10 years ago

This is how things have always been. It's the reason for our patent system, to incentivize companies to open up their proprietary tech.

zer0gravity• 10 years ago

We are heading onto a more closed ecosystem than we are used to up until here.

Not necessarily. Balancing forces always come up when this kind of things happen. The closing in of big players may give a big boost to open source hardware alternatives, which combined with the advent of 3d printing, may very well lead to the democratization of hardware...

jgalt212• 10 years ago

gave us the immense positive benefit of moving all capex to opex

Isn't it more correctly stated moving fixed costs to marginal costs? capex is mostly fixed, and opex is mostly variable, so you are not strictly incorrect.

stepanhruda• 10 years ago

http://opencompute.org proves the exact opposite

zengid• 10 years ago

Looking at what's going on in Shenzhen[1] in regards to hardware hacking, I'm not worried.

[1] https://youtu.be/SGJ5cZnoodY

eva1984• 10 years ago

Thinking from another perspective, are those hardware just hardware? Not really, they are solutions, with tons of software supporting them. There is no point selling individual pieces, and they will not work out of box.

donavanm2• 10 years ago

Is it really any different in practice? TPM is installed in basically every computer used in basically none. Even though its been a uniquitous "standard" for two decades its effectively impossible to correctly do attestation.

Similar arguments could be made for UFI and similar. Just because some can write code for it only a handful of major suppliers actually do.

Are those any more accessible than this?

taeric• 10 years ago

I think you would have a hard time showing this is any less closed than earlier days. For good and bad. IBM used to be at your door to replace hardware you didn't know was broken yet, and couldn't have fixed if you had known.

NelsonMinar• 10 years ago

This is what security looks like when your threat model is well funded government agencies.

dx0343• 10 years ago

Exactly, and I think it's worth noting that they likely only apply this level of security because of state actors. Which then shows that they are trying to prevent eavesdropping by NSA & Co., they probably just realized too late how far advanced they were.

DannyB2• 10 years ago

. . . how far advanced, and the lengths they would go to.

nickik4• 10 years ago

Eric Grosse from Google says as much here, ...

Edit, it seems I have copied the wrong video, please us the link in the child comment

a_imho5• 10 years ago

Which then shows that they are trying to prevent eavesdropping by NSA & Co.

Why would the NSA eavesdrop on Google, they are in bed with them, aren't they?

nickpsecurity2• 10 years ago

Same threat model with black hats. Here's a recent example from high-security field that's remarkably simple but stops tons of attacks:

http://www.cc.gatech.edu/grads/c/csong43/oakland16-hdfi.pdf

Its related work & extension page has a ton of references to other things showing just how much work it is to stop regular black hats in systems without verified software. Nation states just do more of the same stuff.

Edmison's has a nice survey & design for when you don't trust anything outside the SOC:

https://theses.lib.vt.edu/theses/available/etd-10112006-2048...

NSA targets those levels, too. At that point, your bases are covered so long as the fab receives and doesn't alter your design. Plus the complex tooling works...

amelius3• 10 years ago

But if they don't own an IC fab, how do they know it is secure?

swegg2• 10 years ago

Basically splitting the trusted circuit and testing the parts separately. This requires a trusted master circuit, but it can be arbitrarily small.

See https://perso.uclouvain.be/fstandae/PUBLIS/177.pdf

amelius3• 10 years ago

But what if the malicious code is time activated? (just an example)

nickpsecurity2• 10 years ago

They don't. Too many attack vectors. I illustrate here:

https://news.ycombinator.com/item?id=10468624

My solution was to print the TCB on a process node that was verifiable by eye. Then, verify a random sample of each batch. Possibly speed it up with image processing algorithms if producing the same component or components.

Note: Deviations from intended circuitry in deep sub-micron can have measurable differences at analog or RF level. DARPA is funding research to do such things. A monitor at visible node could then be combined with CPU's on cutting-edge node. Common practice in commercial sector is obfuscations, though.

foobiekr• 10 years ago

The actual document - https://cloud.google.com/security/security-design/ - was linked previously.

It is interesting that they are doing some variant of trusted computing mostly because their homogeneity would allow Google to build a robust containment architecture with much more rigorous whitelisting and a robust SW distribution rules that go beyond what a measuring host and local SW bundle verification can do. So defense in depth.

We (skyport systems) do the same thing as a service for enterprises (we sell and operate cloud-managed trusted systems as a service) and I will say it's pretty hard to get people to think about depth and trustworthiness when the entire security industry has trained CIOs to believe that all they need to do is install some random agent on their VMs.

Good for Google.

gerbilly• 10 years ago

Still it doesn't make me want to use their services.

They may indeed be really good at securing their data but 'their data' ironically is derived from my emails and browsing history and that of my friends.

tlb• 10 years ago

"Before a decommissioned encrypted storage device can physically leave our custody, it is cleaned using a multi-step process that includes two independent verifications. Devices that do not pass this wiping procedure are physically destroyed (e.g. shredded) on-premise"

Why not just shred all decommissioned disks? Someone must be buying them for enough money that Google created a multi-step process for cleaning and verifying them. Presumably Google keeps disks in commission until they're no longer economic in their own operation.

So, does anyone know about the operation that makes profitable use of disks that are no longer economic for Google?

hodgesrm• 10 years ago

Just speculation but they could be used elsewhere in Google for test/dev or pre-prod environments.

wang_li• 10 years ago

I'd imagine it's easier to recycle an intact HDD than a jumbled mix of silicon, ceramic, steel, aluminum and what not.

phire• 10 years ago

Probably because of the verification.

You can't really verify a shredding, the pile of shredded remains no-longer has a serial number. I assume the cleaning process cryptographically verifies the identify of the disk both before and after the wipe, making it impossible to sneak a drive out.

For those drives which fail the cleaning process, they probably have a complex process with multiple witnesses to ensure it actually gets shredded.

willk• 10 years ago

My guess to the decommissioned disks is that HDD manufactures will sometimes give a hefty discount on disks if they can have them back at the end to run diagnostics on them. My company has a no disks leave the company policy and there has been talk about modifying this for the discount on disks.

bogomipz3• 10 years ago

I was curious about this:

"There's plenty more in the document, like news that Google's public cloud runs virtual machines in a custom version of the KVM hypervisor."

Does anyone know if this "container inside kvm" is true of their internal infrastructure as well or its just an extra layer of security for their public facing cloud?

wmf2• 10 years ago

Internal Google stuff does not use KVM and that's one reason it took them a while to offer VMs — they had little experience with it.

bogomipz3• 10 years ago

Do you or anyone else know if there is another reason for doing this besides security?

DanielDent• 10 years ago

A lot of stuff from this made it's way into the chromebook. There's a verified boot process, hardware assisted key management, rollback protection, ...

And it's all open source and nicely documented for anyone who cares to look. With a bit of work you can actually create your own chain of trust and run your own verified boot process.

It's very cool.

knowaveragejoe• 10 years ago

What is this called? I'd like to look into it, but cursory searching is giving me more vague results.

Google reveals its servers all contain custom security silicon

Related Stories