Google Has Spent $21 Billion on Data Centers

http://www.datacenterknowledge.com/archives/2013/09/17/google-has-spent-21-billion-on-data-centers/

by ajaxguy • 13 years ago

111 69 13 years ago

a3voices• 13 years ago

I wonder how much money Google has spent per user. $10? $50?

panacea• 13 years ago

...and did they improve the fortunes of their advertising client base, or merely suck the wind out the prevailing sail without offering anything in return?

fsckin• 13 years ago

You're thinking of Groupon.

lsc2• 13 years ago

It is a valid question. As a 'brand building' advertiser, I never felt like most of the google products delivered as much value as they claimed; they all seemed to be targeted at products people will buy after the first impression (which is... not, generally, how products in my sector are sold. People buy because they have heard of me. The 'hearing about me' is important, not the last click.) and the tracking mechanisms all seemed... naive. (now, that last part might just be the effect of them dumbing it down to the point where a small business owner, like me, can understand whats going on, but man... in that case, we advertisers trust google a lot.)

Of course, I'm small, cynical, stupid, and don't spend much effort on advertising, so it's possible my impression is wrong.

mrtksn• 13 years ago

according to here: http://www.internetworldstats.com/stats.htm

there are 2.5B internet users. since google is more than a search engine(android, gmail, adwords and so on) we can assume that most of these users somehow interact with google services, so just adding other expenses and deviding the number to 2.5 billion would give a quite accurate result.

brd• 13 years ago

Google hit a billion unique per month in May of 2011 and they've spent $21 billion in the last 7 years. Assuming no growth (which is absolutely not the case) they've spent an average of $3 per user per year.

tunisian123• 13 years ago

google doesn't throw her money away !! I'm positively sure that they will get it double a triple :) they don't do charity

dudus• 13 years ago

Google actually does a lot of charity

http://www.google.com/giving/

praptak• 13 years ago

Publicly traded companies even if they do "charity", they do it expecting good PR in return. Any other kind of charity (i.e. done without publicity, only to make the world a better place) would be against their duty to the shareholders.

27182818284• 13 years ago

google.org is their charitable extension.

antidaily• 13 years ago

So they should use Hadoop?

wikwocket2• 13 years ago

I'm not sure if you're posting sarcastically, but Hadoop was built based on whitepapers Google released about MapReduce and the Google File System. I think they are covered in this department!

bpicolo• 13 years ago

He's definitely being sarcastic.

DennisAleynikov2• 13 years ago

They're posting in reference to [1] I believe. It's supposed to be a funny comment. They are pretty damn well covered in this department, I agree.

[1] https://news.ycombinator.com/item?id=6398650

znowi• 13 years ago

How much of it was subsidized by NSA?

http://www.theguardian.com/world/2013/aug/23/nsa-prism-costs...

jonknee2• 13 years ago

Nearly nothing considering your article mentions "millions" for the whole program and Google has spent 21k million just on its own. Google makes more on interest from its cash on hand than it has received from the NSA.

melling• 13 years ago

It's really disappointing that you're responding to what is the #1 comment. The NSA noise just overshadows much of the meaningful discussion on HN.

domdip• 13 years ago

Has anyone dug in and figured out how these #s are impacted by MMI?

cobookman2• 13 years ago

My question is if google spent 21 billion, why did microsoft spend > 15 billion on its datacenters. How much datacenter resources is each microsoft customer using? It can't be from just office 365, Azure, and Bing. So is the majority from just software updates/downloads?

adamnemecek• 13 years ago

Don't forget about Xbox Live and Skype data centers. I'm guessing that those might be contributing a nontrivial percentage.

cobookman2• 13 years ago

Ahh forgot about Xbox Live. I know they recently announced having 300,000 servers: http://www.engadget.com/2013/05/21/microsoft-updates-xbox-li...

Mind you, 300,000 * 3k per server is still only 900 million.

dekhn• 13 years ago

I would guess they are trying to build up their cloud business. They have lots of cash, and they want a successful business. It's easy to justify a large buildout in that case, with the belief that the revenue will come. This is a common trap for innovators: you have to spend a lot just to stay in the game, while also preparing for the future. MSFT missed a lot of opportunities over the past few years due to their focus on Windows and Office (as Ballmer admitted recently) and now they have to play catchup.

InTheSwiss3• 13 years ago

Yeah Microsoft went all in on the "living room computer" with the Xbox and Xbox 360 which made sense at the time of the original Xbox however the vision they had wasn't what came to fruition. What did we get instead? Well we are still in the middle of the transition but it is most certainly away from super powerful set to boxes and more towards things like the Apple TV and Chromecast with powerful tablet computers doing most of the work and just pushing the output over wireless to the little receiver.

Now obviously MS saw this which is why we have things like Surface now but I am really surprised that they decided to kill off the Media Center Extender platform they had. They could have easily taken that and built a device similar to the Chomecast/Apple TV for dirt cheap and had a decent head start.

Their plans for Xbox One are interesting with all of the TV features and HDMI pass through but the Xbox One is seriously expensive so it still only really appeals to pretty committed console gamers.

With Ballmer leaving in the next year I hope Microsoft bring in a decent tech-oriented CEO although I have a feeling they will go for another business guy who will do similar to what Ballmer did. Not to say Ballmer did a bad job. Sure he had some duds but he wasn't an awful CEO. I am going pretty left field here but I would love to see J Allard come back but I don't know if he is really MS CEO material. He certainly has a good image though and is respected by most of the tech community and I could see him being a Steve Jobs type figure head for MS if they played it right.

Then again I am just some random guy on the internet who hacks code for fun so I am probably totally wrong about it all but you have to agree seeing Allard back would be pretty interesting :)

drill_sarge• 13 years ago

Technet, Windows Update, Xbox Live, Hotmail, Bing, Skydrive and so on.

InTheSwiss3• 13 years ago

I often wonder what the bandwidth for Windows Update (and MS Download Center for OS patches) is. I would love to see some details on it if it has ever been discussed by MS engineers?

wikwocket2• 13 years ago

Don't forget Sharepoint services for large companies too.

Skydrive and Hotmail are some of the services much larger than people realize. Hotmail gives you very large inboxes, like Gmail, and Skydrive backs cloud storage and backup for most Windows devices.

curiousDog• 13 years ago

There's also Dynamics CRM online (Salesforce competitor), Sql DB as a service and some windows phone services

jonknee2• 13 years ago

I wonder if they included their own software license cost... One division billing another division.

1SockChuck2• 13 years ago

Microsoft has been incentivizing its internal divisions to shift from legacy data center space to Azure. So some of that spending is for new servers and data center space to in-source services hosted by third-party providers. Or, in some cases, older Microsoft data centers. More here: http://www.datacenterknowledge.com/archives/2011/06/01/at-mi...

jcampbell1• 13 years ago

Does any one know the relative costs of hardware vs building and cooling? If I had a cool billion to spend, how much would I spend on building/cooling/power/generators vs servers and networking hardware?

ihsw• 13 years ago

Power failover and HVAC systems are on-going costs that usually exceed server hardware costs (both initial and replacement) by a long margin. Over a three-year period you'll probably get to spend 1% on actual server/networking hardware.

I'm just pulling that number out of fat air, but you'll be spending far more on other things.

lsc2• 13 years ago

Power failover and HVAC systems are on-going costs that usually exceed server hardware costs (both initial and replacement) by a long margin. Over a three-year period you'll probably get to spend 1% on actual server/networking hardware.

This is true... if you are running on 10 year old servers, and brand-new power/hvac equipment. On a more reasonable replacement schedule, if you pay what I pay for power/cooling/rack and what I pay for hardware, you are closer to half and half.

Now, I pay a lot more than google does for both power and harware- but I suspect that the google discount on datacenter space is vastly greater than the google discount on hardware. If nothing else, california power is like 2x power anywhere else. Also, i'm strictly retail when it comes to datacenter space. I rent 23Kw usable from Coresite... meaning I'm renting like 28Kw capacity... so I'm not a single U customer, but I'm still firmly 'retail'

You can alter this quite a bit just by going from X series to L series CPUs or by replacing your hardware more or less often. (I'm on L and E series xeons.)

But, anyhow, from that? I'd think that google probably spends more on hardware than on power/cooling/etc...

incision• 13 years ago

Have a look at this [0][1], Chapter 6 should give you a some idea.

0: http://research.google.com/pubs/pub35290.html

1: http://www.morganclaypool.com/doi/pdfplus/10.2200/S00193ED1V... [PDF]

ChuckMcM2• 13 years ago

Yes, and they aren't particularly relative :-)

But to answer your question you can get pretty close by translating into units of watts and remembering a couple of simple rules.

Rule one is you only use 80% of the provisioned power (this is a safety rule with respect to things like circuit breakers and infrastructure, it keeps random surge events from destroying everything).

Rule two is that your power use efficiency (PUE) ratio determines how much cooling you need.

So lets say your PUE was 1.3 (that means total power / computer power = 1.3, the extra going into cooling) and you were building a 10 Megawatt data center. Well you have 8 Megawatts of actual power (80%) and 8/1.3 or 6.15 Megawatts for machines. Going by 750 watts per server (the PSU label [1]) you can put 6.15/.00075 = 8,200 servers into that data center. At 500 watts/machine you are 12,300 machines. So to make the math easy lets say it is 10,000 machines.

If you servers cost you $5,000 each then you're at $50M in server hardware for a 10MW install. Network switches are $100/port (assuming you're not doing 10G ports) and cable is way more expensive than it should be adding another $20 per port to connect it to the switch, so another $1.2M there. You need to hold them in something and rack solutions go from the less expensive (4 post) to the more expensive (fully enclosed) with a range of $500 - $2500. How many you can put in a rack will change your per machine rack cost up or down, but its somewhere between $25 - $125 per machine so another $1.2M so lets be generous and say our $55M for your 10,000 machines which are going to consume 8MW of your 10MW provisioned power.

Building the data center though, especially if you include land acquisition, that is going to be pricey (of course you can write it off over 27.5 years but still) If we start with the fact that a data center is really just a big warehouse with a cold box inside, we can look at building a warehouse [2] and some folks have some priced out [3] which put it about $50 - $75M range, and that is before you add cooling, back up generators, and pay the local utility to run a 10MW power line into your building. I got some estimates from Crane for a 'pocket' data center project I was looking at (about 3MW) and those extra costs are probably going to exceed $100M (split 33/33/33 into cooling, back up power, and secondary construction).

So back of the envelope 3:1 construction costs over equipment cost. And maybe $20M per constructed MW of compute.

[1] This isn't what you would do since machines don't consume that much, they can only not consume more than that, but whatever.

[2] http://www.reedconstructiondata.com/rsmeans/models/warehouse...

[3] http://www.mohavedevelopment.org/media/docs/One%20Sheets/Moh...

JabavuAdams• 13 years ago

Thanks for the informative post. Are you aware of any non-proprietary tools for doing procedural generation of data-centers?

I'm imagining a generate-and-test cycle given estimates for the various component prices.

mrb• 13 years ago

Yes, see the blog post titled "Overall Data Center Costs", from James Hamilton (Vice President and Distinguished Engineer on the Amazon Web Services team):

http://perspectives.mvdirona.com/2010/09/18/OverallDataCente...

outside1234• 13 years ago

is that all? that's only $10k per machine if they have 2M machines (which I'd guess is a conservative estimate)

NOTE: I'm not being snippy - that seems very efficient given the building, HVAC, networking, and the fact that they have probably had a lot of machines fail.

ffrryuu• 13 years ago

And I have spent tens of thousands on rent.

pearjuice• 13 years ago

They do have 100% uptime, right?

socillion• 13 years ago

Google had a 5 minute outage this August, where global internet traffic dropped by 40%.

http://www.fastcompany.com/3015942/most-innovative-companies...

AznHisoka• 13 years ago

What if you were trying to start Google today (assuming it didn't exist), and you had to bootstrap with maybe only $500 available to spent per month? How would you guys do it?

kineticfocus• 13 years ago

Well this would be a place to start... http://commoncrawl.org. It would be interesting to see a solution to the data center requirement that uses some adhoc bit-torrent like system.

kintamanimatt• 13 years ago

I presume you're talking about hardware budget. You probably wouldn't get very far with such a low budget, but I think your question is how to get the most bang for your buck. The answer is with dedicated hardware, especially for computationally expensive apps. $500/month dedicated hardware will beat the pants off of $500/month of EC2 instances for example, but it comes with the cost of being less flexible.

hashtree• 13 years ago

Amortized over 3/5 years or non-amortized? Different answers :)

wmf• 13 years ago

Read up on the history of DuckDuckGo; AFAIK it was one person in the beginning.

nly• 13 years ago

Hold out with a handful of servers until Google bought you.

alanlewis• 13 years ago

Headline is wrong. 2 data points that the article is based on: 1) Google has spent 21 billion dollars on Capital Expenditures since 2006 (presumably gathered from SEC filings) and 2) "Google says the majority of its capital investments are for IT infrastructure, including data enters, servers, and networking equipment." (source unclear)

So, Google has spent at least 10.5 billion on "data centers" since 2006. There are notable disclosed expenditures that are clearly not "data centers" mentioned in the article, including 2 billion spent to purchase NYC office space. Still a huge number, but off by many, many billions.

swashboon• 13 years ago

Didn't they spend 2 billion buying the 111 Eighth Avenue office in Manhatten at the end of 2010? I believe that is data center realestate.

rads• 13 years ago

Why would they put a data center in Manhattan? Don't these things usually go where the space is relatively cheap?

1SockChuck2• 13 years ago

111 8th Avenue is one of the main data center hubs in New York, but Google is interested in it primarily for office space rather than connectivity.

DannyBee• 13 years ago

It's certainly used as a real google office, it has thousands of googlers in it.

mathattack2• 13 years ago

There's a very steep depreciation on this capital spending too. Every year technology gets cheaper, and prior data center CapEx loses value.

scott_karana• 13 years ago

Well, building a secure, well-ventilated, extensible, vault with reliable power sources, internet backbones, etc, is mainly an infrastructure cost that won't change year-to-year.

But servers are definitely a loss, that's for sure.

Google Has Spent $21 Billion on Data Centers

Related Stories