Urs Hölzle on Google's first data center

https://plus.google.com/+UrsH%C3%B6lzle/posts/UseinB6wvmh

by cramforce • 12 years ago

170 39 12 years ago

plus.google.com

This would be more aptly titled: "Urs Hölzle on Google's first data center"

Oculus4• 12 years ago

Hypothetical: If AWS was available at the time Google was just starting, would it have made sense for them to go with AWS?

timdorr• 12 years ago

Well, AWS explicitly doesn't charge for incoming bandwidth (vs. the "special" deal they got on the 15Mbps line item) which was the majority of their costs. It may have actually made sense, depending on their computing needs.

thrownaway2424• 12 years ago

Definitely not. The google product is not search or gmail or any of that stuff. The google product is dirt cheap computing. Everything else google does is just an attempt to exploit that competitive advantage. Running it on anyone else's stuff would erase the advantage.

Even in the collocation facilities google was developing this skill for example by installing their infamous corkboard servers with the drives attached by Velcro.

Oculus4• 12 years ago

I meant the question more in the sense of starting off with AWS and moving to their own servers like it is popular for most startups to do nowadays. Though you make a good point about gaining the skills for a robust infrastructure early on being important.

chubot2• 12 years ago

Unequivocally no, because it would have been too expensive, probably at least 100x the cost.

AWS is an order of magnitude more expensive than owning your own hardware, and Google was leveraging economies of scale to operate much cheaper than people that owned their own hardware (e.g. fault tolerant software vs. "enterprise grade" hardware)

People didn't realize it at the time, but "search" wasn't Google's only core competency. It was also "warehouse scale computing".

Of course Inktomi under Eric Brewer was another search company that realized that search is a perfect application for clusters of commodity x86 hardware, or networks of workstations (which AFAIK came from academic projects like http://now.cs.berkeley.edu/ )

(Don't outsource core functions: http://www.joelonsoftware.com/articles/fog0000000007.html )

Oculus4• 12 years ago

How early did Google start doing "warehouse scale computing"? I was always under the impression they started doing that as a byproduct of their large size.

banachtarski2• 12 years ago

No, the already small margins would have been consumed in a heartbeat.

Oculus4• 12 years ago

By small margins, do you mean at the time or currently?

damian20002• 12 years ago

They weren't profitable at all for several years (until AdWords) were they?

ChuckMcM• 12 years ago

No, for the same reason it still doesn't make sense to go with AWS. Basically you need a large number of cores that are intimately integrated amongst a large quantity of storage. It was the fact that all the machines could have a bunch of drives on them for "free" (low marginal cost). By interspersing the data and the compute you get maximum bandwidth to local assets.

wmf2• 12 years ago

Isn't that how EC2 works?

mwcampbell2• 12 years ago

What classes of applications benefit from that level of integration between processing and storage?

It strikes me that both AWS and various PaaS providers (especially Heroku) encourage strong separatioin between the application and its storage. Applications are supposed to treat all persistent storage as network-attached, while the VMs or containers running the appliication are ephemeral. I guess you're saying this is untenable for some kinds of application.

nugget• 12 years ago

I have heard that Microsoft's hardware cost to serve 1000 searches is > $1 while Google's equivalent cost is around $0.07. If true, this is definitely a competitive advantage.

jpatokal2• 12 years ago

Both figures seem unlikely, given that Google served ~2 trillion searches last year and did not spend $151 billion doing it. (The entire company spent on the order of $20 billion in 2013, and that's for everything.)

http://www.statisticbrain.com/google-searches/ https://investor.google.com/earnings/2013/Q4_google_earnings...

lpolovets• 12 years ago

The grandparent mentioned $.07 per 1000 searches, not per search. I think that brings the figure down to something pretty reasonable. ($151 million.)

btilly2• 12 years ago

Where did you get your $151 billion figure from?

2 trillion searches at 0.07 cents per thousand is 2 * 10^12 / 10^3 * 0.07 = $140,000,000.

IvyMike• 12 years ago

Estimates of "# of searches Google serves" are always fascinating to me. I don't see statisticbrain.com publishing a source for their estimate, although maybe I'm just missing something.

AFAIK, Google doesn't really publish a precise number. They often say things like "Over a billion searches a day". But then they keep that published estimate the same for years during which growth is obviously happening. At some point, they'll publish a step function upgrade to the estimate.

Of the top of my head 2 trillion searches sounds reasonable but I think there is a lot of fudge factor in that guess.

damian20002• 12 years ago

Here's another anecdote (also sourced from Urs Hölzle) about how Google transferred 9TB of data to their new east coast datacentre via fibre, for free.

http://www.dodgycoder.net/2013/02/googles-fiber-leeching-cap...

scottdrr• 12 years ago

really

jrockway• 12 years ago

The interesting quote here is:

You'll see a second line for bandwidth, that was a special deal for crawl bandwidth. Larry had convinced the sales person that they should give it to us for "cheap" because it's all incoming traffic, which didn't require any extra bandwidth for them because Exodus traffic was primarily outbound.

This shows that there's more to being a successful business than just good code; you need to know how to run a business, too. How many people working on web search ranking would ask for a special pricing exception from their bandwidth provider and get it? The answer is, apparently, one.

patrickg_zill2• 12 years ago

Actually it is a pretty typical arrangement, provided you can deliver on your commitment to a certain ratio of inbound/outbound.

Veus• 12 years ago

Maybe now it is but back in 1998?

vidarh2• 12 years ago

How many people working on web search ranking would ask for a special pricing exception from their bandwidth provider and get it? The answer is, apparently, one.

Anyone who knows how peering and bandwidth pricing works.

Basically, bandwidth prices amongst ISPs is a game of chicken and relative strength and perceived benefits; it's one of those areas where being good at talking and having sufficient traffic can make the difference between you paying consumer rates and the other party paying you to get you to host hardware with them in some circumstances.

(E.g. consider if you run a CDN - for smaller ISPs that pay high rates for bandwidth, it may save them a lot of money to get a pop in their datacentres to reduce their bandwidth usage; and in some circumstances it may even pay for them to pay to get a big bandwidth hog to serve traffic destined for certain other ISPs on their network, to "tip the scales" in peering negotiations)

Urs Hölzle on Google's first data center

Related Stories