Amazon bares its computers

http://bits.blogs.nytimes.com/2013/11/15/amazon-bares-its-computers/?_r=0

by DLay • 13 years ago

124 71 13 years ago

bits.blogs.nytimes.com

venus• 13 years ago

1.5 million requests a second, and holds trillions of “objects”

My god. I used to think I was a bit of a rock star, with (rails) web sites exceeding 100 requests/second on my resume. 1.5 million requests per second ... I cannot even really imagine that.

codex• 13 years ago

These get requests are likely sharded among a large number of machines. Viewed that way, it's much more manageable. Want to service more requests? Just add more machines.

1.5M relational queries on the other hand--that would be impressive.

StavrosK• 13 years ago

Over 15,000 machines, it's roughly the same thing!

lukeschlather• 13 years ago

It all depends on how many servers you have, as well as what sort of databases/database complexity.

For a laughably trivial site, a single host could do a lot of requests per second without too much effort. For a normal Rails app running on a handful of hosts, 100 requests per second is nothing to scoff at, especially if you're on your own.

ddeck3• 13 years ago

These are spread over an enormous number of machines.

OT, but for truly impressive single machine performance, check out the LMAX FX trading platform, which can handle 6 million orders per second on a single core on commodity hardware.....in Java. [1]

[1] http://martinfowler.com/articles/lmax.html

jsnk• 13 years ago

“All of our engineers are focused” on reducing the costs of computing

If this is so, how is it that a startup like Digital Ocean that's only 2 years old find a way to provide $5/month plan that's running on SSD while m1.small on EC2 which is slower than DO costing me around $40/month? At this rate, I should probably buy a server myself and run it at home.

I like AWS and its ecosystem, but I am tired of Amazon continuing to provide shitty service for a ridiculous cost.

walshemj• 13 years ago

MM yes I was at silicon milk about today ad everyone was on AWS - which is slightly worying.

I think that a lot of the start ups around silicon roundabout woudl be a bit lost if they had to pivot and run their own kit.

the CTO says the VC says we have to switch to self hosting we have 4 racks of dual core 1u 's arriving - anyone know how to put them together :-)

dmourati3• 13 years ago

Just find a sysadmin with > 5 years experience, ie, pre-cloud. We all know how it's done.

beachstartup• 13 years ago

right, amazon's engineers are focused on reducing the costs ... for amazon.

gtaylor8• 13 years ago

They cut prices for customers many times a year. Seems like we get 2-3 EC2 price cuts every year, and S3 continues to get cheaper.

They are one of the few vendors we use that regularly passes on savings to us.

gaadd33• 13 years ago

Is it possible to spin up a few thousand of those on DO?

Sure most people use AWS as a glorified VPS provider, and for that it is rather expensive. The fact that DO is only 2 years old actually scares me more since they probably haven't faced as many unique challenges as a company who has been in the business for a longer period of time has.

gtaylor8• 13 years ago

This is something that I feel so many people don't understand about EC2 (ELASTIC Cloud Compute). If you are using it like a traditional VPS provider, you are likely over-paying for something you could get cheaper elsewhere. If you are just looking at a price comparison to another VPS provider or even the private cloud, you are doing it wrong. One must consider how much (or little) ability they need to scale/up AND down at very short notice.

EC2 is a boon for those that need to add and remove capacity quickly. This is the appeal of EC2. It's not going to blow your socks off as far as performance per dollar. It is going to blow you away with its flexibility. If you don't need to scale up AND down quickly and regularly, EC2 is probably not a good fit for you. With EC2, you are paying for that flexibility.

Netflix is one of the penultimate examples of "doing EC2 correctly". Their traffic is extremely bursty. They are constantly automatically scaling up/down, and they do this and so many other things without human intervention.

justincormack2• 13 years ago

Amazon is highly profitable compared to DO. They could cut their prices but why do it?

abalone• 13 years ago

Nope. Amazon is not profitable.

http://www.nytimes.com/2013/10/25/technology/amazons-revenue...

mattbillenstein4• 13 years ago

Cloud is great when you're in the discovery phase as it provides loads of flexibility for a small startup to try stuff out without investing in hardware, but at the point you're spending $50k/month in the cloud, racking servers and building a private cloud can give you better performance at 20-30% of the cloud cost... The main problem here is most startups don't have the expertise in house to make it happen.

Everyone thinks cloud is cheaper, but it's more about flexibility and avoiding capex, but at a certain scale, it makes sense to host it yourself - or pay someone to build it out for you.

I'm working in this cloud -> private-cloud migration space currently if anyone is interested in discussing.

gtaylor8• 13 years ago

This obviously depends on your specific scenario, but paying the maintenance/flexibility premium for something like AWS can be cheaper than something more rigid (see Netflix). I hesitate to make any generalizations like "building a private cloud can give you better performance at 20-30% of the cloud cost", because this is not true for all applications/businesses (it's certainly not for ours). For example, those with incredibly bursty traffic (Netflix, Amazon, other online retailers), can see some big wins with AWS. Not having lots of idle capacity that still has to be managed and paid for, not having to beef up the Ops/IT staff, not having to own your own/manage/maintain hardware, etc.

My advice to others is to be wary about generalized figures like "20-30% of cloud cost" from anyone with a vested interest in getting you to switch. Generalizations are dangerous when every usage case is different.

Also keep in mind that there is a lot of room for middle ground (even with AWS). For some (but not all), going hybrid can be the best of both worlds.

chaz• 13 years ago

I use DO and AWS for different projects because they meet different needs. Of course there are going to be places where one is better suited than the other in both performance and price.

For example, one of my projects requires 1TB of drive space. At DO, I have no choice but to go to the $960/mo 1 TB plan. But I don't need the SSD speed, 96 GB of RAM, 24 cores, or the 10 TB of monthly transfer. But with EC2, I can just use EBS @ $100/mo/TB and pair it with the right-size EC2 instance for the jobs at hand. As an aside, the real question for me is to decide whether I want to invest in a proper dedicated server where I can get good performance and online storage at a lower cost.

DO has done a fantastic job at finding a spot in the VPS market, which is why I'm one of their customers. It doesn't mean Amazon is doing it wrong.

erichocean• 13 years ago

I have a similar need for memcached, but I'm still using DO. I'm running the $5 instances running Twitter's fatcache. For $250/month, I can cheaply get 1TB of memcached storage online and available in our cluster (out of NY2).

Thrown in a some consistent hashing and I can scale up/down as needed. Fatcache is completely kick ass.

vidarh3• 13 years ago

But with EC2, I can just use EBS @ $100/mo/TB

If all you're after is the storage, that is very expensive unless you keep moving your storage tiers up and down very rapidly. E.g. Hetzner has servers at 49 euro (ca. 66 USD) per month for 2TB.

rottyguy2• 13 years ago

Not really meant to be a slam, but startling to me that even after all these years, Amazon still has a PE of 1330+.

3825• 13 years ago

I assume this means the market expects Amazon to make more money (a lot of money) in the future.

rottyguy2• 13 years ago

Yep, I get that. But they IPO'd in 1997. Again, not to detract, and in fact I'm amazed by how they've been able to compete both on the Brick & Mortar stage and Digital Services stage. I've just never seen a company given such love/leniency for so long by Wall St.

melling• 13 years ago

$17 billion in revenue last quarter. They're still not in the top 100 companies by revenue.

http://en.wikipedia.org/wiki/List_of_largest_companies_by_re...

Bezos is definitely going for growth.

kevinchen• 13 years ago

Google publishes some technical papers about its big computers, but does not talk about things like the semiconductors it has developed internally to handle computing at this scale.

Google designs its own ICs now?

jws• 13 years ago

I think I recall them making custom 10gbit Ethernet switch fabrics.

kijiki• 13 years ago

They used merchant silicon in their custom switches. Modern switch ASICs are 2+ billion transistor devices with lots of fancy custom logic for SRAMs and TCAMs and analog magic for PHYs. Even Google can't afford to do that by itself.

seiji2• 13 years ago

For over ten years.

They're also the world's largest computer manufacturer (for internal data center hardware).

Software people tend to think chips and hardware design are magic skills known by only three revered people in the world. It's just another track of school you could have chosen. ("Google writes its own software now?")

colmmacc• 13 years ago

Full-disclosure: I'm an engineer at Amazon Web Services.

"Mr. Hamilton later said privately that Amazon had developed original statistical methods to limit damage from catastrophic failures."

I don't think it's what James is referring to (as it's just the tip of the iceberg), but at Re:Invent we also released Infima - a library for service-level fault isolation and fault tolerance using Route 53 for endpoint discovery;

https://github.com/awslabs/route53-infima

The techniques in Infima give some insight into how we think about multi-tenancy and availability, as well as provide a framework for building other services on top of EC2, Route 53 and so on.

simonebrunozzi• 13 years ago

This one looks great! Any talk/video available that describe it in more details?

mwilcox• 13 years ago

"You've reached your limit of 10 free articles a month." ugh.

medell• 13 years ago

Open in a private browsing window.

natrius• 13 years ago

Or, you know, pay for the product. I don't understand the logic behind complaining about paying $3.75 a week for a product that you use so frequently. It should be embarrassing.

kraemate• 13 years ago

Anyone have the link to the original presentation? Particularly intrigued by the statistical fault tolerance.

dbarlett• 13 years ago

Not the whole presentation, but some slides and better notes: http://gigaom.com/2013/11/15/how-amazon-is-building-substati...

anan0s• 13 years ago

I'm a bit skeptical on the fact that such companies do not disclose their custom infrastructure design. So their value is both in hardware optimizations as well as software ones ?

being a grad student myself, I'm wondering if we'll ever see any specific technical aspects on infrastructures that could drive systems research and provide real problems to be attacked...

XorNot• 13 years ago

I'm skeptical that there's anything that innovative going on behind the veil.

I suspect if you had access you would look at a whole bunch of evolutionary improvements and probably trace them back to which papers you've seen it sort of mentioned in.

Probably not as much custom silicon as the article implies either, simply because on the scale of things Amazon is not a semiconductor company and Intel pretty unambiguously own the cutting edge there.

wmf• 13 years ago

Facebook reveals their infrastructure through the Open Compute Project since it can't compete against their core business.

b0b0b0b• 13 years ago

James Hamilton's blog is not to be missed:

http://perspectives.mvdirona.com/

locusm• 13 years ago

A Dell account exec told me last week that Dell had built and implemented the Sydney AWS data center - any truth to that statement?

nl• 13 years ago

They are in Equinix SY3 (Sydney)[1]. Pretty sure Dell doesn't do DC design for Equinix. I heard some stories about the truck loads of servers going in when they were rolling it out, but I don't remember the vendor.

[1] http://www.arnnet.com.au/article/442144/equinix_expands_its_...

11thEarlOfMar• 13 years ago

What is rather mind boggling is that Google, Amazon, Facebook, Microsoft, Yahoo all have multiple, huge data center installations and, they are all profitable companies. It does not seem to be a zero-sum game yet for market share, and the growth in the world's appetite for storage and access is still not fully served. Can they all wind up 'winning' with profitable operations founded on web infrastructure?

How much storage does one Internet citizen occupy, anyway?

adventured2• 13 years ago

"How much storage does one Internet citizen occupy, anyway?"

That number chases toward infinite. Its growth may slow down, as we reach the limit of how many 'actions' a person can reasonable take that generates content, but it'll never stop growing just on the basis of the unlimited desire for higher quality experiences (mixed with a general moore's law ride that makes it all financially reasonable).

And just when you think storage demand is going to slow down, we'll jump to some kind of virtual reality life recording, that makes it possible for others to walk around in your daily life as though they're actually there, and storage demand will explode another magnitude.

You can remove Facebook and Yahoo from that list though. Their operations are funded by their core businesses and they don't compete in the AWS-style cloud segment. That is, it doesn't make sense to discuss Facebook in regard to whether it can win or make money, compared to AWS or Azure.

So is there room for Google, AWS, Azure, Rackspace, IBM etc? Yep. It's a future hundred billion dollar plus business (AWS will probably be at $10b+ in a few years). No reason there can't be three or four big players in such a large market.

badmadrad• 13 years ago

The price will bottom out eventually and Amazon knows this. What we are really seeing is an infrastructure arms race and that is why AWS is costly. They are not building AWS for the individual user long term. They are building it for the enterprise. Eventually as bandwidth and computing power increases and cost decreases it may be feasible to run your personal blog out of your house or use a VPS on Digital Ocean. However, as we become a global community you will have to think about your application on that scale to be successful and that's where AWS can find longevity.

dschiptsov• 13 years ago

VmWare is a much greater waste.

eitally• 13 years ago

Interesting that they focus on all the infrastructure they design and build/commission themselves. They are also a big customer of my company (http://www.newisys.com/Products/products.shtml). Coincidentally, they call out Facebook at the bottom of the article, in context of the Open Compute Project. Facebook had airflow and overheating problems with their intended OCP design and our Newisys engineers helped with some design adjustments to resolve those, too. Now Facebook is buying cheap Taiwanese-made (Quanta & Compal) OCP-design servers and shipping them to us (http://sanmina.com/) for racking, systems test, and shipping to their DC in Lulea, Sweden (https://www.facebook.com/LuleaDataCenter).

So yes, these guys all do lots of tough engineering work, but it's not in isolation and not without assistance from industry veterans whose careers it has been to ensure properly functioning data centers and networks.

bluedino• 13 years ago

> Mr. Hamilton also said Amazon was not interested in participating in industry standards discussions, which could publicize its many technical advances.

What are the side effects of this in a few years, when all of the high-end stuff has been designed 4 different ways by 4 different companies?

What happens when Dell and HP's server businesses are shells of their former selves? When Cisco is but a memory as their biggest customers have began designing solutions in-house?

You'll have to go with the cloud. You just won't be able to get the best of the best equipment without doing so.

Amazon bares its computers

Related Stories