Amazon’s Scorpion Problem

http://www.feld.com/archives/2014/07/amazons-scorpion-problem.html

by rmason • 12 years ago

63 59 12 years ago

feld.com

ForHackernews• 12 years ago

"While we are in the middle of a massive secular shift from owned data centers to outsourced data centers"

A secular shift? As opposed to a religious one?

adorton• 12 years ago

http://en.wikipedia.org/wiki/Secular_variation

gojomo• 12 years ago

Like many words, secular has other senses. Per a representative definition like…

http://www.merriam-webster.com/dictionary/secular

…Feld is using the sense (3), and especially (3)(c), a quite common usage in economics (as a contrast with transient/cyclical changes).

You might benefit from a more catholic appreciation of vocabulary. (And no, that has nothing to do with the Catholic church.)

ghaff• 12 years ago

Secular also has a meaning of once in a long time or once in an age. One source I looked at says that the Latin root in also the root for "century" in Romance languages. I could argue it doesn't add much to that sentence (which I don't really agree with in any case) but it's not an incorrect use of the word.

ableal• 12 years ago

http://en.wikipedia.org/wiki/In_saecula_saeculorum : "The saeculum in Roman antiquity was the potential maximal human lifespan, or roughly a century"

liveoneggs5• 12 years ago

where do people get these numbers? You should move to physical hardware well before $200k/month.

opendais6• 12 years ago

They probably assume you have to pay for 24/7 DC tech / sysadmin types to do the racking, setup, etc. that AWS automates and are factoring it into the price.

To be fair, that is part of the cost. However, unless you are really big, you will just pay remote hands which is like $100-200/hr. The number of incidents are probably way too few, even at 200k a month, for 24/7 on-site sysadmins/techs.

__john• 12 years ago

Do you have a source or some insight you could share as to when a company should move away from AWS? A quick google search turns up $1000 per square foot[1] for a buidling a datacenter. There's also a nice discussion on reddit[2], but there's no talk of cost. I think this is an interesting economics? problem and would like to hear more on it if anyone has the time.

[1]http://www.webhostingtalk.com/showthread.php?t=920078 [2]http://redd.it/16m9pq

liveoneggs5• 12 years ago

I can tell you that you can rent an entire rack with power for $500-$1000/month (42U);

A decent internet connection from level3 or the colo house internet for another few hundred.

So even at $2k/month we've still got $180k/month to buy servers and storage (or $2M for the year).

$200k/month is a lot of money.

100k• 12 years ago

Yeah. You can get managed hardware (for example at Soft Layer) for a much better price/performance ratio than EC2. The upfront costs are higher but the price for equivalent hardware is way cheaper.

I think EC2 makes total sense for variable load (I paid like $0.25 to run some Coursera homework on a huge computer for 15 minutes and that was amazing) but if you're running something that needs to be up constantly and reliably (e.g., a database) EC2 gets expensive pretty fast.

hamburglar• 12 years ago

I'm not the person that did this analysis, but if I recall, for file storage, somewhere just shy of $30k/mo was the magical point at which it made financial sense to move from S3 to a few stout SoftLayer storage machines at my company.

iancarroll• 12 years ago

I call bullshit on the support and the "knockoff" claim. I've always felt they are the best with support - they've even waived an accidental $120 instance for me. Yhe tools they provide me are exemplary and cannot be replaced IMO.

toddmatthews• 12 years ago

"AWS is not the best product at anything – most of their features are mediocre knock offs of other products."

Does not list a single example.

dasil003• 12 years ago

Elastic Transcoder is a good example. They don't even support variable bitrate, so you have to choose between huge files and video that looks good for a while until you hit some dark scenes or high contrast sequences that exhibit terrible artifacting.

danielweber• 12 years ago

What does this have to do with "scorpion"?

From the title, I thought it would be about Amazon competing with its customers, like Netflix using AWS to compete with Amazon Prime, and how its inevitable that Amazon will eventually have to sting Netflix.

ojbyrne• 12 years ago

From the article: "In addition, they always end up competing on every front possible, hence the chatter about Dropbox moving away from AWS since AWS has now come out with a competitive product. It appears that it’s just not in Amazon’s nature to collaborate with others."

wdewind• 12 years ago

I also was confused but I think he was referring to the constant (alleged) behavior of Amazon towards his portfolio companies in which they act like partners/potential acquirers and then just build the tech themselves.

x0x02• 12 years ago

amazon now is actively parasitic to marketplace vendors: they find the best selling products and compete directly. At some point this has to impact willingness for small shop owners to essentially show their books to amazon, right?

opendais6• 12 years ago

Yes, but that is a MWS issue not an AWS issue.

femto1132• 12 years ago

Anyone who makes the blanket assertion that "AWS is not the best product at anything" is failing to grasp the fundamental advantage of AWS--it is the only (and therefore best and cheapest) "full line" IAAS provider. Obviously simple value added services like Dropbox that primarily leverage just one piece of the cloud (storage) will eventually move to wherever (and everywhere) that they can get their infrastructure cheapest. When complex services like Netflix begin moving off of AWS is when I'd start worrying.

opendais6• 12 years ago

Eh? What do you consider "full line"?

I'm pretty sure Netflix uses it due to the highly variable usage situation. It never makes sense to have a server sitting idle for 23 hours a day.

opendais6• 12 years ago

I'm amused by the downvotes combined with the unwillingness to define "full line".

femto1132• 12 years ago

I think "full line" isn't an entirely static concept in IAAS, but I'm using it here to suggest there's no other player offering the same breadth of services. I haven't surveyed other providers in detail lately, but I expect for any given competitor it wouldn't take you long to find some AWS service (http://aws.amazon.com/products/) that they simply don't have an equivalent for.

EDIT: found someone who's made a feature grid here: http://www.planforcloud.com/pages/resources/cloud_services.h...

This doesn't even include more esoteric stuff like Redshift and Elastic Beanstalk.

bellerocky• 12 years ago

The other thing that AWS has going for it is widely adopted third party support. Pretty much any software, site, tool or service that has any sort of cloud monitoring, deployment etc feature will be integrated with AWS. This not true for any of AWS's competitors. I've never seen a third-party provide support for an AWS competitor before supporting AWS. I know monitoring tools that do not support Azure, Google Cloud or Rackspace. I know libraries that don't support all of them but always support AWS.

In addition most sysadmins and developers will be more likely familiar with AWS's tools, API and dashboard. They'll be more likely to already have their development environment set up with AWS command lines tools. You ever heard of someone with HP cloud tools installed [1]? I frequently browse dotfiles on GitHub and often see non-sensitive AWS credentials and aliases. I've never seen a Rackspace or Google Cloud thing in them.

AWS wins on mindshare. It's also a proven system. Infrasture decisions are really important, and downtime is very bad and when you choose AWS you know at the very least that if your shit goes down because of provider issues everyone else will be down too and you can kind of get a pass (although outsourcing downtime is kind of shitty), but if you're not on AWS and your shit goes down because of the provider people will ask you why aren't you using AWS. Your customers and investors might actually be unhappy with you because of this.

As for support Google also has terrible support, Rackspace's managed support is very expensive and actually probably not what a competent team would want, otherwise it is easy to get a person on the phone at Rackspace, so that is commendable. Support doesn't scale, but if you are paying AWS a lot I bet you can get support.

I don't really like the mono-culture of AWS everywhere, but I feel like the points above need to be raised. A standard on APIs like OpenStack would be nice, but implementations will differ and AWS will fight this kind of change and because of their dominance getting traction will be hard.

[1] this is actually a thing: https://docs.hpcloud.com/cli

andlarry• 12 years ago

AWS is unbelievably lousy at support.

Compared to whom? Google? How many times have you seen tickets like [1] for Android, Gapps, etc? Additionally, the google cloud platform onboarding was so bad, I almost gave up a $500 credit in frustration.

[1] https://productforums.google.com/forum/#!topic/gmail/HTX9cLb...

smacktoward• 12 years ago

Rackspace leaps to mind.

lwhalen2• 12 years ago

Rackspace's support is indeed fantastic, however their product is significantly more friction-ful to use (I'm currently at a company that has purchased a Rackspace Private Cloud, and uses it in conjunction with their public cloud). It works as advertised, kinda-sorta and with a whole lot of spit, duct-tape, and prayer, but I'd much rather use Amazon's system and deal with one support ticket every other quarter, than the weekly support ticket(s) using Rackspace's product seems to require.

mark_l_watson2• 12 years ago

I also received the free $500 credit and decided not to use it because of an on boarding issue.

On the other hand, when Amazon first released EC2, they gave me a $1000 credit to use anyway I wanted, and that I did use.

I really like Google products like GMail, Blogger, and G+ and working at Google the infrastructure was more than wonderful, but for external developers the Google Cloud Platform products really could be a lot better.

blowski3• 12 years ago

I have no idea what the OP is talking about. As far as I can understand, the author is saying Amazon (yes, Amazon) is not hard-nosed enough to win a fight with Google and Microsoft.

#1 - "AWS is not the low price provider", "Being low priced is in Amazon’s nature so this will be intensely challenging to them". This is true, but as the market-leader, do they need to be?

#2 - "most of their features are mediocre knock offs of other products". As @ToddMathews said here (https://news.ycombinator.com/item?id=8104364), he doesn't list any examples of these superior cloud services. I can't think of any. Sure, ELB is a knock off of HAProxy, but in that sense, Gmail is a knockoff of Postfix.

#3 - "AWS is unbelievably lousy at support", "Support is just not in Amazon’s nature". Really? Compared to whom? In my experience, they have always been one of the best providers of support of any online company.

#4 - "Once you are at $200k / month of spend, it’s cheaper and much more effective to build your own infrastructure". And yet many businesses spend more than $200k PCM. Are they stupid? Or is the author wrong?

Semaphor3• 12 years ago

Especially #3. Amazon is willing to let themselves be scammed before denying free replacement products to customers.

opendais6• 12 years ago

That isn't true. I've had to issue chargebacks when Amazon's vendor and Amazon couldn't come to an understanding and I never received the software product w/o a revoked key.

x0x02• 12 years ago

I've had very good customer service purchasing physical items from amazon.

ec2 and associated services' customer service and support has been pretty shitty, and these are from companies paying anywhere from $8k to $96k/mo

d23• 12 years ago

#2 always was what impressed me most about AWS. When I was trying out Digital Ocean, I had to hack together my own scripts to set up the equivalent of cloudfront, auto-scaling, and elastic load balancing. I realized just how important those seemingly small infrastructure tools are when they "just work".

blowski3• 12 years ago

Exactly. The value of AWS is not in the individual pieces, but in how all the pieces fit together, and how a novice is able to do it.

seanflyon• 12 years ago

Regarding #4, do you know how common it is to spend over $200k per month. If it is a rare use case I would not be surprised if those companies are making a sub-optimal decision. In my anecdotal experience, whenever we considered that order of magnitude of spend on AWS we ended up building our own infrastructure.

mark_l_watson2• 12 years ago

That was a strange article.

Isn't it a self obvious truth that AWS is most appropriate when you need lots of computational resources in bursts?

If you often need many extra servers for a short period of time, then AWS is really good for that. Another thing that AWS provides is some very handy infrastructure APIs like S3, DynamoDB, Eleastic Mapreduce, etc.

For flat usage scenarious, then renting beefy VPSs or physical servers is more cost effective.

callesgg4• 12 years ago

$200k that is insane, it is cheaper to go width own hardware long before that.

The server room we have at our office have had max 5 hours of total down time since 2011 And we have moved our entire office under that period. I will try to estimate our general costs in EUR.

Inital costs for servers, switches, cooling, Racks UPS 150K

Replacement of servers every 3-4 years 60K/3 = 20K

Internet line = 10K a year

Electricity = 2K a year

Consulting cost for hardware maintenance = 10K

Old servers we put those in an a room a bit away and use them for backups/failsafe.

That is about 50K a year.

Counting in initial costs and just cause is is till almost nothing compared to 200K a month, I would say that switching to self hosting is financially beneficial when getting to about 10K a month with 3party hosting.

imjk• 12 years ago

Where are your costs for labor. Is that all included in your consulting cost? If we as an advertising startup decided to migrate all that internally, I'm sure our costs would go up not only in manpower but also from the distraction of dealing with that rather than focusing on our product.

callesgg4• 12 years ago

The initial cost was with labor included. The labor is the consulting cost.

But to be honest there is not like one has to do much width the hardware.

When one hardware server fails all the virtual machines will still just continue on another hardware server. Vmware takes care of that stuff and it just works, at least in our testing of pulling the cords to the server (stuff has not failed yeat).

Well if money is no object, you are definitely doing the right thing.

eco• 12 years ago

Without numbers showing the actual quantities involved this cost comparison is kind of useless. Not that I doubt going with your own hardware is cheaper at those levels, it's just not a conclusion I can draw from what you've written. Also, as others said, labor costs need to be factored in.

How much would a comparable setup cost on AWS per year? Once we know that we can draw conclusions.

callesgg4• 12 years ago

We did calculations on that about 2 years ago, then it was about 700K pear year.

But that left out some cost like Shared network folders that our users use, so we would have needed to get some kind of replacement for our netapp(a form as NAS).

mark-r• 12 years ago

Does Google also have the "lousy at support" problem? Their DNA has always favored algorithmic approaches that scale vs. human contact which doesn't. I remember hearing horror stories when they first came on the scene.

callesgg4• 12 years ago

I actually find that google has quite good support (where they have support) Google Apps for business for example.

orandolabs• 12 years ago

We (Orando Labs, https://orandolabs.com) feel the sting of the scorpion. Check out EnduroSync vs Cognito at http://orandolabs.wordpress.com/2014/07/12/amazon-cognito/

guiambros• 12 years ago

I respect Brad and usually enjoy his posts, but this one is full of s...

He doesn't give a single example of why "AWS is not the best product at anything". This is obviously true, but the same could be said about Google, Microsoft, or Rackspace. So what's the point? Who has a better offering?

The discussion of AWS being more expensive than bare metal is valid, but that's an obvious statement. Amazon uses bare metal (albeit at scale), and adds their application stack + support. You're paying for the convenience of purchasing all services integrated from a single vendor. If your business is simple enough and you can just throw hardware and bandwidth to solve your needs, then bare metal will always be cheaper. But for more complex environments (e.g., multi-regions, global distribution, auto-scaling), or when you need to quickly scale up and down (to accomodate surge in demand, for example) bare metal is not an option.

Net net, this sounded like a rant against Amazon's world domination plans, and the threat to the very existence of some of the companies on his fund. He said it best:

"We used to think of Amazon as a potential acquirer for these companies, or at least a powerful strategic partner. Now we know they are just using the bait of “we want to work more closely with you” as market and product intelligence."

If your business model depends on a strategic partnerships with or being acquired by Amazon, you're doing it wrong.

Amazon’s Scorpion Problem

Related Stories