Has Amazon EC2 become over subscribed? (2010)

http://alan.blog-city.com/has_amazon_ec2_become_over_subscribed.htm

by vishal0123 • 13 years ago

141 72 13 years ago

niggler2• 13 years ago

Is there any scale (like xlarge) where you are effectively the only VM on a particular physical machine (thereby obviating the issue with the small instances)?

DrStalker• 13 years ago

You can pay extra for dedicated Instances: http://aws.amazon.com/dedicated-instances/

> Dedicated Instances are Amazon EC2 instances launched within your Amazon Virtual Private Cloud (Amazon VPC) that run hardware dedicated to a single customer.

eurleif3• 13 years ago

$10/hour per region? Wow.

ceejayoz• 13 years ago

You can pay a (hefty) fee for dedicated tenancy, where your EC2 instances only share servers with other instances in your account.

The largest instances (the quad- and octuple-extra-large) likely are on their own servers but I've never seen that explicitly confirmed anywhere.

Xorlev2• 13 years ago

c1.xlarges are thought to be effectively the only VM on a machine. I can't dig up the article that proposed the methodology for determining offhand.

c1.xlarges still underperform far under what I'd expect.

sliverstorm• 13 years ago

Are VMs ever really performant?

staunch3• 13 years ago

...and here's part of why I created Uptano.

https://uptano.com

The flexibility of usage-based billing and instant provisioning is awesome, but it's really not worth giving up dedicated performance IMHO.

jamesaguilar4• 13 years ago

I upvoted because I love HN comments that offer solutions. That said . . . how can you charge half as much as Amazon for the same service and make a profit. I assume their margins are not that fat, so what are you cutting that they offer? Or else what secret have you discovered that no one knows? I guess since this is your business there's a chance that you won't answer, but I'm curious.

coolsunglasses• 13 years ago

I've been looking for this for...ages...

Thank you...so much for posting this.

eurleif3• 13 years ago

This still uses virtualization though, correct? So you don't get full dedicated performance?

oellegaard• 13 years ago

That service looks absolutely awesome. Just signed up and had a look - but I miss a major thing: API.

LogicX• 13 years ago

Somewhere between uptano and EC2 is internap (formerly Voxel.net) agileservers: http://www.internap.com/agile/

trotsky2• 13 years ago

I was trying to figure out how this guy could be so behind the times - just discovered internal network latency at AWS?

Then I saw:

Published: 9:01 AM GMT, Tuesday, 12 January 2010

ultimoo2• 13 years ago

Yes, the year of the blog should definitely be present in the HN title.

That said, what is the deal with the internal network latency and the other points the OP touched upon today? Did Amazon get around to resolving things, or have things worsened?

seldo• 13 years ago

My non-rigorous impression is that latency is roughly the same now as it was in 2010, i.e. not great, but at least it hasn't been getting worse.

t0• 13 years ago

Can you enlighten us? Why is it so slow? Are some people abusing all of that free bandwidth? Is their LAN simply not built to handle much traffic?

trotsky2• 13 years ago

I don't have any inside information, but I would guess it's because their san is so sketchy everyone rolls their own DFS and uses other chatty replication strategies, leaving the network running data and disk on the same topology. These days a lot of that traffic will take 4 hops within an AZ.

redact207• 13 years ago

Have there been any improvements since then?

codewright• 13 years ago

No, it's still common to have in-house "find a good node" code that fires up and takes down machines until one that will be reliable is found.

wmf• 13 years ago

Previous HN discussion: https://news.ycombinator.com/item?id=1048694

rabbidruster4• 13 years ago

While I understand computer scientists are not always the best writers, phrases like "Amazon do have a breaking point" make it hard to continue reading this article.

mattdeboard• 13 years ago

Might have been using it like "The cattle do graze", or "Black Sabbath do have a new album coming out".

philwelch• 13 years ago

I'm curious what you find wrong with that sentence.

icedchai• 13 years ago

We're from America. We speak Murican.

spicyj• 13 years ago

If it's the plural conjugation you're complaining about, know that that's correct in British English: http://english.stackexchange.com/a/1339/50.

rabbidruster4• 13 years ago

Thanks for the link. It sounded extremely weird to me. I didn't know corp were often plural nouns in British English.

pcl• 13 years ago

In British English, a company is plural, not singular. So this construct is correct in some dialects. And he also uses 'armour', which is again a British construct, so I'd guess he's just not using American English.

(I wonder if British law considers a corporation to be a person to the degree that US law does, or if this plural view of a corporation is pervasive in law as well as in grammar.)

mpclark3• 13 years ago

This is oft-repeated, but not actually true. Most people in the UK who care about these things (for instance, sub editors in the printed press) hold that companies are singular entities. The confusion comes with sports teams, which are commonly referred to in the plural, so "Microsoft is" but "Manchester United are".

Edit: For example, here's what the Guardian's style guide says on the matter:

http://www.guardian.co.uk/styleguide/c#id-3022716

Of course, with something as flexible and constantly evolving (and as used and abused) as the English language, it is usually possible to find examples and counter-examples for just about anything. There are also edge cases; the Guardian refers to police forces as plural entities, I believe. Suffice to say, when I was running a back bench, singular was the order of the day when it came to company names.

pinneycolton• 13 years ago

It's common in the UK to use "do" when referring to a collective. The author is from the UK.

mynameishere• 13 years ago

Come on. Quit downmodding people for a common misunderstanding. To any American "Amazon do" really does sound off.

rattray• 13 years ago

Is there a cloud provider that doesn't (yet) have these issues?

Is it possible to run any cloud service at Amazon's scale without these issues?

(genuine questions speaking from a point of ignorance)

rdl2• 13 years ago

Some of the OpenStack providers run their own storage networks using conventional SAN tech. Super expensive but more consistently performant.

dmpk2k• 13 years ago

My experience with SANs is that they are anything but consistent. Local storage is a better idea: fewer moving pieces to go wrong, fewer moving pieces to understand and debug, fewer possible sources of contention, and the latency is low.

SANs in a cloud environment optimize for the wrong thing. Servers by and large have a high uptime -- since their falling over is comparatively rare, this is simply a problem I've never had difficulty with. What I have had in spades, before I learned better, were database problems due to wild fluctuations in latency to the SAN.

It doesn't help that when SANs kick the bucket, they tend to affect a lot of things.

stonith• 13 years ago

There are some using Ceph for the volume service who shouldn't be terribly expensive. Dreamhost for example.

joefarish• 13 years ago

Which providers are they? Do you have any experience of using them?

JoshGlazebrook• 13 years ago

I've had these issues with the micro instances before. One deployment will be so sluggish it's almost unusable, but starting up another one results in one that is just fine (micro wise).

rbc• 13 years ago

One think that I think may be oversubscribed at EC2 is the API layer for controlling things like instances, ELB's and autoscaling. This seems to be most obvious in Virginia. During the lightning storm in Virginia last summer, it seemed like API access fell off a cliff. I'm guessing that was because everyone was trying to move their services from the impacted availability zone.

dotBen3• 13 years ago

I'm loathed to perpetuate a 3 year old article, but...

One of the key contributing factors to this kind of network degradation of AWS (/or other cloud vendor) is the abundance of the "bad neighbor test" - where a client performs tests to see if they can achieve a 'preferred' amount of CPU/IO on the host of their new instance.

Resource sharing rules at the host level actually means that if everyone is trying to max out their instance, you would still get the equal share you are entitled to and guaranteed with your instance, so what the bad neighbor test really means is whether you can actually go into your neighbor's CPU allocation due to their under-use.

Well, if everyone does that then the system degrades as someone has to a party using less than their allocation and the amount of instance spots that don't fail the 'bad neighbor test' become non existant.

The overall health of the entire network would actually be better if folks didn't do this practice and instead everyone simply evened out their use across their instances that enjoy additional resources and stuck it out with the %age of their instances that only achieved their guaranteed minimum resource use and no more.

My company uses another "cloud-like" vendor and although we don't perform 'bad neighbor' tests upon new instances, it is fair to say our application benefits from the fact that the majority of the instances on their network are under-utilized and we can push into the max CPU of the host beyond the limits we pay for. Where instances do share 'bad neighbors' (ie we can only get what we paid for and no more - boo hoo, etc) we still keep the instance but simply route around that and distribute less load than on other nodes in our network.

That doesn't become the most cost effective mechanism, but the "savings" of the 'bad neighbor test' are probably negligible and ironically by not doing this we become the "good neighbors".

jamesaguilar4• 13 years ago

Actually, if the system is well-isolated and correctly subscribed, at worst you'll get exactly your reservation. It is not necessarily the case that someone has to lose.

dotBen3• 13 years ago

Right. The problem is that a 'bad neighbor' test is often defined as when one can only get that guaranteed minimum amount of resource.

That's the part that doesn't scale.

ultimoo2• 13 years ago

Since you are well-versed with this issue -- I know that Amazon offers tenancy options while creating virtual machines. Is this option not utilized because the price of single-tenancy is higher than the headache of "bad neighbor test"?

dotBen3• 13 years ago

So as I mentioned we don't use EC2, we use another cloud service, but the economics and technical issues are the same.

But on EC2 Dedicate Instance (ie single tenancy) my guess is that if your application (or, business model) relies on each of your nodes being able to utilize more than your equal share on the host then in fact you would NOT want more than one of your instances to exist on the same physical host, in order to maximize the chances that each instance can grab all of the resources on it's given host.

If this is your model, having all your instances on the same physical host would be disastrous. In fact, there's (economic) argument for Amazon offering customers the complete opposite - pay to guarantee that no two instances are ever instantiated on the same physical host.

Has Amazon EC2 become over subscribed? (2010)

Related Stories