Summary of June 29 AWS Service Event in US-East

http://aws.amazon.com/message/67457/

by rdl • 14 years ago

118 78 14 years ago

aws.amazon.com

rdl21• 14 years ago

Impressive that the failure of grid power (frequent) and a SINGLE GENERATOR BANK causes this much chaos on the Internet.

cperciva4• 14 years ago

If I"m reading it right, it's a failure of grid power and two or more generator banks: "each generator independently failed to provide stable voltage".

rdl21• 14 years ago

Ah, yeah, you're right (I was assuming they had 2 generators and needed both for the load, but it looks like they have 1+1 or N+1, probably split by room, like a real colo facility usually does).

kalleboo• 14 years ago

The generator thing confused me a bit. It seems like the main issue wasn't that the generators failed, but that they took far longer to spin up than expected, so the automatic switchover failed to take, and they had to do a manual switchover at a later point (at which point the UPSes had already started failing). Or am I reading it wrong?

I wonder what the additional cost would be to leave a generator running 24/7 (at minimal load), so you never have an issue with spinning them up. Are they designed to run 24/7/365, or will they wear out too quickly?

rdl21• 14 years ago

Very high cost in both fuel (a big diesel probably uses 25% of peak fuel at idle) and maintenance (idling a lot destroys the engine), and pissing off the neighbors. You're generally not allowed to run backup generators 24x7 in most areas.

It's different if you're in the middle of nowhere, but even then, maintenance issues would be prohibitive.

(It would be so much fun to build a big datacenter next to a hydro dam; you'd get both pretty high availability hydro power and the HV feed going to civilization could be used in reverse to bring power if the dam goes down; pick a site where the power goes off in two directions to two grids and you've got 2 grid feeds plus local hydro plus anything like diesel onsite backup. Then you just have issues with getting diverse fiber in, but maybe along the HV routes. Northern Oregon/Southern WA is essentially this -- which is why Amazon, Google, Facebook, Microsoft, and more are putting huge facilities there.)

As an aside, you need to test your generators with actual load. AC is a little more complex than just "on or off", too. The "real" way to do this is to do periodic tests with a load bank (basically a huge resistive load which converts the electricity from the generators into heat, although if you were smart, you could get big pumps and pump a pond instead, which could be used for something useful later). Most people don't, they just spin it up, or sometimes they use facility loads for test, which is good in some ways, but if the test fails, the facility load drops, which makes people sad. Backup power can fail in all kinds of exciting ways other than just not delivering power, so it's possible you'll switch to generator, generator will fail, and you won't be able to switch back, or in some cases, generator will kill your UPS and drop your load. The bigger the system the easier it seems to be for really weird things to happen, but the more individual sets of equipment you have, the higher the chances that at least one will fail in some crazy way. Which is one of the reasons the Google UPS-in-Server thing is so exciting.

rdl21• 14 years ago

I'm kind of confused why UPS failing doesn't lead to an emergency EBS shutdown procedure which is more graceful than just powering it all off. Blocking new writes, letting stuff complete, and unmount in the last 30 seconds would save a LOT of hassle later.

EwanToo3• 14 years ago

Blocking new writes on it's own would cause instant filesystem corruption for all the hosts using EBS, unless they had already completed their writes and had time to flush their disk caches.

You'd need to integrate it with each VM running, possibly just be sending it the equivalent of a shutdown command from the console, so that it understood that the disk was going away in X seconds, and to shutdown any databases immediately, flush all cache, unmount filesystems, and shutdown itself.

It wouldn't be massively difficult, but not as simple as simply shutting down EBS.

joahua• 14 years ago

Wouldn't there be contention issues around writes to EBS in this case? I know very little about this, but I keep hearing about EC2's relatively poor IO performance and imagine that there would be a big fat traffic jam if every running instance received a signal to get their house in order before filesystems are forcibly unmounted.

cperciva4• 14 years ago

I was wondering about that too -- I'd love to see the same thing happen with EC2 instances, sending an instance stop signal five minutes before power goes out so that daemons can be stopped and the OS can sync its filesystems to disk before shutting down.

My guess is that this functionality is missing for historical reasons -- the original S3-backed instances didn't have any concept of powering off an instance.

rdl21• 14 years ago

If they did that, they could almost cut down to 6min battery per site and skip the generators. In the event of any problem, sound the 5 minute warning, and relocate all jobs to another datacenter.

(generators and UPS might be cheaper, especially since all facilities have them, than getting all users to be good at handling failure gracefully. It's ok for an in-house datacenter doing batch processing, or with great developers, like Google, but probably not for a generalized IaaS)

zhoutong8• 14 years ago

The problem is that a lot of inconsistencies are caused by the software, not I/O or cache. For example, if your database is in a middle of transaction and everything is in the memory, you can't "let the stuff complete" without installing a software on the database server to monitor power failures.

rdl21• 14 years ago

Yes, you'd have to implement this in the application layer. There would be an API which sends out the 5 minute warning to all systems, which would then need to clean up and die before hard shutdown. This has existed in the physical server and UNIX world forever.

latch• 14 years ago

I don't know how big these data centers are, but why don't they just build a power plant right next to it, dedicated and with underwire lines?

I'm a fan of decentralized power generation, and it would seem like large consumers would have the most to gain.

Is this a regulation issue? I imagine Amazon becoming a provider of electricity (even if it's just to itself), can become a political mess.

robryan• 14 years ago

I would assume it would be a price issue more than anything.

mjwalshe2• 14 years ago

Yes you have to be Govenment RnD to aford this the big widtunnels at one RAE establishment(twinwoods) in the UK had there own HT link to a local power plant.

I suspect that Area 51 and the like do somthing similar. Though Google has sited one DC near a source of Hydro power not sure if they where running there own link to it or not.

cperciva4• 14 years ago

build a power plant right next to it, dedicated and with underwire lines

They did, and it's called "backup generators". :-)

More seriously, power plants go offline on a regular basis -- typical availability factors are around 90% due to the need for regular maintenance. You need to have a power grid in order to have any reasonable availability.

rdl21• 14 years ago

People normally try to site big datacenters with dual high voltage feeds from separate substations.

Taking down the entire grid causes many problems, and basically doesn't happen (it happened in the Northeast a few years ago....) http://en.wikipedia.org/wiki/Northeast_blackout_of_2003

The problem is diesel generators basically suck, especially when left powered off. In the long run, I predict fuel cells will take over in the standby power market.

gee_totes• 14 years ago

In the short run, what's your alternative to diesel generators? I don't think they suck.

endersshadow• 14 years ago

I visited a large data center about five years ago, and they had a power plant on site. However, California has state regulations that they have to use a percentage of power from the utility company (i.e.-they can't go completely off grid).

Maxious• 14 years ago

"many clients, especially game consoles and other consumer electronics, only use one IP address returned from the DNS query."

Is this referring to Netflix?

EwanToo3• 14 years ago

Netflix definitely had the ELB issues, I think it was described by them as "ELB routing issues" - the service was up, but some people were being sent to the wrong AZ.

That certainly sounds like this doesn't it?

rdl21• 14 years ago

It's especially interesting to look at Amazon.com vs. AWS.

IIRC, Amazon.com doesn't use ELB, they use some proper load balancers (with DNS above) in front of several Regions of ELBs.

I wonder if there's a market for global load balancing + advanced DNS tricks as a service, even if you're using Amazon entirely. Generally extending Amazon's IaaS with features Amazon should implement hasn't been a great strategy, as Amazon eventually implements an inferior but cheaper or easier (or at least default) solution for non-core parts.

Kind of surprised to see www.amazon.com goes to DC for me (from SF), though. It's possible it's all in US-East, across all the AZ's, and just has localized sites like co.uk in other places.

zhoutong8• 14 years ago

AWS officially recommends CNAME records for ELBs, but the IP addresses don't change regularly and also CNAME for root host name won't work if other records are present, so many sysadmins straightaway use A records with the ELB IPs.

rdl21• 14 years ago

I have never understood the design decision to require zone apex to be an A Record. (I mean, I know that is what the RFC says, but I don't know what the RFC authors were thinking back in ancient times.)

ceejayoz• 14 years ago

> the IP addresses don't change regularly

This is only true if you have a stable level of traffic, IIRC. If they have to scale the ELB up or down the IPs will change.

kiwidrew• 14 years ago

It seems to me that almost all of the issues revolved around the complicated EC2/EBS control systems. Time and time again, we hear about an AWS failure which resulted in brief instance outages. If it was just a dumb datacentre, the affected servers would just boot up, run an fsck on the disks, and return to service. But because of the huge complexity added by the AWS stack, the control systems inevitably start failing, preventing everything from starting up normally.

I can't help but get the feeling that, if it weren't for the fancy "elastic" API stuff, these outages would remain nothing more than just minor glitches. At this point, I don't see how you could possibly justify running on AWS. Far better to just fire up a few dedicated servers at a dumb host and let them do their job.

zhoutong8• 14 years ago

It takes time to boot up VMs, basically. System booting is one of the most expensive operations in virtualization, especially when the disk image is somewhere in the network. There will be a lot of random I/O, memory allocation and high CPU usage. It takes a few seconds of CPU time for a decent server to initialize a KVM VM. Try multiplying that by hundreds of thousands of instances.

rdl21• 14 years ago

It's undeniable that the added complexity of AWS makes it harder to predict how it will behave in any specific failure condition, which is one of the basic things you want to work out when designing it. "Traditional" infrastructure is hard enough (especially networking), and still fails in unique ways, and we've had 10-100 years to characterize it.

Having a common provisioning API in a bunch of physically diverse centers IS a huge advantage for availability, cost, etc, though. If you have 100 hours to set up a system, and $10k/mo, you have two real choices:

1) Dedicated/conventional servers: Burn 10 hours of sales and contract negotiation, vendor selection, etc., plus ~10-20h in anything specific to the vendor, and then set up systems. Get a bunch of dedicated boxes (coloing your own gear may be better at scale but 10-20x the time and then upfront cost...), set up single-site HA (hardware or software load balancer, A+B power, dual line cord servers for at least databases, etc.

2) Set up AWS. Since the lowest tiers are free, it's possible you have a lot of experience with it already. 1h to buy the product, extra time to script to the APIs, be able to be dynamic. You could probably be resilient against single-AZ failure in 100h with $10k/mo, although doing multi-region (due to DB issues and minimum spend per region) might be borderline.

In case #1, you're protected against a bunch of problems, but not against a plane hitting the building, someone hitting EPO by accident, etc. In case #2, you should be resilient against any single physical facility problem, but are exposed to novel software risk.

The best solution would be #3 -- some consistent API to order more conventional infrastructure in AWS-like timeframes. Arguably OpenStack or other systems could offer this (using real SANs instead of EBS, real hw load balancers in place of ELB, ...), and you could presumably do some kind of dedicated host provisioning using the same kind of APIs you use for VM provisioning (big hosting companies have done this with PXE for years; someone like Softlayer can provision a system in ~30 minutes on bare metal). Use virtualization when it makes sense, and bare metal at other times (the big Amazon Compute instances are pretty close) -- although the virtualization layer doesn't seem to be the real weakness, but rather all the other services like ELB/EBS, RDS, etc.)

Basically, IaaS by a provider who recognizes software, and especially big complex interconnected systems, is really hard, and is willing to sacrifice some technical impressiveness and peak efficiency for reliability and easy characterization of failure modes.

true_religion• 14 years ago

> Dedicated/conventional servers: Burn 10 hours of sales and contract negotiation, vendor selection, etc., plus ~10-20h in anything specific to the vendor, and then set up systems. Get a bunch of dedicated boxes (coloing your own gear may be better at scale but 10-20x the time and then upfront cost...),

Or just use a provider that you're already familiar with, at whatever list price they give you. This is fair since in option #2 you're assuming that we already know about Amazon, and won't stress about the 3x dedicated server cost.

> set up single-site HA (hardware or software load balancer, A+B power, dual line cord servers for at least databases, etc.

Many high-end providers will simply do this for you, and have setups ready on rack for you to use.

It's not really that hard so long as you're not going past say... 15 boxes. For anything less than that, I can pretty much guarantee you'll find a high-end provider that will have you set up in 4-6 hours maximum.

giulianob2• 14 years ago

I have only read briefly about it but there's also the new Metal as a Service (MaaS) with the latest Ubuntu. From what I understand, it gives you those APIs to manage bare metal for those cases where virtualization doesn't make sense.

pauly007• 14 years ago

So far the comments have focused on the technical aspects outlined as points of failure in Amazon's summary: grid failure, diesel generator failure, and the complexities of the amazon stack. What are your thoughts on Amazon's professionalism in their response and action plan going forward? If you're an AWS customer does this style of response keep you on board?

rdl21• 14 years ago

The level of clarity in post incident reporting by Amazon is excellent. During-incident, sub-par. Amazon seems to try to minimize any "more than just a single AZ is affected" in their realtime reporting during outages. There's also a disconnect between the graphics and the text.

What I don't like is that they make repeated promises about AZes and features which are repeatedly shown to be untrue. They also have never disclosed their testing methodology, which leads me to assume there isn't much of one. That makes me unlikely to rely on any AWS services I can't myself fully exercise, or which haven't shown themselves to be robust. S3, sure. EC2, sure (except don't depend on modifying instances on any given day during an outage). EBS, oh hell no. ELB, probably not my first choice, and certainly not my sole ft/ha layer. Route 53, which I haven't messed with, actually does seem really good, but since it's complex, I'm scared given the track record of the other complex components.

bifrost• 14 years ago

"What I don't like is that they make repeated promises about AZes and features which are repeatedly shown to be untrue."

Thats pretty much SOP in the hosting business, nobody really knows whats going to happen because nobody really knows how to test. Most developers can/do not understand what ops failures are like, and therefore most testing is only superficial.

giulianob2• 14 years ago

Well if you look at their status page ( http://status.aws.amazon.com/ ) none of the statuses shows red. Yellow clearly says its for performance issues and red is for service disruptions. If this isn't service disruption then I don't know what is.

donavanm• 14 years ago

Check your SLA definitions. http://www.virtualizationpractice.com/technically-the-amazon...

Edit: Not that I agree or disagree with that SLA. Just noting it is documented somewhere besides a glyph in a legend.

Cloven• 14 years ago

Reading between the lines from both the posted note and their persistent failure to provide correct statuses: the ops guys over there are in full CYA desperation mode somewhere around 100% of the time, and a culture of 'it wasn't me', fuelled either by job fear, or promotion fear, or fear of being noticed by bezos, is in full bloom.

nmcfarl• 14 years ago

I like that all times are reported in PDT for an event that happened on the east coast - it says something about priorities.

harshreality• 14 years ago

At least they specified the timezone, and at least they got the timezone correct (PDT, and not PST) [1].

[1] Not only do a lot of communications not specify a timezone at all, but some communications specify a timezone incorrectly. I recently submitted a ticket with Apple dev support, and the automated reply specifies their hours of operation as 7am to 5pm PST. They're probably clueless, and they mean PDT, but if someone takes them at their word, those hours of operation are off by an hour.

smackfu• 14 years ago

If the outage happened in two time zones, which time zone should they use to report? The priority is to make the write-up clear, which means picking a single time zone and being consistent. I guess you could argue it should be UTC, but practically it makes no difference.

cperciva4• 14 years ago

I guess you could argue it should be UTC, but practically it makes no difference.

Everybody knows, or should know, their time zone's delta to UTC. I'm sure there are lots of people who don't know their delta to PDT.

WALoeIII2• 14 years ago

It says that the company is headquartered in Seattle, Washington which is on Pacific Daylight Time.

res0nat0r• 14 years ago

The majority of the AWS team lives and works here in downtown Seattle, so all of the times are in PDT.

droithomme2• 14 years ago

So generators at multiple sites all failed in the exact same way, being unable to produce a stable voltage, even though they are all nearly new, have low hours, and are regularly inspected and tested.

It's impossible that it's an amazing coincidence they all failed on the same day. The fact they were all recently certified and tested means that that process doesn't work to ensure they will come on line any more than the process worked at Fukushima nuclear plant.

They don't give the manufacturer or model, and they say that they are going to have them recertified and continue to use them. So that means they are not going to fix the problem, because they don't know why they failed.

You can not fix the problem if you do not know what caused it.

ejdyksen• 14 years ago

It was only at one site that the generators failed, but it was two generators in that site:

In the single datacenter that did not successfully transfer to the generator backup, all servers continued to operate normally on Uninterruptable Power Supply (“UPS”) power. As onsite personnel worked to stabilize the primary and backup power generators, the UPS systems were depleting and servers began losing power at 8:04pm PDT. Ten minutes later, the backup generator power was stabilized, the UPSs were restarted, and power started to be restored by 8:14pm PDT. At 8:24pm PDT, the full facility had power to all racks.

It sounds like they were too optimistic about their generator startup times.

jaylevitt• 14 years ago

To my ears - and maybe this is just wishful hearing - it sounded like they were very, VERY strongly pointing the finger at a certain unnamed generator manufacturer, but doing so in a way that incurred no legal liability.

That manufacturer is probably flying every single C-level exec out to the US-East data center, over the July 4th holiday, to personally disassemble the generator, polish each screw, and carefully put it all back together while singing an a cappella version of "Bohemian Rhapsody", including vocal percussion.

And if they do it to Amazon's satisfaction, Amazon has hinted that they might decide not to out them to the rest of the world. That's called leverage.

ams61102• 14 years ago

The difference is that at Fukushima the generators were underwater from the tsunami. A bit difficult to get them started, in that condition.

drags• 14 years ago

For multi-Availability Zone ELBs, the ELB service maintains ELBs redundantly in the Availability Zones a customer requests them to be in so that failure of a single machine or datacenter won’t take down the end-point.

Based on how they behave in outages, I've always been curious (read: suspicious) about whether ELBs were redundant across AZs or hosted in a single AZ regardless of the AZs your instances are in.

It's good to hear that they are actually redundant and to understand how they're added/removed from circulation in the event of problems.

WALoeIII2• 14 years ago

In my experience you get an IP returned as an A record for each AZ you have instances in. Inside each AZ traffic is balanced equally across all instances attached to the ELB. The ELB service itself is implemented as a Java server running on EC2 instances, and it is scaled both vertically and horizontally to maximize throughput.

jpetazzo• 14 years ago

It's interesting to see that they had similar issues 2 weeks ago (power outage) and it looks like nothing was done in those 2 weeks to address the issue, since it happened again this week-end.

Summary of June 29 AWS Service Event in US-East

Related Stories