The reliability pillar of the AWS Well-Architected Framework [pdf]

https://d1.awsstatic.com/whitepapers/architecture/AWS-Reliability-Pillar.pdf

by robfig • 7 years ago

149 60 7 years ago

d1.awsstatic.com

inlined• 7 years ago

The section about calculating the availability of hard and redundant dependencies ignores the fact that systems often fail in tandem. For e.g. you might have a primary and secondary database in different AZs, each with 99.9% availability. This gives read operations a hypothetical uptime of 6-9s. But hidden SPOFs like operator error, VM infra failing, load balancer or other networking outages can make the 0.01% failure time overlap, blowing away your 6-9s dependency guarantee.

Tinned_Tuna• 7 years ago

p.14:

"Because 99.999% availability provides for less than 5 minutes of downtime per year, every operation performed to the system in production will need to be automated and tested with the same level of care. With a 5 minute per year budget, human judgement and action is completely off the table for failure recovery"

candiodari• 7 years ago

That just gets you back to automated systems, for example, causing their own failure, then responding to it by causing more failure.

For example, a relatively common occurence is a BGP link getting saturated. Get that situation bad enough and the BGP session will go down, which will redirect all that traffic to another link with another BGP session, which then proceeds to go down. Meanwhile the original session comes back up and ... And then the failures synchronize and cause a third link to go down (each time taking more traffic with it and therefore causing failures faster).

The second issue discussed is that the math in the statistics only works if the failures never synchronize. That's true for a lot of statistical analyses and mostly people ... just don't care. Yes that makes those analyses wrong. But we don't have a better way of doing those analyses.

crankylinuxuser4• 7 years ago

(erased)

FigmentEngine8• 7 years ago

This is not true. AWS provides multiple constucts to help isolate failures, such as Availability Zones that provide seperate locations, and Regions that are designed to be completely isolated from each other. You can also use different AWS Accounts to provide logical seperation of your different workloads and limit management.

aarghh• 7 years ago

Couldn't agree with you more. Specifically, certain (poorly-thought out) failure handling mechanisms actually end up transmitting the very condition that triggered the first failure to other parts of the system - thus creating cascading failures.

koolba• 7 years ago

Many run into this when they learn the hard way that “RAID is not a backup” as wiping out all your data on one set of disks also wiped it on the other. Ditto for naive replication with no rollback capability. It’s a fancier version of the same thing.

“DROP TABLE customer”

“Oh crap! Better failover to DR. Oh crap! It’s dropped there too?!”

earless1• 7 years ago

exponentially delayed replication down the replica chain...

juskrey2• 7 years ago

Out of the model risks is what usually kills you.

quirmian3• 7 years ago

This is a tautology, because if the risk was successfully modelled, it would not be able to kill you.

random42• 7 years ago

"... For example, if a system makes use of two independent components, each with an availability of 99.9%, the resulting system availability is >99.999% ... "

This does not seem correct.

necro• 7 years ago

I suppose it depends if the use/need is in serial or in parallel.

alextheparrot2• 7 years ago

Independent and redundant components is the missing context the quote was taken from, wherein the quote is correct.

maltalex• 7 years ago

What they probably mean is that if each component can fail independently with a probability of 0.001 (0.1%) then the probability of both of them failing is 0.001 * 0.001 = 0.000001 (0.0001%)

If the system depends on just one of the components working then 1 - 0.000001 = 0.999999 (99.9999‬%)

pkolaczk2• 7 years ago

This is correct, however in reality completely independent components are very rare. Even things that seem independent and truly redundant e.g. jet engines of an airliner, are much more likely to fail after one of them fails. Therefore this line of reasoning must be applied with extreme care.

unixhero• 7 years ago

Unless someone explain or argue that this statistic, when a system with such high degree of availability let's say >=99.9% also can be said to have other properties beyond just the mere statistical nature. If not the resulting availability should be 99.8001%

crankylinuxuser4• 7 years ago

(erased)

klodolph6• 7 years ago

Yep. Amazon (GCP, Azure) are making bank off the idea that you can just pay opex to run in multiple regions and lay off your SREs, but if you really wanted high availability (and not just some service credits when things go wrong) you would spend the engineering effort to go multi-provider. At that point on-prem looks better, but at least with multi-provider you’re no longer in the business of ordering hardware and power.

The idea that failures between different systems are uncorrelated doesn’t even work at a hardware level (e.g. two different sticks of RAM), it’s pure fantasy when you’re talking about software stacks, especially when you have things like the blow-up-the-world button called “BGP.” I’m deeply uncomfortable reading formulas for availability that add or subtract nines, if there’s real money on the line, you can afford to do some better math.

Disclosure: Work at a cloud provider, opinions are my own.

cheeze2• 7 years ago

Something that often goes ignored is that in high availability situations like that, humans are the biggest risk by a huge margin. Computers (almost always) don't accidentally wipe out the prod database thinking it was beta, that's usually a human that made a simple mistake.

Dumping money into IaaS while neglecting to recognize that high availability requires extreme quality of both engineering, as well as testing/qa is a super common mistake that's being made these days. This means slowing down, which most upper management doesn't want to do.

Things like executing DR plans regularly to test that they wofk are extremely costly to the business,and are one of those "IT" things that you hopefully never have to actually do in production. But without things like that, what's the point of 3xing the cost of your fleet and adding complexity (another great vector to cause failure) for "availability"?

I think the answer often comes down tok the fact that people are hard to hire. Computers are easy to provision.

FigmentEngine8• 7 years ago

I can't say for your employeer, but at AWS this is not our intent. Our job is to help customers build better architectures, and if you read the framework you will see its agnostic of vendor.

You can't approach reliability with the "the sky is falling down, so theres no point" approach. You are actually going to have think about component failure, blast radius and what happens afterwards a failure.

rather than spreading FUD, why not help customers by showing them the better math?

lrem• 7 years ago

Uh, don't you need to HIRE some SREs to get that multi-everything setup engineered correctly?

cheeze2• 7 years ago

Of course AWS wants you to use them. They would never market for a competitor, that just makes 0 sense.

In terms of reliability, AWS and GCP have solid track records in terms of multi-region availability (I can't say the same about Azure... They seem to be years behind still.) Cost absolutely goes up when you're building out regional fault tolerance, but that's true whether you use a mixture of cloud providers or any specific provider. If you need high availability, you need to waste some level of resources (and in turn, money), it's part of the design.

I'd argue almost anyone would be fine running out of a single aws region. There are specific cases where extreme availability matters, but in that case you're making the tradeoff of cost and simplicity for reliability.

It would be interesting to see something like a study on whether using multiple regions in one cloud provider makes more sense over having your load spread across providers in terms of complexity versus fault tolerance.

FigmentEngine8• 7 years ago

This is not the intent at all, I work in the AWS Well-Architected team, and we are engineers trying to help other engineers. If you follow the best practices you will spend less (there are performance and cost optimization pillars).

crankylinuxuser4• 7 years ago

(erased)

cavisne• 7 years ago

AWS has not had a global outage in a very long time (I think there was one S3 outage that was global because at the time there was only 2 regions).

The venn diagram between services that need very high availability, and yet have such low traffic that distributing resources across 2 or 3 AZ's is a waste of money, should be pretty small.

Multi cloud for a individual service will never make sense as all the providers know how to price network transit to make this infeasible.

iblaine2• 7 years ago

Interesting that security is 1 of 5 pillars and hardly gets a mention in this paper. Who wrote this and why?

FigmentEngine8• 7 years ago

It was written by the AWS Well-Architected team, and is based on curating best practices from Solutions Architects working with customers and engineering teams. Its one of Six main papers, with the main framework, and then one per pillar https://aws.amazon.com/architecture/well-architected/

There is also a free tool available to review architectures. https://aws.amazon.com/well-architected-tool/

The aim is to help customers learn best practices for building and operating in the cloud. (I work in the AWS Well-Architected team)

maxmcd• 7 years ago

https://d1.awsstatic.com/whitepapers/architecture/AWS-Securi...

iblaine2• 7 years ago

Thank you

Jedi72• 7 years ago

This is total submarine marketing by AWS. Im sure some very smart people with good intentions at AWS produced this, but the fact is they're not an independent academic providing objectively "the best" practices, they're a company trying to sell you something. They probably in full belief in their own abilities believe the AWS way of doing things is best, but frankly, I know as much as you do, or at least if you want to win me over you need to comprehensively prove it, not just preach from the hill.

TL:DR stop telling me you're doing better architectures in the same of selling me something

sciurus• 7 years ago

I don't look at the well-architected framework as "the AWS way is the best way of doing things" but rather "this is the best way of doing things on AWS".

The reliability pillar of the AWS Well-Architected Framework [pdf]

Related Stories