I build High Availability Platforms so Cloud is not for me

http://blog.networksaremadeofstring.co.uk/2012/07/01/i-build-high-availability-platforms-so-cloud-is-not-for-me/

by nicktelford • 14 years ago

65 42 14 years ago

blog.networksaremadeofstring.co.uk

This is an excellent cautionary article... I still don't get all the love for AWS after they have shown that they have less uptime than many dedicated server providers.

Reliability is the greatest feature you can offer, as a downed service can't be used by anyone.

peterwwillis3• 14 years ago

The big trade-off with AWS is it allows poor people to afford lavish computing resources for small amounts of time, at the expense of things like performance and reliability. If you have slow, steady growth metrics it makes much more sense to host yourself. Planning for peak capacity and buying only what you need is cheaper than dynamically scaling what you need.

The product is geared to a specific kind of customer, and frankly I have no idea why companies like Netflix use them. Hell, Netflix actually buys whole instance blocks [servers] just to work around the shared i/o of EC2. You'd think just buying a real dedicated server would be cheaper and easier.

legutierr• 14 years ago

Netflix does get massive discounts, though. I remember reading that it's like 70%.

ceejayoz3• 14 years ago

> The product is geared to a specific kind of customer, and frankly I have no idea why companies like Netflix use them.

I'd imagine Netflix sees enormous fluctuations in required server capacity. Not many people watch movies at work, but when they come home, or when it's a weekend, traffic probably spikes tremendously.

gouranga• 14 years ago

For me, I'd rather go dedicated or colo at least, simply because I know what the process is when TSHTF and can communicate it reliably to my clients.

With black-box cloud services such as AWS, Azure, you just don't know what is happening.

Peace of mind is also important!

ceejayoz3• 14 years ago

On the other hand, though, if half the internet is offline with an AWS problem, chances are pretty decent a lot of folks will just go "oh, the Internet is having troubles again".

omh• 14 years ago

The point about high-availability platforms is that your customer won't go "Oh, the Internet is having troubles again". They'll probably call you, or perhaps start losing money at an eye-watering rate.

rdl• 14 years ago

The problem is AWS makes representations about how available certain components will be, and the independence of various features like AZs, which turn out to be lies in practice. When a higher level system builds on top of AWS and relies on these representations, the higher level system fails.

Carriers lie about route diversity on specifically ordered diverse circuits all the time. I rarely physically trace every power cable in a colo to make sure A and B are independently fed; if someone has done it correctly 30 times before, 31 is probably going to be ok. If it is critical, or something which might be difficult, I may check, but since I am confident it will pass, I'll bundle a bunch of checks together so it only passes if all the checks pass.

jxcole• 14 years ago

I think the real reason people like Amazon cloud is for people like me who are decent programmers but who don't know or don't want to know how the system works internally. This guy sounds like he has a lot of what I would call "ops" knowledge which is great for him, but if I decided to start running my own website today I probably wouldn't have the chance to learn all of that stuff.

giulianob• 14 years ago

Cloud providers are trying to abstract it all away while in reality the real world factors cant be abstracted. E.g. latency if servers aren't close enough, I/O performance due to virtualization, disk latency if you are using block storage, etc... have to still be understood. What I've noticed with people using the cloud is that they have to still understand how the cloud services work and many times make up for it in app logic. Look at Netflix for example having to make their entire infrastructure extremely tolerant to latency due to the spiky nature of AWS. At the end of the day, you will have to deal with these factors. You can either do it by learning how the cloud works and figure it out there or by running your own infrastructure. One important point the article makes is that if you roll out your own you can at least control those factors.

It's not to say that clouds aren't useful. They provide a lot of value to many apps but it's not a silver bullet that makes it so you can be ignorant of infrastructure. Maybe one day it will be but as of right now it's important to understand the tradeoffs if you plan on using the cloud.

seiji2• 14 years ago

don't want to know how the system works internally.

I see this sentiment a lot. People seem to be proud of technical ignorance ("We don't even know where are servers are! lol!") because they are too busy doing whizbang gollygee Important Work like making the world's four billionth PHP CRUD app and setting up a DB without durability guarantees.

"But we don't have time to learn... we are busy being cool and popular!" Sometimes you have to slow down and actually learn things. Sometimes you should be an intern for a year before being a CEO. (It sounds like you've already resigned to not learn things: I probably wouldn't have the chance to learn all of that stuff.)

what I would call "ops" knowledge which is great for him

That's kinda impressive: dismissal condescension towards someone who knows more than you. It provides the ego with a nice padding. "They know more than I do, but what they know is silly and useless in these modern cloud-based times. I feel sorry they actually had to learn how things work."

Programming means manipulating a system. Basic/beginner programmers pretend the entire system is the API of their favorite (read: ruby) programming language. You can get pretty far not learning more than that, but you will always be limited. You will always be pretending to understand more than you actually do.

If you aren't a "computer person," you'll stop at the basic level. You'll spin your wheels cranking out things that feel the same forever. What separates a "computer person" from people just in it for the glory or money is their insatiable knowledge hungerness. They want to be good, better, and then optimally best.

Intermediate programmers realize (and have a mental model of) how everything works together including a basic CPU, bus, memory, network (from hardware up to packet switching through routing and DNS), your language stack and heap, calling conventions, directly attached storage, network attached storage, and half a dozen other things depending on speciality (graphics? wireless? robots? distributed system? embedded system? web?).

Advanced programmers do not exist.

ianterrell• 14 years ago

>> don't want to know how the system works internally. > I see this sentiment a lot.

I like that sentiment a lot. We have finite time and finite brain cycles. Stand proudly on others' shoulders and build something bigger.

There's no shame in programming in high level languages to deliver more value more quickly. Ditto with operations, ditto with everything.

jakejake• 14 years ago

I do think it's true that good programmers have a passion for understanding as much as they can. But there are so many levels and it's impossible to master everything. Not only that, but things are constantly evolving. If you set about to master everything you can drive yourself crazy.

I have a friend who is an incredible engineer, he writes high performance code at the C and Assembly level. He knows almost nothing about higher level code. You could say in a way that he's lazy because he wouldn't know how to write a simple web page, but that would be missing the point. He's critical for the work that he does.

Being a jack of all trades is helpful at times, but a team of people who specialize can go into depth that just isn't possible for one individual.

michaelt• 14 years ago

Personally I'm interested in the design of a many systems - but I can see a case for wanting other systems to present a single well-defined, correctly implemented interface.

There was a time when, to make a program with sound, programmers had to handle different sound cards differently. You used to get settings in games to choose whether your card was Sound Blaster 16 Compatible.

Were those the good old days, or the bad old days?

scott_w• 14 years ago

I think you're interpreting it wrong.

What you see as celebrating ignorance, I see as celebrating simplicity: "We don't _need_ to know how it works any more!"

For many use cases, this is perfectly acceptable (and the article rightly notes this).

"That's kinda impressive: dismissal condescension towards someone who knows more than you."

I saw nothing dismissive about that statement whatsoever. I read it as "they know more than I do about that particular subject, but what they know is not useful for me to know at this moment in time".

catshirt• 14 years ago

so are you suggesting that everyone should learn everything?

or that people shouldn't attempt things without expertise?

mdc• 14 years ago

There is value in specialization. You denigrate the OP with your condescending "too busy doing whizbang gollygee Important Work" and mischaracterize him as using words like "silly and useless", but the OP never said that. All he said is that he's focused on one part of the work and doesn't know that much about the others.

I've been considering these choices a lot. In my own career, I've made a conscious choice to pursue lots of different things that I found interesting, but I've become a generalist, pretty good at lots of different things but lacking the deep domain knowledge in any one of them that a specialist gains. You seem to be arguing that specialists should not exist, everyone should go for broad rather than deep. I think there's a place for both, and becoming a specialist often requires the discipline to stay focused and not pursue every interest and curiosity that rears it's distracting head.

I think there's value in a programmer knowing the whole stack, from language APIs to hardware implementations, and being able to understand the consequences of every line of code. But time is finite and someone at the beginning of their careers must make choices on where to build their expertise.

Draiken2• 14 years ago

Very well put. Every good programmer knows the underlying parts of what he is using, and unfortunately the new trend is that programmers don't need to know this stuff anymore.

It's true that in some cases they don't have to, but they should, and for some reason people get offended when you tell them that...

rhizome• 14 years ago

I see it simply as a failure of developer-centric hiring.

paulsutter• 14 years ago

Yeah it's a real hassle to learn all that stuff.

Fortunately everyone here reads Hacker News so we already know that AWS has downtime that can affect whole regions, and that platforms like Heroku don't yet provide resiliency that spans regions.

So if you don't really care about occasional downtime, you don't need to bother learning all that hard stuff. It's perfectly ok with me if your service is down.

swiil• 14 years ago

This article reads to me like someone who lives in an old business model watching the internet come to town and screaming that it will never catch on! Will there be a place for legacy architecture like this for many years to come? Yes. But each time these cloud architectures fail, we are learning how to deal with that failure and improve our collective availability to remain up through out disaster.

mfenniak• 14 years ago

My thinking is that I want experts working on the low-level configurations (power, racks, UPSes, etc). I don't have that expertise myself. Now, I could try to hire some experts to set up highly available hardware from me... or... I could rent some that has already been set up. I think the basic assumption you have to buy into is that your cloud provider knows how to do this better than I do.

I think if we look at a AWS reliability, the interesting and sorta-unanswerable question is: would these applications have been more, or less, reliable, at the same cost, if they were running the infrastructure themselves? My guess: less. It's so easy to screw this stuff up.

joeblau• 14 years ago

I like this article. I used to work on HA systems and I saw systems that were really HA all the way down to the electrons. Right now, I would rather have someone like you who enjoys building HA architecture and I would rather be writing software.

The problem is that you are good(expensive) and most people can't afford to build out and maintain a true HA system so they go for the cheap alternative and in that case, you get what you pay for.

famfamfam• 14 years ago

Not to mention the fact that in addition to you being good (and therefore expensive) to have a HA system, you would need at least two of you.

hafiend• 14 years ago

"Instagram did everything right with load balancing, horizontal scaling and lots of monitoring but they still went down."

"Right" means something different to you than me. I build real-time systems and quasi real-time systems across RTOSs and nixes.

They relied on a single cloud provider. If you want seamless resiliency you need redundant pairs at the very least across all failure scenarios.

Can you go overboard on this? Absolutely.

But did they do _everything_ to mitigate this ahead of time? Absolutely not.

Did they do this "right" according to what their business requirements (/risk profile) were? Who knows.

njharman• 14 years ago

Ya sure. I'd rather have some downtime than have to "run cables". It's not worth the effort for 95% of systems to be that resilient.

Draiken2• 14 years ago

As stated in the article, it's not wrong to do what you're saying, but the article is talking about people that can do that and should do that.

Quoting the article: When it gets to the point where you start measuring downtime in dollars rather than “time I would’ve been doing something else” it is time to move your critical infrastructure to something you know.

notatoad• 14 years ago

This post seems like somebody looking for an excuse to attack "the cloud". AWS's issues last week were caused by a whole datacenter going down, due to an electrical storm. No matter what your configuration is, whether you use cloud hosting or traditional hosting, losing a datacenter is going to throw a wrench in things. Even if your failover works perfectly, the increased load on whatever you fail to might bring things down.

The procedure for high availability is the same for cloud hosts as traditional hosts: multiple datacenters and multiple providers. If somebody made the decision to only host only with amazon and only in US-East, the traditional hosting alternative to that is not a high-availability network spanning 5 datacenters. You're comparing apples to oranges. You can host with AWS across multiple regions, just like you can host with any other provider in multiple datacenters.

imperialWicket• 14 years ago

At some level, I can agree with this sentiment. You don't know where the servers are, you can't see the cables, and a lot of things are just plainly out of your control.

TL;DR: AWS can realistically provide high-enough availability for most; but I think the article makes good points for those looking to get deep into the nines.

However, I think the real issue is that while you can make fairly HA systems on AWS, it quickly gets expensive. A huge pull for AWS (IMO) is how cheap it is for an org to spin up four or five servers of various specs, and keep them running or shut them down. In order to achieve some aspect of HA through AWS, you need to mirror your entire setup in three places (not to mention some archives in S3, and ideally some archives on non-AWS). The cost savings start to quickly degrade as you configure your AWS-hosted system for real HA (or real close to HA).

IMO, many of the organizations that use AWS are in a position where they probably wouldn't have true HA configured if they were using VPS or physical hardware, so it's not so much of a concern. For the larger organizations who have more funds and more dedicated Ops employees, we've only seen a couple of issues that should have affected availability, and usually availability drops because they cut corners. Granted - at least two of the recent outages showed errors on the AWS-side, and decisions that Ops made for AWS regions/AZs should have been fine, but weren't. That in mind, any HA sysadmins out there would have shuttered to think of relying on two different mechanisms that were potentially in the same data center (that is, backup servers or load balanced servers in the same region).

I build High Availability Platforms so Cloud is not for me

Related Stories