AWS: the good, the bad and the ugly

http://blog.awe.sm/2012/12/18/aws-the-good-the-bad-and-the-ugly/

by brkcmd • 14 years ago

226 85 14 years ago

blog.awe.sm

dirktheman• 14 years ago

Excellent post, and it reflects my experience with AWS, too. I really like the concept, and love S3 (and Glacier), EBS is just too unreliable for us to rely on for our entire business. Nice read, thanks!

crcsmnky• 14 years ago

I'd be curious to hear about their backup/restore procedures with just ephemeral storage.

seldo12• 14 years ago

We use Percona's XtraDB streaming backup to take backups of our smaller databases, and incremental backups of the larger DBs. We store them in a series of places: on a backup instance within east-1, on a second dedicated backup instance in west-1 (in case east-1 ever bites the dust completely, such as during hurricane Sandy), and then long-term archival on S3. S3 is good for smaller databases but for our biggest ones it takes multiple hours to download a full archive, hence the backups in multiple places.

However, our primary strategy for uptime is redundancy -- every db has at least one slave, and we are spread across multiple availability zones.

ceejayoz4• 14 years ago

Any good resources for XtraDB's streaming backups? I'm looking to move in that direction early next year.

patrickgzill4• 14 years ago

did you ever examine DRDB (I think it is integrated into some of the Linux kernels)?

joevandyk• 14 years ago

With postgresql, you can do streaming replication to a few machines. I have one large machine on standby, and then a couple really small ones, some on a different provider that just receive the updated db data using pg_receivexlog.

Stuff not in a database goes on S3.

FireBeyond• 14 years ago

Good article - it discusses things in a good way to get an overall quick look into how the ecosystem works.

I'm yet to play with AWS on any significant level, but this is the kind of thing to bookmark.

Further to that, are there any recommendations for books/sites/entries that discuss more best practices?

seldo12• 14 years ago

I've not found any, which is part of why I wrote this post! It seems a lot of people blow time and money re-discovering these things.

yegg• 14 years ago

http://alestic.com/ is the best thing I've come across (Ubuntu focused, but many general solutions as well).

FWIW, we (at DuckDuckGo) ended up in much the same place: ditched EBS, avoid anything that relies on EBS, and multi-zone and multi-region redundancy (also for latency purposes). For ephemeral storage purposes, we end up mainly using xlarge machines since they have the greatest stability and speed (with 4 drives in RAID-0).

IgorPartola• 14 years ago

> For these reasons, and our strong focus on uptime, we abandoned EBS entirely, starting about six months ago, at some considerable cost in operational complexity (mostly around how we do backups and restores). So far, it has been absolutely worth it in terms of observed external uptime.

So what do you use now for your persistent storage? This might be the most interesting part.

seldo12• 14 years ago

All our persistent storage is on "ephemeral" drives. If we lose the instance, we lose the data, so we have a lot of redundant slaves and backups (see my other comment).

bps4484• 14 years ago

How do you manage this? In my experience ephemeral storage is about 8GB. Are you able to just parition all data so that you never reach this limit, or is there a way to increase ephemeral space that I don't know about?

PS great article

chimeracoder• 14 years ago

This is a good approach if you can get enough redundancy with your instances and if instance failure is uncorrelated.

However, what do you do if there's a massive outage that affects most or all of your instances simultaneously? Diversification that fails when you need it the most isn't very helpful.

patrickgzill4• 14 years ago

Have you done any calculations as to what it would cost to rent say, 20 x $100 a month dedicated servers spread across multiple datacenters, that can do virtualization with OpenVZ, Xen, or KVM (takes care of network, power, bandwidth, hardware issues) vs. what you spend monthly with AWS?

Bluntly it seems like you must have spent some dev or ops time learning all this and migrating away from EBS etc. even if you didn't hire someone.

Frankly, if your bills are greater than $3K per month with AWS I question whether you are truly saving anything.

(I figured that midsized instances vs. dedicated servers are about 5:1 in terms of performance)

seldo12• 14 years ago

If I knew how to setup OpenVZ and create < 10ms latency VPNs across multiple datacenters, then I would be a lot better at ops than I am. While I'm much better at ops now than I was when I co-founded awe.sm, I am still primarily a developer.

Our ops guy is much better than me, but his time is better spent working on higher-stack stuff like deployment automation, monitoring and efficiency tuning than on re-inventing a virtualization stack to save a few thousand dollars every month.

If we were bigger, it would be more worth the time and money spent. But without doing the math, we would have to be quite a lot bigger, I think.

dredmorbius2• 14 years ago

> If I knew how to setup OpenVZ and create < 10ms latency VPNs across multiple datacenters, then I would be a lot better at ops than I am.

I've got the Nobel Committee for Physics on line 2 should you accomplish that trick for DCs more than 30 km (18.6 mi) distant.

Edit: Nevermind (misplaced decimal).

patrickgzill4• 14 years ago

I am not trying to be sarcastic, I am just tired of some of the AWS fanboi's (not you) who act as if AWS solves all of your problems immediately for $50 a month.

There are real costs no matter which way you go.

PS, would be very surprised if you had even 45ms latency between AWS-east in Virginia and any of their facilities on the west coast...

encoderer• 14 years ago

You can find essays from AWS users who swear it would be prohibitive--financially and mentally--to ever switch.

And you can find essays of people who run their own rack who swear the same thing about moving to AWS.

I worked at an all-AWS bay area startup that ran a $100k/mo Amazon bill, and now I work at a bay area startup that runs all its own hardware.

It's about tradeoffs. It's not just about cost. (Though there are many cases where Amazon is cheaper and it has little to do with your monthly spend, and certainly not an arbitrary threshold like $3k/mo.)

Writing this at the close of my employer's Open Enrollment period, I'd say this: Comparing AWS with running your own hardware is like trying to compare two health insurance plans. Each brings its own seemingly impossible tradeoffs. All you can do is bring your experience and judgment to identify core issues, make the best call you can, and suppress the urge to dream of the road not taken when you're knee deep in whatever hell you're experiencing that would be a non-issue if only you had went the other way.

Though if you loathe the AWS "fanboyism" so much then maybe it's the company you're keeping: Few currencies in a startup are as valuable as flexibility. The younger the startup, the truer that is.

malachismith2• 14 years ago

Thank you so much for sharing this. It's about time people started pointing out that EBS is a disaster. If we all do, maybe Amazon will finally fix it.

dogas2• 14 years ago

We here at PipelineDeals also abandoned EBS-backed instances after their 2nd outage.

Instead we rely on instances that use an instance-store root device. During the EBS outage, our instance store servers did not have any issues, while our EBS-backed servers really struggled throughout the day, with crazy high loads.

http://devblog.pipelinedeals.com/pipelinedeals-dev-blog/2012...

seldo12• 14 years ago

Great post! It's fascinating to see that other people have come up with the exact some solution -- we also run a "skeleton crew" server setup in us-west.

ra2• 14 years ago

Really interested to learn about how you use Chef to bring up and configure your ephemeral instances.

Also, do you backup your databases anywhere other than on other ephemeral instances?

dogas2• 14 years ago

We still use EBS for database backups via ec2-consistent-snapshot. The slave DB that performs this is not production-facing.

We have 2 separate chef recipes for our DBs:

One is for a full-time slave. This recipe will set up the db to use the EBS volume.

The other is for a slave that will be promoted to a master. In this case, we do a little extra legwork to do a bit-by-bit copy of a recent EBS snapshot, onto the ephemeral disk.

Sounds like I have more blog posts to write!

blantonl2• 14 years ago

This is a great writeup and is completely on target for realistic deployments on AWS.

We're a big user of AWS (well, relative, but we run about $10K/month in costs through AWS), so I'd like to supplement this outstanding blog post:

* I cannot emphasize enough how awesome Amazon's cost cuts are. It is really nice to wake up in the morning and see that 40% of your costs are now going to drop 20% next month going forward. (Like: Recent S3 cost cuts). In Louisiana, we call this Lagniappe (A little something extra.) We don't plan for it, nor budget for it, so it is a nice surprise every time it happens.

* We've also completely abandoned EBS in favor of ephemeral storage except in two places: some NFS and MySQL slaves that function as snapshot/backup hosts only.

* If the data you are storing isn't super critical, consider Amazon S3's reduced redundancy storage. When you approach the 30-50TB level, it makes a difference in costs.

* RDS is still just a dream for us, since we still don't have a comfort level with performance.

* Elasticache has definitely been a winner and allowed us to replace our dedicated memcache instances.

* We're doing some initial testing with Route 53 (Amazon's DNS services) and so far so good, with great flexibility and a nice API into DNS.

* We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project.

* If you are doing anything with streaming or high bandwidth work, AWS bandwidth is VERY expensive. We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com. That easily saves us $1000's a month in bandwidth costs, and if we need overflow or have an outage there we can spin up temporary instances on AWS to provide short term coverage.

pearkes• 14 years ago

> We're scared to death of AWS SNS - we currently use SendGrid and a long trusted existing server for email delivery. Twillo will is our first choice for an upcoming SMS alerting project.

Can you elaborate on this a bit? Why are you scared of SNS? Data loss / latency etc?

jeffbarr2• 14 years ago

Yes, please. Let me know and I'll make sure that the team sees it.

jaytaylor4• 14 years ago

SNS is frequently a huge PITA to get working, often requiring SSH tunnels and the like.

malachismith2• 14 years ago

The idea that SNS isn't first-in, first-out is scary enough to me - but the fact that it's also "at least one copy will be delivered" (rather than "only one copy") makes it kind of insane.

seldo12• 14 years ago

My ops guy has reminded me that we do, like you, have one EBS drive for snapshots of one particularly hefty DB in addition to our other backup methods.

jbarham• 14 years ago

> We've opted to go with unmetered ports on a cluster of bare metal boxes with 1000TB.com.

FYI http://www.1000tb.com/ is the website for a landscaping company, not about bandwidth...

brianhc• 14 years ago

try 100TB.com

vosper• 14 years ago

What do you see as the benefits of using Elasticache vs your own memcached servers? It seems to me that memcached is so simple to configure and manage that paying a premium to have someone else do it isn't that great of a win.

We switched to Elasticache recently as a test, and found that it works fine but don't see any compelling reason not to just manage our own memcached servers.

Edit: Also, what scares you about SNS? We use it for our oncall alerting and have seen no reason to be concerned about reliability - but I'd love to know if we should be!

ceejayoz4• 14 years ago

I think SNS was a typo for SES.

jaytaylor4• 14 years ago

You might try using the SendHub API (https://www.sendhub.com) for your SMS alerts. Way easier to get started with than Twilio's, and the groups feature makes it very easy for people to subscribe to groups of alerts.

Disclaimer: I work at SendHub, but I've done it both ways and I definitely prefer ours ;)

ghshephard3• 14 years ago

I don't get the 100tb.com model. Checking out their website, I see:

Intel E3-12303.2 GHz8GB2 x 1TB100 TB$201.15 1Gbit Dedicated Port

So, for $200/month, they'll give me 100 Terabytes/bandwidth on this server.

250 megabit/second at 30 days in terabytes = 81 Terabytes. A decently peered/connected Pipe costs, at this volume, around $5-$7/megabit @95th, or $1250.month.

So either:

  A) Their connectivity isn't hot.
  B) If you actually use that 100 Terabytes per server, 
     they start to curtail/rate limit/traffic shape you.
  C) Some other option I haven't considered?  Maybe they 
     just rely on the vast majority of their customers not 
     using that bandwidth, and subsidizing (significantly) 
     those that do?

blantonl, appreciate any insight you can provide - sounds like you've got great control over your environment. From the sounds of it, 100tb.com provides bandwidth @$0.33/megabit.

jsatok• 14 years ago

I haven't done a ton of work with 100tb.com, just a couple servers with them a few years back. It's great bandwidth. They're a SoftLayer reseller, so the nextwork blend is pretty good. I never got close to the 100tb cap, so am not sure if they'd impose some metering if you're doing it on a recurring basis. I feel like most of the people I've talked to about 100tb.com have gone in with the mentality that it's nearly unlimited, but end up using less than 1tb. If you aggregate across a ton of resold servers, I bet it's possible to make the model work pretty well.

e1ven• 14 years ago

Something that you want to keep in mind with 100TB is that they have pulled companies before, and left them in pretty dire straights.

http://gigaom.com/cloud/simplecdn-takedown-a-cautionary-tale...

I don't want to try to guess why they did that, but it's something that should never be forgotten.

blantonl2• 14 years ago

blantonl, appreciate any insight you can provide - sounds like you've got great control over your environment. From the sounds of it, 100tb.com provides bandwidth @$0.33/megabit.

I suspect it is C) - the simple reality is that most Web hosting use cases don't consume bandwidth at rates that saturate an unmetered port, other than those that violate the TOS from 100TB.com (i.e. running a CDN reseller)

We've never, ever had a single issue with 100TB.com. The servers we have with them are solely in place to provide MP3 streaming capabilities, and we certainly do get our money's worth.

trotsky2• 14 years ago

I don't know about 100tb specifically, but some general notes about the "hundreds of dollars for a dedicated port" market:

yes, the transit is not super. i wouldn't run voice on it, but i'd surely run bulk transfers off of it.

of course they're counting on underuse, that's why it's sold at port level instead of usage

they may not be shaping, but guaranteed you'll see primetime saturation pushing you (way) below port speed

this kind of bw costs them about $0.50/mbit when bought via 10g xc

patrickgzill4• 14 years ago

Current bandwidth pricing: Level3 (quite good) about $3/Mbps, Cogent (not as good), 70 cents/Mbps. So $700 to $3000 for a 1Gbit pipe depending on quality, though usually they will charge you a bit more and you will have 1Gbps on a burstable 10Gbe connection.

However - most people don't use anywhere near this amount - really, they are selling you a server connected to a gigabit switch.

As soon as over-subscription kicks in (it does immediately, since you don't start using 250Mbps the day you have your server installed) their bandwidth costs go to a fraction of what you are paying monthly.

falcolas• 14 years ago

> RDS is still just a dream for us

In my experience, with large data sets (I'd say > 100 GB), you want to run your own database instances, if only so you have control over the small fiddly bits you need control over.

RDS: 1) Uses drbd for master failover (this introduces network overhead on disk writes), 2) Only gives you access to super commands through stored procedures, and 3) doesn't let you tune InnoDB to your dataset or performance requirements.

deckchair• 14 years ago

I agree, a good post. AWS has stated that they are working on the link between EBS and ELB: https://aws.amazon.com/message/680342/

The point in the post that the AWS management console runs on EBS is fishy...

trotsky2• 14 years ago

The elasticity, no capex and api management are great. But bending over backwards to deal with their terrible SAN, saturated layer 2 and low reliability easily wastes at least as much engineer time as it saves in ops. If the OP's 100 boxes are similar to most of the other AWS deployments I'm familiar with it's likely that'd be 10 or less actual machines to manage.

Don't get me wrong - the idea is great and it's great for really low end workloads where the gotchas don't matter or really big systems where you'd be managing against most of those issues anyway.

But in the middle of the two there is a dead space that I bet a lot of shops are stuck in - double digit instance numbers running a workload that a 2-4 servers could handle with plenty of leg room.

If you're deciding on a cloud provider in 2012 I think it makes a lot of sense to shop around. There are lots of people doing the on demand api deployment thing now with different trade offs. I like joyent a lot (local reliable io) or providers with cloud and a colo area even if it's exorbitant - as paying five hundred dollars a month for instnaces one ssd could replace sucks.

moe2• 14 years ago

I second this. If you are in the position to commit either way I'd suggest to first run a simple benchmark with your app on EC2 versus some cheap dedicated server.

As parent says: EC2 does have its place, but for deployments in the 10-20 hosts range (or an app with "special needs") you tend to pay through the nose. Keep in mind that the amount that you pay for a fraction of a server on EC2 will rent you the entire box elsewhere.

druiid2• 14 years ago

Pretty much exactly this (I've posted this statement nearly word for word several times in response to articles like this). Basically EC2 or Rackspace Cloud, etc. are very good for small startups running a handful or two of instances, or really large companies that need hundreds/thousands of instances under which scenario hardware might very well be the more expensive option.

jwilliams• 14 years ago

Nice read, but I wish they had included the location (and AZs) that are in use. I've used Oregon, California and Virginia with different results.

The comment around Ubuntu is interesting and I wish there was more detail there.

We use mdadm to run RAID across multiple EBSes. mdadm is great, but has a kink that it will boot to a recovery console if there the volume is "degraded" (i.e. any failure). This is even if the volume is still completely viable due to redundancy. This is obviously very bad, as you've got no way of accessing the console. It's an unfortunate way to completely hose an instance.

It's an easy one to miss, as you rarely test a boot process with a degraded volume. When it happens though - hurts a lot.

(If you'd like to check on this, make sure you have "BOOT_DEGRADED=yes" in /etc/initramfs-tools/conf.d/mdadm).

seldo12• 14 years ago

We are primarily in us-east-1, as I mentioned in the post, with a skeleton set of DB slaves sitting in us-west as an emergency recovery if all of east-1 goes down.

In terms of AZs, were are distributed roughly evenly across all AZs in east-1.

leoc• 14 years ago

> The failure mode of EBS on Ubuntu is extremely severe: because EBS volumes are network drives masquerading as block devices http://joyent.com/blog/magical-block-store-when-abstractions... , they break abstractions in the Linux operating system. This has led to really terrible failure scenarios for us, where a failing EBS volume causes an entire box to lock up, leaving it inaccessible and affecting even operations that don’t have any direct requirement of disk activity.

Assuming that I/O is reliable and fast-ish was a bad OS design decision in the 1970s, as the Unix guys themelves soon realised, though it was an understandable mistake at the time. Continuing to develop and use OSes with an I/O interface that assumes I/O is reliable and fast-ish, in 2012 - it's pretty bad, isn't it?

zurn2• 14 years ago

Hardware and software have co-evolved, so that disks provide an illusion of error free operation until they throw in the towel and die. And they have consistent performance. This has worked OK so far.

With network filesystems (eg. NFS) you can choose to return an I/O error to the application when you hit a timeout or a network error (the -o intr mount option). This is rarely used since applications aren't used to dealing with them. So can't really blame the OS here either.

spartango2• 14 years ago

On one hand you're right: when block devices fail, they're pretty much gone--if you've ever tried to read from a bad sector you'll know this exactly, or worse still on SSDs, where the disk will fail to show up to the bus.

That said, I'm not sure I agree with the idea that we got away with a lack of error handling because disks had consistent performance. Magnetic disks have always had incredibly inconsistent random IO performance, and even inconsistent performance between different parts of the platter(s). And in the spirit of co-evolution, we engineered around it: OS disk caches are critical to decent performance on HDDs.

I think it's not that we found disks to be consistent, as much as our solution to their suboptimal behaviors was caching, rather than error reporting/timeouts. I believe this is because caching was the most transparent approach; a good cache makes a variable-speed disk look just like an ideal disk, so an application can be written assuming the disk is perfect.

Interestingly, as distributed systems have evolved, we've ended up having to engineer the error-handling constructs that might have been used for block devices; we see them in network filesystems (as mentioned) as well as most other network services. Applications have been designed to deal with errors. We just haven't propagated those constructs down to the disk devices in the recent past.

daemon13• 14 years ago

From alestic.com

>> Both EBS boot and instance-store AMI ids are listed, but I recommend you start with EBS boot AMIs.

Why two opposite recommendations from alestic.com [authority on AWS] and practitioners?

Not a flame - I am planning my AWS deploy strategy and need to make a decision between these two approaches.

spartango2• 14 years ago

As a bit of background: a lot of the performance pain around EBS is the inconsistency in performance between IOs. Many times you'll get a nice fast op, but sometimes IOs will get slow suddenly, or even stuck. You can imagine that internally, this is EBS seeking on magnetic disks or getting blocked by a clogged network. Regardless, inconsistent performance is nearly as bad (or perhaps worse) than consistently bad performance in certain cases:

Thus, most of the time, performance issues with EBS bite you when you have application data on an EBS volume. If you're constantly hitting the volume to serve data to customers, you'll be seeing the hiccups in EBS performance and passing them right on to customers. Clearly this is a suboptimal experience, and can lead to other failure modes; a slow/stuck bunch of ops can get an application to completely stall.

It's important to note here that EBS has no error reporting system or timeouts; common operating systems aren't very good at handling disk IO errors, so EBS will never produce them, even when its having issues. This lack of transparency can make handling/working around stuck EBS operations nigh impossible.

So, having experienced these pains, people generally say "Don't constantly use EBS if you care about your application performance/stability," and they aren't wrong.

That said, if you use EBS as a boot volume only, you are likely not hitting that volume enough to feel the pain of slow-IOs; once booted you won't touch it much. At the same time, using EBS as a boot volume has all kinds of conveniences: the ability to start and stop the instance without loosing data, snapshots and AMI creation, persistent machine-specific configuration, etc.

I hope that's useful. EBS is not a perfect service, but it definitely has its uses. EBS as a boot volume has made using AWS quite a bit friendlier.

Note: I worked on the EBS team at AWS a few years ago. I currently use EBS in my startup's infrastructure. If you have further questions, feel free to reach out to me.

seldo12• 14 years ago

As you say, the real trouble with EBS is when serving application data. We run a lot of databases, and EBS was just not good enough for them even when there were no system-wide issues.

Our problem with EBS as a boot device was that occasional system logs, etc. would go it, and when EBS was down the inability to reach the disk (the block-store abstraction problem) would lock up the whole OS.

jiggy20112• 14 years ago

How much does AWS save in terms of admin vs a standard VPS?

I assume it runs a standard linux distro, therefor patches , firewalls, dependencies etc are still an issue surely?

druiid2• 14 years ago

Here is actually a real issue with many of these startup type companies using EC2 or similar services... one of the very benefits that are brought up (not needing to have someone building and maintaining servers) is also the person whom generally would be taking care of those services (or abstracting them away with CM solutions).

Many times what I've seen is simply that position is simply 'filled' by developers whom have ideas about designing and securing scalable systems, but when it comes right down to it... they get hacked within some period of time because it turns out, they don't know these things well enough to take the place of someone whom dedicates their time to it.

ceejayoz4• 14 years ago

> I assume it runs a standard linux distro, therefor patches , firewalls, dependencies etc are still an issue surely?

AWS's security groups pretty well cover you on the firewalls front (although some folks like to still run software ones on their instances to be doubly sure), but you're otherwise correct.

For a single instance, you're not going to see significant differences. It's running large clusters of instances you're going to see things being better than the average VPS provider, with stuff like Virtual Private Cloud etc.

jiggy20112• 14 years ago

I mean assuming that you want to open SSH you are going to need to secure that either using keys or fail2ban. If you have a web server you might need mod_security (or does AWS reverse proxy everything?). If you want remote database access you need to configure that too.

Does AWS provide easier tools for that stuff? Because you can't really cover it with a general firewall.

novaleaf• 14 years ago

from reading the "bad" parts in the article, man, what a pain in the butt. I guess i'm glad to be on Google app engine and not have to worry about this stuff.

pjungwir• 14 years ago

I've run a few projects on AWS and agree with how much it simplifies your life, but like the OP the big sticking point is EBS. I wasn't aware that ELB relies on EBS. Good to know!

I've also looked into ephemeral storage, but ultimately I decided to just rent a dedicated machine from elsewhere. Building a B2B site, I'm not as worried about massively scaling on a dime. The project has still used transient EC2 instances for a few odd things, though. It's nice to have that option when you need it!

Shorel• 14 years ago

To me the decision is simple: if you need less than n servers, AWS is better and cheaper.

OTOH, if you need equal or more than n servers, rolling your own dedicated servers is better and cheaper.

The only issue is to determine the current value of n.

dschiptsov• 14 years ago

Yes, ability to clone a new box you don't own in a minutes the only real advantage.

And for that very uncommon requirements you have to pay per Kb and per hour rates for having no service or guarantee at all.

And no, you still need a sysadmin who understand how AWS works and what to do when AWS says "Oops, your "_____" isn't available".)

AWS: the good, the bad and the ugly

Related Stories