A Comprehensive Guide to Building a Scalable Web App on AWS

https://www.airpair.com/aws/posts/building-a-scalable-web-app-on-amazon-web-services-p1?wed

by jkresner • 12 years ago

542 129 12 years ago

jkresner3• 12 years ago

Apologies in advance for the table of contents going way off the screen. This is the biggest post published so far. We'll be doing some UX work on the table of contents widget in the next week.

coolgeek• 12 years ago

While you're at it, how about a reasonable print stylesheet? I had to do a lot of surgery on the node.js tips article, but still gave up because I could only get a max of 19 lines of the code snippets to print

jkresner3• 12 years ago

We've scheduled a lot of UX design polish on our posts schedule in the next 2 weeks along with a big announcement first week of feb for you to watch out for.

subrat_rout• 12 years ago

Josh. This article is great and very helpful. Also I would like to know if you have any plan to write more articles on AWS that are more in depth.

Few examples such as: 1. What are the security measures somebody needs to put in place to host a HIPAA compliant Web app etc.

2. What are the popular stacks that can be set up on AWS without much hassles and with less expertise.

3. What are day to day activities required to maintain few web apps on AWS. etc

jkresner3• 12 years ago

Josh has a part 2 in the works :)

rwfilice• 12 years ago

Agree would love to see the HIPAA compliant article.

wallflower• 12 years ago

Please create an eBook and sell this on the Kindle store. Thank you! Amazing dive deep and fly over high overview.

artifaxx6• 12 years ago

Out of curiousity, why go with AWS when Linode, Digitalocean, etc appear to be so much more cost effective? Is the simplicity of spinning up AWS instances really great enough to counterbalance what appears to be a significantly greater cost? Is it the flexibility of different AWS services?

taf2• 12 years ago

It's more than just cost savings it's

Flexibility of networking, Regional coverage, Tooling , Performance,

On the cost side with reserved instances it's less expensive.

artifaxx6• 12 years ago

Really? I know reserved instances lower the price but they still have shown themselves less performant for the dollar in the benchmarks I have seen. Are there any good articles you could recommend with benchmarks showing AWS to be more cost effective?

ridruejo• 12 years ago

The advantage of AWS is the scale at which it can operate and the rich ecosystem of services (S3, ELB, CloudFormation, RDS, etc) that you can use as building blocks for your app. DigitalOcean, Linode are more like VPS providers and are only suitable for a subset of the use cases AWS covers (or Azure or Google Compute Platform)

artifaxx6• 12 years ago

Good point, it becomes clear that if you are operating at the scale of a large company the large scale and rich ecosystem are important. It was just a bit surprising to me that for smaller scale instances AWS is so much more expensive. It would seem they should be cheaper due to economies of scale.

stephenr9• 12 years ago

If you are/have skilled sysadmin staff, there is very little if anything AWS offers that you can't run on your own rented/owned phsyical/virtual servers.

I'm not suggesting that's ideal for all companies, but this idea that somehow it's impossible to run a load balancer or store files without using AWS/Azure/Google 'clouds' is a bit ridiculous.

joshpadnick3• 12 years ago

Article author here. So, you're right that while AWS does continue to lower prices, they're still not the cheapest game in town. Frankly, they're not even necessarily the most performant game in town.

What they are really competing on is breadth and depth of service. The article goes into a lot of those services, but, as one example, if you launch an instance in EC2 you can allow it to access secured buckets in S3 without any need to store keys/passwords on the instance itself thanks to IAM roles.

Another example is services like AWS Lambda, which is a hosted way to run a function without any need to manage servers.

The list goes on and on...direct VPN connectivity, Hosted Active Directory, CloudHSM. While I'm biased, my perception is that AWS is pretty far ahead of the pack.

artifaxx6• 12 years ago

Thanks for all the replies. I am just researching all of this for my own startup and it is important to understand all the tradeoffs. And it is clear from your article that AWS has a very deep feature set. Its a good article!

gtaylor2• 12 years ago

Was that Hosted Active Directory from one of the marketplace AMIs, or are they offering this as a built-in service now?

pjwal• 12 years ago

I think that's spot on, it is all of the bells and whistles along with a fairly predictable management cost, even if you are new to AWS, that make it a safe choice for a big swath of the market.

Kudos to you for touching on containerization, I would just like to add that the many emerging platforms and tools (https://pbs.twimg.com/media/B33GFtNCUAE-vEX.png:large) are rapidly starting to provide an alternative to AWS that was not there before in their ability to realize more competitively priced hosting alternatives without sacrificing on administrative overhead.

DenisM• 12 years ago

For 99% of startups the cost of web hosting is negligible next to the cost of labor. Even the tiniest gain in productivity will overshadow hosting savings.

Not valuing your time is easily among the biggest mistake you could make.

artifaxx6• 12 years ago

That is very true, time is quite valuable. It appears that for my use case AWS would be at least 5x as expensive though. I won't know for sure until I actually try running some benchmarks. Have you had experience showing AWS to be significantly faster to deploy and administer than Linode, DigitalOcean, etc?

joeyspn• 12 years ago

2 years ago I would've gone with a vanilla 3 tier app in AWS with cloud formation... Now I'd pick DO with a CoreOS cluster running Docker containers for each service... or maybe Atlas by Hashicorp [1]... No vendor lock-in, still flexible and fun to build... =)

[1] https://atlas.hashicorp.com/

albertoleal• 12 years ago

Unfortunately DO doesn't pool bandwidth.

count• 12 years ago

Security. The internal ops/security game at AWS is waaaaaay more advanced than any of their competition.

aidos3• 12 years ago

It can also totally depend on what you need, instance-wise. I have a workload that includes a lot of processing and high-memory requirements.

A 60GB r3.2xlarge spot instance on AWS will cost you about $50/month while on DO you'll be paying $640/month for a 64GB machine.

Granted, the DO machine will be way more performant, and you have to have a workload that suits spot instances.

Every time I look at things like DO to see if I'm missing out I price up my current AWS setup and they're always astronomical (for my use case).

Then there's all the other control around AWS; it's so much more advanced than all the other services. Nobody else really comes close to the flexibility they give you.

For a lot of (probably even most) web apps that probably doesn't matter. Have a look at your use case and decide if AWS is worth it.

stephenr9• 12 years ago

You're comparing apples with watermelons. The only thing vaguely similar is the memory, and it's only cheap because you get it randomly as the market fluctuates.

In every other regard - disk space, CPU cores, data transfer, availability - the AWS spot instance option is MUCH worse than a regular VPS like from DO, or even a dedicated physical host.

Ignoring the CPU core difference, a regular (i.e. so you can use it all the time) instance with extra EBS storage and data transfer to match the DO instance you mentioned, would be $1500/month.

Places like Rimuhosting will rent you a pre-used dedicated server with 128GB RAM, 2TB HDD + 200GB SSD Dual Xeon (12 cores, 24 HT) physical server (which you can then run a single Xen VM on, for better portability) for $400 a month. Yes it's more than your $50 a month, but its completely yours, for as long as you want it.

To truly take advantage of a spot instance and NOT have it impact your business, your workload needs to be very tolerant of interruptions and variable processing time per day. For the vast majority of people using EC2 (web/app servers running 24x7) it's not a realistic option.

aidos3• 12 years ago

Oh for the.... Could whoever down-voted this please have the decency to chime in with a reason?

If there's anything incorrect with what I've said, please point it out. I'd love to know about other providers that would give me more bang for my buck.

Is it the assertion that spot instances are a viable option for some use-cases? Is it the fact that I compared something to DO?

davidw• 12 years ago

I started writing a response, but then moved it here, since this seems such a common discussion:

http://journal.dedasys.com/2015/01/22/when-to-use-aws-and-wh...

tl;dr: I would not use AWS unless you know that you need what it offers.

darkFunction• 12 years ago

I'm a real newbie in terms of web server deployment, but for my first web app I went with an EC2 VPS because I had no idea what sort of traffic to expect and how I would handle spikes. Despite having a pretty decent VPS from Inception hosting (whom I recommend, no affiliation), I wouldn't know what to do to scale up.

EC2 seems to handle all that nicely by letting you just upgrade instance types as you need them, and providing burst performance for spikes in traffic. It seems like a pretty fire-and-forget solution and I've not paid them any money yet (10 days, 30,000 hits)... So far I'm happy but really I've no idea what I'm doing.

jwilliams3• 12 years ago

This links to a HN discussion on the relative merits of VPN access versus bastion/jump boxes (https://news.ycombinator.com/item?id=8637154).

This conversation didn't ever seem to bottom out to conclusion? In particular I was wondering how servers connected _outwards_ in the VPN scenario.

kainosnoema• 12 years ago

In either case—assuming you run a VPC—you usually configure one or more NAT instances to allow EC2 instances to communicate with the internet (http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NA...). I suppose a bastion could do double-duty as a NAT, but in most cases you want one per availability zone to add isolation and redundancy.

dbarlett• 12 years ago

You can also configure HA NAT with an ICMP heartbeat: https://aws.amazon.com/articles/2781451301784570

jwilliams3• 12 years ago

Doesn't this -somewhat- mitigate the advantage of having zero-surface area?

I'm using a Bastion setup, so don't get me wrong, just want to understand how strong the pros are for the VPN route.

On your last note. I just run one Bastion as a general rule. They're quick enough to spin up another instance (in a different AZ if necessary). Generally our services won't die if the Bastion or NAT is down.

maslam• 12 years ago

We (Appuri) use VPCs exclusively. There are pros and cons. I'll try to list the top:

Pros: - Logical isolation. You can put instances (and RDS, Redshift) etc. inside logical subnets that are not addressable from the outside world. - VPNs. If you really want extra security, you can wire up VPN so one of your VPC subnets shows up on your corporate subnet.

Cons: - A complete pain to manage with SSH-based tools. Most deployment tools (Ansible, for example) and even lower-level tools like fleetctl don't play well (if at all) with jump boxes. Example - Ansible Tower requires instances that are publicly addressable OR placing a Tower instance inside a VPC (which means we can't use it to manage multiple VPCs) - We have had to write our own workarounds for the above con. - Complexity. There are more concepts to learn about. - Lack of portability. I don't know if all cloud providers (Azure, DO etc.) even support VPCs the same way AWS does. This makes our infrastructure less portable than I'd like

jwilliams3• 12 years ago

I think you'll find you can solve this via ssh config. Specifically using ProxyCommand -- in the case of Ansible anyway. You can then ssh reference an internal address.

I was asking less about VPCs in general, more the use of the VPN->VPC or Bastion approach to bridge into that network.

saryant3• 12 years ago

We use fleet within a VPC and our approach is to just have a single "ops" box in the public subnet and then use $ETCD_ENDPOINT (or whatever the environment variable is) so that etcdctl/fleetctl can connect to one of the boxes in the etcd cluster.

We disable password login on the ops box and set up 2FA on SSH connections. We haven't taken the step of whitelisting IPs but it's probably something we should do.

I just finished moving our last EC2-Classic service into VPC. It's been less of a headache than I anticipated.

Rapzid2• 12 years ago

If you need a VPN, you'll know it. My advice if you don't need a VPN now is to leave a few smallish subnets in your VPC for NAT'ing if you need to setup a VPN in the future. Then, just white-list the IP's that have access to your bounce boxes. Create a Temp security group and add your home/hotel IP on demand, blow all the rules away regularly. You have essentially have a multi-factor protected white-list for your bounce boxes at this point.

The logic here being you'll only need certain things on the VPN, and you can put them or give access to them through the small subnets. You can simply use NAT to translate a /28, /29, or whatever to an available block in your corp network. This won't work for everyone, but I think for a lot of cases you don't need full access to the VPC via VPN and you'll know pretty early on if this will work for you or not.

chuckcode• 12 years ago

Great to see article putting it all together. Also going to be great to send this to people who think that using AWS cloud means that you can skip hiring people with skills in systems administration.

tieTYT2• 12 years ago

The scariest thing to me about AWS is that I might accidentally bankrupt myself while I learn to use it. I've seen horror stories on HN before. Articles titled "How I spent $32k with AWS, a for loop, and a simple typo" or something like that.

Normally when I learn something new, I learn by tinkering and breaking stuff. I don't feel comfortable doing that with AWS. I'm hoping people will tell me I'm way off base because this fear has stopped me from getting the ball rolling.

withdavidli• 12 years ago

Here's a story for reference: http://www.devfactor.net/2014/12/30/2375-amazon-mistake/

More about bots scanning github for api keys, that's pretty scary in itself because I know pushing keys by accident happens a lot.

juliangregorian• 12 years ago

Yeah, Amazon's key model is a pretty big weak point, both Google Cloud and Azure handle it better. It would be safer to use different sets of keys, like one to create new machines, and app-specific keys that can update only, but that's more work and more headache to manage. Google Cloud just makes me SSH tunnel, which I like.

skuhn• 12 years ago

AWS does have a way to warn you when your bill crosses a certain threshold: http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/...

oafitupa6• 12 years ago

All the warnings in the world can't replace a configurable $ limit. We don't want to have to rely on them "forgiving" us if we make a mistake.

tieTYT2• 12 years ago

That's great, thanks for the info. I wish most getting started guides would set this up first/early.

eropple4• 12 years ago

AWS is extremely understanding about those sorts of overruns, if you don't make a habit of it.

themartorana3• 12 years ago

I run a million-dollar business on it for $4500/m, give or take. It's not Webfaction (who I love and highly recommend) or Linode. It's virtualized data-center territory that can support some massive stacks.

They give you enough rope to hang yourself. I'm ok with that, because it works brilliantly when you have the time to put in to it. (They also almost always help out or completely forgive someone that accidentally ran up a massive bill in a short period of time.)

Without AWS, my company wouldn't exist - I could never have initially afforded dedicated hosting or collocation. AWS changed the game. You can argue they're not the cheapest or even the best anymore, but it works this way because of Amazon.

Edit: what's with the down-votes?

oafitupa6• 12 years ago

What's your point? What you said changes nothing for people like tieTYT and me.

seunosewa3• 12 years ago

You could have afforded VPS hosting in the beginning (starts from $20 per month for decent specs), and immediately switched to dedicated hosting once you outgrew the VPS (from $100 per month). AWS was never the cheapest for any given level of performance; you've always paid a huge premium for rarely needed flexibility.

IkmoIkmo• 12 years ago

On a sidenote, what's your business?

mikelbring• 12 years ago

Yep I had a $800 bill once because of this.

balls187• 12 years ago

Amazon has been forgiving for the occasional spike in billing.

If you understand AWS pricing, and don't publish your AWS keys, you should be good to go.

mdnormy• 12 years ago

A friend amasses $200 bill overnight due to bad looping when experimenting with Lambda. This is all too common of a story now. Amazon should do something other than goodwill refund.

richbradshaw• 12 years ago

This isn't true at all - the main issue that most of those articles have faced is either:

* Committing root credentials to a public git account (equivalent to your root account on every machine + ability to purchase new servers…) * Spinning up instances during free period, then forgetting and getting billed the $10/month on them a year later * Not reading the costs, and trying to replicate what you might normally purchase in AWS and being surprised at the cost, rather than buying 101% of capacity required.

I'm using it to host a mid sized website with an intranet type app with servers spanned across multiple data centres, built in redundancy etc for around $100/month after reserved instance one off payments. It's crazy cheap and in my view, incredibly powerful even for smaller businesses.

jon-wood• 12 years ago

Given the size of reserved instance pre-payments it seems a bit disingenuous to claim you're paying around $100/month. Reserved instances are definitely the way to go if you have a steady state load that warrants them, but you do have to look at that pre-payment as part of your monthly costs to get a proper view of what's being paid.

EwanToo• 12 years ago

You can really easily setup alarms on cost, start with a $50 alarm, $100, $200, etc and you'll get notified by email when you've spent that much.

runamok• 12 years ago

You can easily setup alerts and such when you go over a certain cost run-rate. Of course you need to know how to do that. http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/...

chacham15• 12 years ago

These guides are really hard to read (at least for me) because there isnt really a point (until the end) where I can stop and go try out some of what I've learned. Perhaps, if it were formatted in the way of "steps to setup a scalable web app on AWS" it would be more palatable.

Rapzid2• 12 years ago

This article appears to include a lot of very good advice(speaking as an AWS solutions architect). I might suggest a emphasising a few things such as not having keys on login accounts(they negate multi-factor auth if leaked), and to ALWAYS pick or create a new IAM role if you aren't sure an existing one fits for the EC2 instances.. But perhaps this sort of advice is not appropriate for the article.

Much respect for the amount of work that went into this. I'll try to get through it all here at some point :)

jacques_chester• 12 years ago

Missing from the list of PaaS offerings is Cloud Foundry, which already runs on AWS.

Obvious disclaimer: I work on CF for my dayjob.

AliCollins• 12 years ago

Does anyone know of a similar article for the Google Cloud infrastructure?

minikomi• 12 years ago

Amazing post! Thank you very much. Would love to be able to read this on kindle - you should make an epub! haha

skc• 12 years ago

This is a great article.

I knew I'd see a bunch of people stating that AWS is expensive and you should use a dedicated server or a VPS. But there are many applications built by people like me who are lone developers or small teams of developers who either don't have the admin skills or simply don't want to admin their own servers and the fact that AWS handles quite a lot of this for you is sometimes worth the the added cost.

lazyfunctor• 12 years ago

Nice post! Thanks for writing it. Do you know of any similar resource for google compute platform?

A Comprehensive Guide to Building a Scalable Web App on AWS

Related Stories