Designing a scalable API on AWS spot instances

https://blog.adapty.io/designing-scalable-api-on-aws-stop-instance/

by iwitaly • 6 years ago

129 57 6 years ago

blog.adapty.io

ocdnix• 6 years ago

Turns out this is about EC2 spot instances for ECS. How would it compare to ECS Fargate spot these days?

I'm also missing a discussion about designing for interruption, either by not keeping state, or by being able to shed state quickly, to be picked up by other instances.

Also, if you set up EC2 spot with a launch template or ASG with very differently-sized instance types (to reduce risk of running out), is there a way to even out the load coming through an ALB? The least-connections scheduling can help in some cases, but a connection might not map 1:1 to one unit of load. The ALB can use weighted balancing, but on the target group level. Dunno how easy it would be to allocate different instance sizes to different target groups and weigh them accordingly.

ollyculverhouse• 6 years ago

AFAIK with Fargate a lot of this is handled for you, as long as you have the auto scaling group.

We have this setup with two capacity providers (FARGATE_SPOT and FARGATE) with a 75/25% split, meaning that even if there are no spot instances available we will still be up.

The benefit of Fargate being that we don't need to care if certain instance sizes are not available as that is handled by AWS.

l33tman2• 6 years ago

Cool, when fargate launched they didn't have a spot possibility (AFAIK) and since we run ECS on Spot instances it would just be a massive increase in cost to switch to FG, but if it now can use underlying spot instances, it might be worth looking at again..

pluies• 6 years ago

(Not the OP, but running a fairly similar setup, albeit for EKS nodes rather than ECS nodes :) )

Fargate Spot is about a third of the price of Fargate (at least in eu-west-1 now according to: https://aws.amazon.com/fargate/pricing/ ); so the savings are roughly identical.

Re risk of running out, our current strategy is to use different-but-closely-similar instance groups; so for example we have an autoscaling group running a mix of:

- m5.large - m5dn.large - m5n.large - m5ad.large - m5d.large

Which are the same price on Spot instances, but I'd wager it'd be pretty rare to have all these families reclaimed at once.

(We also use some on-demand only ASGs with lower priority in the cluster-autoscaler to ensure that if it _does_ happen, then we'll have a fallback)

makkesk8• 6 years ago

"Our backend system is built on AWS. Today I’m going to tell you how we had cut costs"

This is a recurring topic here on HN and it boggles me and makes me wonder if people know that there are other platforms than aws, azure and google cloud out there that are very capable and much much cheaper.

Unless any of the big 3 has a feature or certification you need I don't see any reason to use them at all due to the insane complexity and cost.

So why do you or your company who uses any of the big 3 use them if you had to cut cost at some point?

jorams2• 6 years ago

I can give you the reason we moved from Digital Ocean to Google Cloud: Spaces (object storage) was ridiculously slow and unreliable. There were several incidents per week where Spaces had bad connectivity or simply seemed to have crashed entirely, and that's not something we can deal with for production traffic. We now run a cheap CDN in front of it to cut down on the ridiculous bandwidth costs, but Google Cloud Storage has been very reliable and fast.

saddlerustle2• 6 years ago

This. There is no alternative for high performance object storage outside of AWS S3 and Google Cloud Storage. Even Azure's offering is wonky.

Lots of providers claim to offer object storage, but try hitting them from couple thousand cores and they all tend to immediately fall over.

csunbird3• 6 years ago

is it that bad? we are thinking about bootstrapping a project, which will use spaces for file storage and we will be using them frequently.

jrockway2• 6 years ago

I run my own Docker registry for my personal projects backed by DO Spaces. It was so slow that I couldn't actually believe it, and had to instrument the registry code extensively to account for all the time spent in the storage API, to see what the actual problem was. Turns out Spaces is really, really slow.

PetahNZ• 6 years ago

Tell me who/what I should migrate my Elastic Beanstalk/Aurora Serverless/S3 stack to to save significant costs?

dijit4• 6 years ago

The solution to saving costs is to go for cheaper cloud providers or run k8s on VPS's/colocated physical hardware.

I'm not sure where the idea came from that this is hard to do.

To answer your questions in order:

1) People warned aggressively about lock-in of cloud providers with proprietary extensions and you must have chosen not to listen; so, I have little sympathy.

I'm not saying it's black and white; but I hope that you got your velocity required to hit market faster and have made more than you spent because this is the price you paid; and now to get out you'll have to invest a little time on cleaning house, that's the reality of lock-in.

2) S3 is pretty easy as there are "s3 compatible" FOSS projects; min.io comes to mind, or ceph with a RADOS plugin, or Riak with the s3 plugin... there's also s3proxy with a multitude of backends..

3) Aurora serverless is replaced by knative

4) Beanstalk is just classical servers with an auto-scaler component, auto-scaling depends greatly on your provider, so I can't say how easy or hard it will be, if you're using kubernetes then understanding your load should be easy at least.

CSDude• 6 years ago

Always this poor argument pops up. 1st AWS and big 3 is much more reliable. 2nd AWS is not only a VPS provider. Changing vendors is not the answer to all cost related problems.

gridlockd2• 6 years ago

1st AWS and big 3 is much more reliable.

I think that's a myth. People assume it's true because it should be true. I don't think it is true.

2nd AWS is not only a VPS provider.

It's a glorified VPS provider. Most of the stuff doesn't matter to most of the people using AWS, but they go for it so they can put it on their resume and because they don't want to get fired for choosing something that's not a big name.

qeternity2• 6 years ago

This is not true at all. We run a large setup in OVH Gravelines and another in us-east-1 and we’ve not had a single outage in 3+ years with OVH. I can’t say the same for AWS.

ethanwillis• 6 years ago

It's less reliable than a guy running a website on a raspberry pi with a solar panel from harbor freight http://pi.qcontinuum.com/

secondcoming• 6 years ago

It's probably because when devs jump into the cloud they can be distracted by Shiny New Toys and cost is only considered once the first bill comes in.

marcinzm2• 6 years ago

So why do you or your company who uses any of the big 3 use them if you had to cut cost at some point?

Because it takes X hours of effort to cut costs and using another provider would have required Y hours of effort where Y >> X. Y may be greater due to reliability issues, missing features that you need to build yourself, training costs of new employees, etc.

edit: Also X is paid once you're succeeding, Y has to be paid before you're succeeding which makes Y even more expensive in opportunity cost.

yellow_mixer• 6 years ago

It’s also much easier to hire engineers to develop and maintain theses platforms.

l33tman2• 6 years ago

I'm sure there are other services that are cheaper in some of the areas, but at least the argument about "insane complexity" is moot; if you've used AWS before it's no problem to set something up from scratch. I've done it numerous times and it's a one day job to setup most of the infrastructure required with VPCs, IAM users, policies, databases, scaling groups, storage configs etc.. That part of it is not really an argument to switch at least.

pm90• 6 years ago

I don't think moving cloud providers is an effective way to solve for cost problems; especially considering the engineering cost involved in making these migrations, which reduce the marginal utility of such a migration to 0, possibly even <0.

If you're really looking to save on costs, hosting solutions (i.e. hosting racks or some managed solution) are probably what you would need to look at. But then there are other costs involved there as well (infrastructure team, upfront capex costs etc.), but it might still be worth it if you run e.g. a ton of batch processing over a ton of data.

sunilkumarc• 6 years ago

Very well written article!

If someone is interested in learning all the AWS concepts, here's an awesome e-book which is written by the legend Daniel Vassallo himself.

https://gumroad.com/a/238777459/MsVlG

MrPowers• 6 years ago

Great article. Cutting ec2 costs is important, especially for companies with heavy data engineering / data science workflows. Spark service providers make it easy to spin up huge clusters (100+ nodes) to perform ad hoc analyses. The costs can quickly spiral out of control, even if you're getting 3x cost savings on the spot market.

Some tangential thoughts:

* Is there an AWS API that returns the cheapest availability zone in a region for a given instance type? Or is the GUI that's screenshot in the blog the only way to see?

* I have seen the 90%+ cost savings for certain instance types

* Sometimes you lose a spot instance, look at the pricing history graph to confirm the price spike, and don't see any spike that was above your bid price... it can be frustrating

luhn• 6 years ago

I'm not aware of any API (surprising for AWS), but you use Spot Fleet to get an array of spot instances optimized for cost.

Having a bid above spot price does not guarantee you'll keep the spot instance. AWS can terminate a spot instance at any time if they need the capacity—That's the deal. It used to be more closely tied your bid price, but they've been moving away from that.

jakozaur• 6 years ago

For service type workloads (e.g API service with 99.99% uptime SLA) we keep comparing on-demand vs. spot.

In reality, you would like to compare Reserved Instances as you can get 60% discount.

So in us-east-1:

- spot costs: 35-43% of on-demand

- RI 1 year standard: 60% of on-demand

- RI 3 year convertible: 46% of on-demand

So if you have some base load that you can commit to running for 3 years, the price gets often at spot range while not having to worry about losing capacity.

In-reality sometimes combining reserved for some base capacity 60% + 40% spot for spiky seems to be the winning combination for many companies.

kpotehin• 6 years ago

Good point, thanks! We're looking at saving plans, probably will use them too. But when you're doing prototypes it's pretty much impossible to commit to 1 year of usage, let alone 3:)

zerubeus• 6 years ago

You can still reserve classic instance for one year and when you don't use it anymore you can sell it in the market

lyalu• 6 years ago

interesting article!

ditansu• 6 years ago

Very useful article! waiting a terraform best practice

arecurrence• 6 years ago

There's an excellent implementation using AWS lambda to manage spot instances at https://github.com/AutoSpotting/AutoSpotting

What's fantastic about the autospotting implementation is:

1. It dynamically replaces existing instances with spot instances just by setting a tag

2. Rather than replacing the instance with a fixed spot instance type, it will choose the cheapest that fits the requirements.

3. If there are no spot instances available that fit the requirements, it will spin up on demand instances until spot instances are available.

If you have experience working with spot then you will know that these are really outstanding features that hopefully amazon will bake-in in the future.

WatchDog• 6 years ago

Seems like amazon has long been transitioning spot instances away from being a method of efficiently utilizing excess capacity, towards being a discounted service for less risk-averse businesses(businesses that can accept the risk of their service being terminated at any time).

Even if in practice AWS never sees large spikes in compute demand and corresponding large scale instance preemption, most businesses I've worked with won't accept the risk of having OLTP systems be taken down at any time.

No longer does spot seem to be a service where one can get a bargain for their compute intensive offline/batch workloads that are much more tolerant of preemption.

Given that the spot prices seem to be very flat, and preemption is rare, amazon presumably have a fair bit of underutilized capacity, does anyone know if amazon uses this capacity themselves, or offers more aggressive spot pricing to select clients?

Designing a scalable API on AWS spot instances

Related Stories