Farewell to the Era of Cheap EC2 Spot Instances

https://pauley.me/post/2023/spot-price-trends/

by ericpauley • 3 years ago

254 124 3 years ago

austinshea• 3 years ago

This seems to ignore the fact that the prices are consistently low if you are tracking the lesser used instance types/regions, even when talking explicitly within AWS.

Why would that change?

ericpauley7• 3 years ago

It's tough to know for sure. Many of the instance types in these regions are at the price floor, so there could be increases in aggregate demand that just don't show up because the DCs are overbuilt. These markets are thinner so if demand shifts to them (big if because moving regions is hard) we might quickly see price spikes.

cyberax3• 3 years ago

Spot prices don't directly reflect capacity. EC2 Spot used to be a real spot market, with actual real auctions where higher bids displaced lower bids.

But it was changed a while ago, so prices are now set algorithmically based on predicted demand/supply.

We run stateless calculations on EC2 across regions, and we definitely see that instances are harder to come by. Especially instances with GPUs. And for many instance types, the price advantage of EC2 Spot compared to committed spending is not significant anymore.

MichaelZuo2• 3 years ago

Huh, do you remember what the rationale for the change was?

epberry• 3 years ago

I know at AWS they drill into folks that spot capacity isn’t guaranteed capacity. I didn’t even think about more demand for spot reducing supply or additionally AWS just simply not investing in as much hardware to accommodate spot because of lack of growth. That said, we're seeing leading indicators that this trend will reverse in as cloud growth returns, https://www.vantage.sh/cloud-cost-report/2023-q1

ericpauley7• 3 years ago

The q1 on-demand metrics are definitely very interesting!

One possible cause of this trend is actually still consistent with spot demand growth in aggregate: when customers deploy spot fleets with on-demand backup capacity and all spot prices get closer to on-demand, those fleets would become more likely to revert to on-demand, rather than a more expensive pool of spot instances.

Measuring this would be a great opportunity to leverage the vantage point that your team has!

alien_5• 3 years ago

Except for Karpenter and EMR, no other AWS services (including Spot fleets) failover between Spot and OnDemand when losing Spot capacity.

You simply run Spot from other instance types, and if no Spot capacity is available you run at reduced capacity.

Alternative solutions such as AutoSpotting.io are doing it and allow you to run with failover capacity.

kenhwang• 3 years ago

Even on-demand isn't guaranteed capacity. There have been a handful of times in my career where I've tried to spin up a lot more instances and got met with an out of capacity error. There's a special kind of reserved instance for guaranteed capacity.

JoshTriplett• 3 years ago

Likewise; I've tried to spin up c6i.24xlarge instances and gotten an error that there aren't any more available in the region.

nijave• 3 years ago

A few years ago we ran builds on m5d instances (the ones with local nvme) and they'd pretty regularly run out of capacity in certain AZs (although it was usually only a single AZ and there'd be a slight delay before the ASG grabbed one from a different AZ)

hn_user2• 3 years ago

Exactly this. Our group regularly maxes out instance types in a region for some of our simulations. Have to overflow the requests over to other regions in those cases. Unfortunately the load is not consistent enough to pay for the reserved capacity.

lflux• 3 years ago

It's happening to us on the reg with m6i SKUs in us-west-1 the last couple months.

electroly2• 3 years ago

You have the option to get On-Demand Capacity Reservations now which are separate from RIs. Capacity reservations don't require 1- or 3-year commitments like RIs do. This can be used if you want to get your savings through Regional RIs or Savings Plans instead.

uluyol• 3 years ago

It would be a bit odd to build capacity for spot instances. The whole point of low priority, preemptable instances is to fill up the capacity that you have available, but have earmarked for scheduling big jobs or handling failures (say a power domain or decent chunk of the network goes down).

Otherwise they would be no different from any other instance type.

kmeisthax• 3 years ago

Damn it, there goes my plan of using spot GPU instances to train AI cheaply.

jeremyjh• 3 years ago

I was wondering if this could be a contributing factor. There is almost unlimited demand for training models - and a lot of value to derived from doing so - and it seems like a perfect workload for spot.

Zenst• 3 years ago

Seems like such jobs would be suited to a batch submit type service? Does anybody offer that as does seem a viable area to offer if not.

dougSF70• 3 years ago

I have just purchased 5% of a data center's Deep Learning servers. 10 GPU / 384 GB of RAM, 2 x Xeon 10-Core processors. I am putting them in a solar powered data center and plan to lease them out at $0.2 per GPU hour.

Havoc• 3 years ago

Check out tensordock

paulddraper2• 3 years ago

Spot instances are underpriced, as most tenants underestimate their tolerance for preemption or overestimate its likelihood.
In reality, instance preemption is rare in most instance families

I have often seen the following:

1. The demand for spot instances increases for a period of time.

2. Supply is inelastic so prices quickly rise to the full bid amounts.

3. Most users are unwilling to go hours without any spot instances running, so they use very high bids.

You either have to tolerate hours of "down time" every few months, or pay exorbitant prices during once in a while.

On multiple occasions, I've seen the spot instances priced significantly higher than on-demand.

EDIT: Apparently AWS changed the Spot pricing a few years ago though. My anecdotes are 5 years old.

vasco• 3 years ago

I've seen the spot instances priced significantly higher than on-demand

How is the cap price not on-demand cost? Why would you not just swap to on-demand at the turning point?

idunno246• 3 years ago

there's a cost to engineer something that automatically switches or someone going in and manually changing it. so the spot prices has to be higher than ondemand + switching costs. the new pricing models(a couple years old) though have mostly alleviated this

paulddraper2• 3 years ago

Because you don't want to be pre-empted.

jacurtis3• 3 years ago

This shocks me too. When I first had to design a system around spot instances I assumed it caps out at the on-demand price. But in practice that is very much not the case. The spot price can routinely go above on-demand, with all the downsides of spot still being applied to the instance.

It is because many systems are only built to support spot. When running a workload, they might not want to give up availability so they just pay the higher price. To be fair, this is a game that AWS engineers. They essentially want to kick people off the infrastructure to free it up for other uses. So this is a way to signal how badly you don't want to be kicked off. A lot of times after a price goes above on-demand pricing, it will dip down below fairly quickly as workloads all re-adjust. Some companies are willing to play that gamble, that a short period above on-demand is still cheaper in the long run when you average out the invoice at the end of the month.

psanford• 3 years ago

What is the reason anyone would pay more for a spot instance than an on-demand instance?

great_psy• 3 years ago

Maybe poorly architected software, where it’s cheaper to increase the value of a variable than have a bunch of API calls that check the current onDemand price and call different APIs depending on which number is smaller.

sokoloff• 3 years ago

If an instance on-demand is, for the sake of easy discussion, $1.00/hr and the spot rate is usually $0.30/hr, it might make sense to bid $1.25/hr (or $1.26/hr) if you didn’t want to get pre-empted.

Most hours, you’d pay something close to $0.30/hr. Some hours, you’d pay over a $1.00/hr, but you’d save money overall against on-demand.

(This ignores Reserved Instances and Savings Plans.)

ericpauley7• 3 years ago

Interesting. While my data does show this being the case in a handful of pools (1-5 out of 13k pools at any given time), it appears to occur only exceedingly rarely.

jacurtis3• 3 years ago

To be fair, its also easier than ever before to use spot instances.

If you set up an EKS/K8s cluster and install Karpenter. You can configure it very easily to use spot instances anytime prices are available for less than on-demand and to use on-demand when spot instances are unavailable or too expensive.

You end up never thinking about it but having full availability.

In practice it means the ceiling on your bill is the on-demand price, but you usually average out much lower.

I suspect as more and more people switch over to this model then the use of spot instances will stay closer to full saturation, with those discounts becoming negligible.

nickjj2• 3 years ago

I think this comes with the limitation that your workloads might be terminated without your control?

They have a whole section in their docs about this: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-int...

It's all about how your spot instances can be terminated. Perhaps this could interfere with a workload that might take a long while to finish.

On paper your app should be resilient to these things but in practice it's essentially an unforgiving chaos monkey that will delete servers from your cluster.

If you can handle that, you can definitely save a lot though. Especially web servers where it's expected you'll be responding in a few seconds while maybe your background workers are churning through longer lived tasks. The problem there is you need to already be big enough to have different nodes handling web vs worker traffic. A lot of decently sized apps can operate in a world where maybe you have 2-5 nodes on your cluster and your web and worker pods land on any of them based on Kubernetes' default scheduler.

andrewstuart5• 3 years ago

EDIT: if you really need GPU compute power, go buy a GPU from the computer store around the corner - they are cheap, fast and available.

I gave up on ec2 when they started requiring you "request for quota" to start a gpu instance.

You have to "request for quota" even if you want to run a single instance.

You have to specify which specific instance type you want.

You have to specify which region you want it in.

Then you sit back and wait for some human to "approve your request".

In my case this took 24 hours.

In this time I literally could have walked (not driven) down to the local computer store, bought a computer, pushed it back to my house in a shopping cart, spent a few hours configuring it and still be left with 16 hours to have a sleep, eat and do some other things.

AWS quota system is so far from "scalable" and "elastic" that it's effectively useless. You can't design any sort of infrastructure around that sort of quota system.

I dumped AWS at that point. Mind you, Azure has exactly the same quota system.

Seriously, just rent a server from Ionos or Hetzner or get a fast Internet connection and self host. It's faster and better and cheaper than any of the big clouds.

toomuchtodo• 3 years ago

We’ve come full circle. Cloud providers have turned into dedicated server providers.

andrewstuart5• 3 years ago

Also, I felt that Amazon was playing sketchy little games with spot instances.

It's not impossible but quite technically challenging to find out how much you are actually paying for a spot instance.

It felt really dodgy to me that I might have started a spot instance thinking I was paying the minimum listed rate but somewhere amongst the formulas and terms and conditions AWS actually decided I would be paying maximum rate.

They don't actually tell you up front what your spot instance are costing, no doubt it would not be hard for them to do so, but it seems a deliberate strategy to hide this information.

It's just not worth the game playing when self hosting or renting a server from Ionos is cheaper and takes away the uncertainty.

KyeRussell3• 3 years ago

Comparing with self-hosting or renting a server is just dishonest. At that point you may as well compare with spinning up an actual EC2 instance. Your alternatives miss the main original value proposition of EC2 altogether, let alone spot instances.

ericpauley7• 3 years ago

I think it’s tough to blame Amazon here. p4d server hardware costs around 100k and they were almost certainly having countless hours of use on these by stolen aws accounts and credit cards. This doesn’t excuse the requirement to specify region and instance types in advance. The region part at least can be explained by the fact that Amazon tries to make regions as independent as possible.

my123• 3 years ago

I gave up on ec2 when they started requiring you "request for quota" to start a gpu instance.

Sadly, the scourge of crypto mining combined with credit card fraud makes it really hard to do otherwise for any sizeable hoster. :/

Negitivefrags• 3 years ago

Maybe they should let you bypass the quota system if you pay in advance via bank transfer then?

jacurtis3• 3 years ago

Wow, never realized that that could be a great money laundering strategy.

1. Take cash, load it onto prepaid cards.

2. Buy GPU instances.

3. Mine crypto.

4. Pay for compute with #1

5. Sell crypto

6. "Clean" cash

I just don't have the mind of a criminal I guess, it sounds like this was figured out a long time ago.

There are (were) enough random kids in thier basements making millions on crypto that it probably wouldn't turn too many eyebrows either by the IRS if you actually tried to report your crypto earnings cleanly, they would likely spend their efforts chasing down the people who weren't reporting.

dragonwriter• 3 years ago

Google Cloud does this, too, for GPUs on their compute instances, but the approval (I assume if there is manual approval, there are conditions which trigger it that I missed) was near instant in my experience.

carbocation• 3 years ago

This is oddly downvoted, but it is accurate.

When I use university-linked accounts, I get insta-approved for almost whatever I want. (2,048 spot T4 GPUs? Go for it.)

When using my personal CC-backed account, my max quota is often an order of magnitude lower.

ajb• 3 years ago

I can understand why though. One employers first use of a GPU instance was when we were hacked and someone fired up a few to mine crypto, only a day off which cost us a few $1000 that, fortunately, AWS refunded. It's quite likely that most users don't use them and that it is a good signal that you've been hacked.

They have a quota system for sending emails too, and it's not because they need to purchase any hw for sending more emails. It's because those are also a magnet for hackers.

jldugger• 3 years ago

AFAICT, its more that they can't keep GPUs in stock. Even large users of GPU have to request quotas in new AZs.

andrewstuart5• 3 years ago

If AWS was really clear about what you were spending then it would be easy to be running an app that tells you the AWS usage and alerts you of anomalous patterns.

My bank contacts me if there are any questionable transactions.

Surely AWS is capable of this?

late2part2• 3 years ago

It's okay for an infitely burstable product to delay provisioning services by 24 hours and require you to capacity plan in specific detail before you can use things?

I don't agree. Their reality distortion field is fooling customers, AWS has their cake and eats it too. Try being a startup and spin up 1000 c6in.8xlarge for a 2 hour batch job.

dylan604• 3 years ago

and still be left with 16 hours to have a sleep, eat and do some other things.

I hope somewhere in those 16 hours you bother to return the shopping cart. Or are you one of those that just leaves it where ever because you can't be bothered? I will be bringing this up at the next tenant's meeting.

SilasX3• 3 years ago

Do they still wait until the very last moment before launching to tell you that the account you're logged in under doesn't actually have permission to launch EC2 instances?

http://blog.tyrannyofthemouse.com/2016/02/some-of-my-geeky-t...

hcarlens• 3 years ago

Having your own GPU is nice, but it's power hungry, heats the room, and only really makes sense if you use it a lot.

For a list of cloud GPU providers with rough price comparisons, I created this page: https://cloud-gpus.com

Nextgrid3• 3 years ago

Having your own GPU makes sense if it works out cheaper than the cloud equivalent (not just the GPU bit but everything else - bandwidth/storage fees, etc).

Given the prices of cloud infrastructure, it doesn't take much before walking to your local computer store and buying a GPU (or more!) becomes more cost-effective.

coliveira• 3 years ago

AWS is following the same path as other web technologies offered by big tech. In the beginning, they want to lure as many people as possible to their new, shiny product, so they offer irresistible prices. The next step, which is the important one, is to secure corporate and government contracts for that technology. Next thing you know, they will see small clients as a nuisance, and do everything to make their lives more difficult.

lucb1e• 3 years ago

from the computer store around the corner

They closed last year, it's now a 50-minute bus ride and then ~10 minutes walking (one way). Apparently we were one of the last customers to buy most parts for a desktop there. I fear we may not be able to refer to corner stores much longer if they're not of a type most people actually need on a weekly basis, with things moving to online sales

In my case this took 24 hours.

Even shipping is faster than that, so yeah point still taken

malux85• 3 years ago

24 hours? You were lucky! When I requested approval it took a week

KRAKRISMOTT• 3 years ago

Pray tell where exactly are we supposed to find a cluster of A100s available at a reasonable price? RadioShack?

sabalaba• 3 years ago

Lambda GPU Cloud is extremely cheap as well (1/4 the price of Amazon) and we have recently added lots of capacity.

Cpoll2• 3 years ago

and still be left with 16 hours to have a sleep, eat and do some other things.

Well there's your problem. You don't need to sit back the whole time while you're waiting for approval, it's okay to leave your chair.

Jokes aside, I think this probably depends on use-case. You can't easily return the computer. And you can't easily buy (and then return) 100 computers if you have a large one-time computation to run.

milesward• 3 years ago

The analysis ignores the basics: AWS controls the supply. At some point there’s a limit to that control via supply chain (meaning they wouldn’t be able to make it cheaper when they want to) but if the number is going up, it’s pretty likely that’s totally intentional.

ericpauley7• 3 years ago

"Ignore" is a pretty strong word here. This is an analysis of demand. Of course Amazon controls supply, but allowing compute to sit idle at these price points is not profitable so we can assume that they are selling available compute. The question then is whether aggregate instance demand is increasing with respect to available supply and pushing prices up.

You'll have to explain what you mean by "intentional" given raising prices above market equilibrium would leave compute idle (again, very unlikely). In that way amazon is just a market actor (albeit a large one) just like customers.

moomoo114• 3 years ago

I stopped using cloud hosting until _it is actually necessary_

I have two home workstations that are collecting dust. I have 6 and 16 core CPUs with HT hooked up to 32 and 128gb ram.

I figured out how to make my dynamic IP still serve requests accounting for IP changes using a background script. If someone has a better solution I’d appreciate that.

Both these machines are more than enough to do everything I could dream of building and they host dozens of my little apps and services that I use for myself.

I will only use cloud when necessary for commercial product and services.

tecleandor• 3 years ago

  - Rent a cheap VPS with an static ip address (for example, the most basic at Vultr)
  - Set up wireguard or tailscale/headscale (or your preferred VPN) both on the VPS and your workstations, so you can access the services in your workstations from the VPS through the VPN
  - In the VPS configure Nginx/traefik/caddy (or your preferred reverse proxy) pointing  to the services in your workstations

Ta-da! Now people access through your VPS, but the processing is done at home. The VPS is cheap, and it doesn't matter if you home IP address changes. You could even take your workstation, move it to any place with internet connection, and it would keep serving pages without changing anything.

moomoo114• 3 years ago

Whoah! That's awesome! Thank you for making my weekend better :D

I'll try setting that up over the weekend. Just wondering - would you recommend using your suggested setup for a "production" app?

Considering the hardware I own is powerful enough to host postgres + a half dozen services to run the app including a bunch of real time processing. If my calculations are correct I should have enough resources to handle a few thousand or more concurrent users, and at an old gig we were able to service millions of DAU with even less power. As long as I can meet <500ms client request times I think it should be fine, considering there's gonna be extra network hops.

kobalsky2• 3 years ago

I figured out how to make my dynamic IP still serve requests accounting for IP changes using a background script. If someone has a better solution I’d appreciate that.

cloudflare costs $0, scales up to your gigabit connection without blinking, and there are docker containers that keep the IP updated like it if were dynamic dns service.

moomoo114• 3 years ago

Thx. Is there a specific name for this offering? Sorry first time using cloud flare lol, just digging around.

Zetice• 3 years ago

Yeah but welcome to the era of steeply discounted savings plans. Isn’t it like half of the cost of in demand, or something kind of crazy?

Had a couple of calls with their “please don’t leave us” team and they cut our bill more or less in half (more through fixing my boneheaded designs though).

kbumsik• 3 years ago

I notice the spot cost has grown up for 3 months recently, especially in U.S. regions, while some other regions are not affected at all.

Why is it happening to the U.S. regions especially?

arecurrence• 3 years ago

There's a service that handles obtaining the cheapest spot instances to your specs by a former aws engineer https://github.com/LeanerCloud/AutoSpotting

Effectively, it starts up cheap spot instances (based on specified criteria) across a variety of instance types to replace whatever regular instance in an autoscaling group comes online and then spins down the regular instance.

EG: That m4a you wanted may be expensive... but nobody is using m4ad so it's 85% off and it meets the specified CPU/RAM requirements... auto spotting will spin it up instead.

Having used it on and off over the years it is sometimes eyebrow raising to see 4xl boxes running cheaper than the xl box they replaced :)

shmoogy• 3 years ago

I use spotinst and spot instances have been noticeably more expensive the past two months :(

mnutt• 3 years ago

It works, but it may depend on how resilient your workload is. We have timed spot instances that come online shortly before the top of the hour in the morning eastern, and via specifying instance alternates we'd often get instances that were much larger than our low-ish bids would warrant. Then most of them would get pre-empted <2 minutes later and there would be a bunch of churn as we had to replace them with what we should have gotten originally.

alien_5• 3 years ago

The author of AutoSpotting here, glad to see it mentioned in here and happy to help clarify such issues.

Are you sharing this as an experience of using AutoSpotting or other mechanisms to launch Spot instances?

What you described may still happen occasionally and it happened a lot with older versions, but the recent versions of AutoSpotting, especially the commercial edition available on the AWS Marketplace (although the community edition available on GitHub also does it to some degree) should avoid such situations in the default configuration should be much more reliable than before.

Actually when it comes to capacity, when Spot capacity is not available and we failover to on-demand, the latest AutoSpotting version will failover between multiple instance types with the same diversification used for Spot instances so you're pretty much guaranteed to have capacity even better than your initial single-instance ASG.

We also use the capacity-optimized-prioritized allocation strategy with a custom priority based on lowest cost, but also preferring recent instance types if available for more performance and lower carbon emissions.

Let me know if you have further questions about AutoSpotting, Im happy to help.

pachico2• 3 years ago

The article fails to recommend to use re-balancing feature that allows you to run spots in production without worrying too much.

It doesn't mention either that price-capacity-optimized allocation strategy.

If you want to run spots in production, please read about three two. There is no major drama if you apply them.

ericpauley7• 3 years ago

The point of the article is pricing, not preemption. The strategies you’re talking about improve preemption but they won’t save you money if pricing is increasing across the board.

pachico2• 3 years ago

They do as they might be the reasons why you can use spots in the first place

jnsaff2• 3 years ago

Are the tables in the post generated by ChatGPT?

Most of the percentages are just wildly incorrect.

Farewell to the Era of Cheap EC2 Spot Instances

Related Stories