Introducing Preemptible VMs

http://googlecloudplatform.blogspot.com/2015/05/Introducing-Preemptible-VMs-a-new-class-of-compute-available-at-70-off-standard-pricing.html

by boulos • 11 years ago

300 85 11 years ago

googlecloudplatform.blogspot.com

gnur• 11 years ago

I wonder how big the chance is that the machine is turned off in the first hour.

I use Google Compute for some personal project and the typical run time for a VM is about 10 - 20 minutes. If the vm has 90% chance to survive the first hour, it could be worth the trouble to make my process more fault tolerable.

boulos8• 11 years ago

We're not ready to put hard numbers on it. As our docs (https://cloud.google.com/compute/docs/instances/preemptible) say:

The probability that Compute Engine will terminate a preemptible instance for a system event is generally low, but may vary from day to day and from zone to zone depending on current conditions.

Give it a shot in us-central1-a and let us know how it goes!

[Edit to undo my quote text (it wrapped poorly)]

pacala• 11 years ago

I couldn't figure out from the docs, can one restart a preempted preemptible instance, or needs to start a different one? Would it be possible to restart in non-preemptible mode, so the job completes but at a higher price? We still want to complete our workloads, one way or another :)

ikeboy• 11 years ago

How does their pricing compare to ec2 spot instances?

boulos8• 11 years ago

You can find a list of our prices here: https://cloud.google.com/compute/#pricing . Preemptible VMs have flat per-machine type pricing compared to EC2's bidding-based pricing. For example, a Preemptible n1-standard-1 is 1.5 cents per hour in the US and a preemptible n1-standard-4 is 6.0 cents per hour.

miles932• 11 years ago

Well, spot varies. Sometimes it's low, but the price varies based on demand. Using prices as of right now for example, Currently in California, c3.8xl instances are quite popular, running $1.68 an hour on an instance that's normally 1.912 an hour, roughly a 13% discount. 70% strikes me as better than 13%, especially when it's predictable. (and 70% below an already substantially lower hourly price...)

[1] http://storage.googleapis.com/pricingcomparator.appspot.com/...

[2] http://storage.googleapis.com/pricingcomparator.appspot.com/...

disclaimer: I work at Google on Google Cloud Platform

vosper3• 11 years ago

The long-run price for a c3.8xlarge in California is somewhere between $0.40 and $0.50, though you're correct that right now they're $1.68.

California tends to be more expensive than other AWS regions, though (I can't remember the reason - perhaps just availability?). If you're in us-east-1 the price sticks around $0.20 per hour, and eu-west-1 is rock-solid at $0.32 per hour.

If you have work that can be done on spots it can be a good idea to make it region-agnostic so that you can take advantage of better prices for different instances in different regions.

vosper3• 11 years ago

The fundamental difference seems to be that Google's prices are fixed, whereas AWS uses a "market" model, which frequently sees crazy prices (well over the on-demand price) especially in us-east-1.

I say "market" because no-one really knows how the spot market place actually works. We've had machines run for weeks, and other times the prices fluctuate in bizarre ways and we can't get out preferred instance types for hours or (in the worst case) days. There's an interesting analysis here: http://santtu.iki.fi/2014/03/20/ec2-spot-market/

CoolGuySteve2• 11 years ago

We had the same problem with getting priced out of c3.8xlarge in Virginia. We fixed it by changing our allocation algorithm to find alternate instance types and zones. For example, instead of 1 c3.8xlarge, it might pick 2 c3.4xlarge instances or a cc2.8xlarge. Seems to work so far.

I looked through the pricing table and played with the calculator, it seems something equivalent to our needs would cost around a third more on google but each cpu would have twice as much ram. Not worth it for us.

teraflop2• 11 years ago

I don't really see a fundamental difference except that Google's offering is more opaque and less flexible. Instead of "your instance got killed, but you can restart it if you're willing to pay $x/hour", you get "your instance got killed, tough luck."

EC2's spot market fluctuates based on supply and demand, and there's no reason to think the same forces won't apply to GCE.

vosper3• 11 years ago

@boulos, since it looks like you're involved with this: The link to the product pricing [1] goes to a generic landing page for Google services, which is a little confusing. I think you should link directly to the instance pricing [2].

[1] https://cloud.google.com/pricing/#compute-engine

[2] https://cloud.google.com/compute/pricing

boulos8• 11 years ago

Yeah, as part of our larger pricing announcement we're making sure people go to /pricing so they get a bigger picture before diving straight to the table ;)

[Edit: Point taken though]

andybak• 11 years ago

Any reason you couldn't use these for conventional web stuff? 30 seconds could easily be long enough to bring up another instance and sync data if needed.

CydeWeys3• 11 years ago

By "conventional web stuff", do you mean serving a website or similar? Yeah, I don't think it makes sense to host a website on a service that is preemptible. You need your webserver available to respond to requests 24/7. It can't just randomly go down. Pre-emptible instances are more suited for batch processing and big data computation runs.

comex• 11 years ago

If you have a website that can be distributed among arbitrarily many frontend servers, it could make sense to put them on preemptible instances, with the database on a regular instance. (In the unlikely case no preemptible instances are available, you could always automatically switch the frontend servers to regular ones.) However, I think this could only possibly be efficient pricing-wise if your traffic was extremely bursty.

spankalee4• 11 years ago

No reason at all. If you have enough preemptible VMs, and they spin up fast enough, then when one goes down, just spin up another. If for some reason GCE's overall preemptible spare capacity drops and you can't get a new one and it's important to you, then spin up a regular VM.

ErikRogneby• 11 years ago

I would say it depends on the size of your pool. 30 seconds _is not_ long enough to bring up another instance.

hillsarealiv3• 11 years ago

EC2 spot instances give 2 minutes notice...

sologoub• 11 years ago

You'd have to make sure that if a node gets terminated while processing a request, that request can be re-routed to a live VM. Unless you don't care about users randomly seeing errors of course.

giulianob• 11 years ago

They give you 30 seconds to shut down. Should be plenty of time to gracefully end your current requests and shut down the web server.

tedchs• 11 years ago

That could be an interesting model if your web app is a single-page / static app served from Google Cloud Storage, with Javascript making AJAX calls to your backend and configured to gracefully handle transient errors as servers come and go.

jxf• 11 years ago

This is a great announcement. Preemptible VMs are a great strategy if:

    * you don't care too much about when a workload finishes
    * it can be chunked into small units of interruptible work
    * the cost of resuming interrupted work is zero or small

Examples of workloads that might look like this:

    * geocoding large batches of addresses
    * map-reducing large event streams
    * content scraping/crawling

Notice that you don't need to run all of your cluster as preemptible if you're using bdutil [0], too -- some of the workload can be preemptible, and some not. So you can guarantee a minimum processing throughput, but get extra throughput for very cheap.

I think that's a great way to do it, though I wish it was generalized for all kinds of clusters, not just for Hadoop ones.

[0] https://github.com/GoogleCloudPlatform/bdutil

boulos8• 11 years ago

Yeah, I added it to bdutil since it was clearly useful and yet well understood that it would do the right thing. I'd love to see more cluster and machine management tools deal with "machines can come and go".

The guys on the go team are using it for their buildbots (http://build.golang.org) which was already prepared for the box to die.

mark_l_watson• 11 years ago

Nice! This ties in with the discussion yesterday on HN about data center utilization. Good for saving money and the end result must be lower global energy use if this also catches on with other providers (AWS already offers something similar).

When I worked at Google in 2013 part of my job was running very large calculations that were often pre-empted. I usually ran at the lowest priority and was in effect using spare capacity. No hassles, assuming that getting runs completed was not too time sensitive.

GreyTheory• 11 years ago

Could you point me at the datacenter utilization article? I can't seem to find it, and it sounds interesting

branola2• 11 years ago

Any developer who was burned by App Engine pricing in the past is likely to take the lure of a "70% off" discount with a grain of salt because of the much higher cost of incorrectly relying on Google's pricing pledges.

In case anyone forgets, the pricing changes on Google App Engine caused many developers to abandon apps they had developed on the platform because of hundreds of percent increases in pricing:

https://groups.google.com/forum/#!topic/google-appengine/rSh...

spankalee4• 11 years ago

App Engine was in preview when its price changed.

branola2• 11 years ago

It was in "preview" for 3 years.

During those 3 years no warning was given that the pricing changes were going to be of such a magnitude.

As a result people had sufficient time and not enough warning to build entire businesses on an infrastructure that they later had to abandon.

That's unforgivable.

CydeWeys3• 11 years ago

That goes back to 2011. Has anything been changed on people more recently?

asah• 11 years ago

yes: http://techcrunch.com/2014/03/25/google-drops-prices-for-com...

I run a $MM enterprise business almost entirely on GAE/python with a staff of ~40, public-facing site, etc and the monthly bill is under $1500/mon. Sure, I'd prefer lower $ and faster performance but truthfully, I'm no longer complaining: GAE/python saves me $$$ in IT staffing costs including security upgrades on dozens of packages that are either pre-integrated or I don't need at all (SQL & NoSQL databases incl multi-DC failover, memcache, reverse proxy, email hosting, auto-scaling, etc. etc.)

csense• 11 years ago

I can immediately off the top off my head think of no less than four examples of useful services Google has discontinued or EOL'ed: Google Reader, Google Code, Google Wave, and XMPP support for Google Voice.

So as well as potential future price hikes, you should be prepared for the possibility that this service might not be around for the long haul. Therefore you should skip using any Google-specific functionality, and instead implement a design that allows you to easily migrate to another VM vendor.

Klathmon• 11 years ago

I never got the "google shuts everything down" mentality that has become so prevalent.

I could easily beat those 4 with a multitude of examples from both Apple and Microsoft, that doesn't mean that any of them are untrustworthy, just that they evolve and continue to grow.

At least when google shuts down a service, they give a good amount of "heads up" to those using it, provide examples of trustworthy equivalent services from competitors, and ALWAYS provide an export function if it makes sense to have one.

pekk2• 11 years ago

You're right, though I wouldn't compare App Engine to things like Google Reader. When Google EOLs a highly-visible free service like Reader, it generates a lot of outrage, but consumers can find a replacement like Feedly at reasonably low cost.

The risk of writing apps on a platform like App Engine is far greater because people tend to spend years building software and a business tightly interwoven with the platform, only to be shut down by unpredictably massive rate hikes because you basically have to rewrite to get off the platform, not just the app itself but all your ops stuff too.

While it is possible to dig yourself into that hole using lots of Amazon services, Amazon doesn't require you to use platform-specific APIs, many of their services like EBS are very easy to replace, and Amazon's services have not been subjected to the same project-ending rate hikes. In practice, people move on and off EC2 all the time. So if you are going to trust a vendor, it's more reasonable to trust Amazon.

biot• 11 years ago

Amazon has discontinued Flexible Payment Services, Amazon Webstore Services, and others. Presumably your warning to not rely on proprietary implementations is vendor-agnostic?

kbenson• 11 years ago

That thing that jumps out at me from that example data set is that they were all free services. That's not to say there aren't fee-based services that Google has discontinued, or that your point isn't valid, just that your provided data points don't seem entirely relevant to me, given the subject.

abritishguy4• 11 years ago

Before the pricing change app engine did not support concurrent requests to a single instance for python. As a result, applications used many more instances than they needed.

When using the calculator this would make the price increase seem enormous to many customers. When concurrent request support was added (it was available to trusted testers at this time) all a user would have to do was add "threadsafe:true" to the app.yaml file to enable it (assuming their code wasn't doing anything silly).

jontro• 11 years ago

Deploying to app engine caused us big pain in the long run. I just had to spend two days migrating a legacy app to the new hrd. The migration was painful, app engine "billing bugs" when upgrading etc made the process real hard.

Also when the bugs occured, I realised there was no way to get quick support.

Thankfully we moved our main app away to ec2 around when they increased the pricing a couple of years ago.

4k• 11 years ago

I might be missing something important here. But according to their pricing page[1], a preemtible VM with 30GB memory costs $86.4 a month.

Why would someone go for this when cheap dedicated host providers like hetzner etc offer powerful dedicated servers with 64GB memory and multicore server grade CPUs? The comparison only gets worse taking into account that Google's offering is preemptible and can shutdown and come up as they wish.

[1] https://cloud.google.com/compute/#pricing (0.12x24x30)

CydeWeys3• 11 years ago

It's about scaling. Google's offering is a cloud service and Hetzner's is not. You're always paying the same amount per month with Hetzner, whereas you'd only pay that much with Google if you have consistent full utilization (which seems rather unlikely; most workloads do not look like this). Secondly, if you create something that becomes popular, you can scale it up on GCE a lot more easily than you can with a dedicated hosting provider.

rwmj• 11 years ago

It's like asking why do people use Serviced Offices which are usually 10x the price of a monthly rented office. Answers: because they only need the office / VM for a few hours. Because they can immediately get an office / VM of any size they need. Because they can walk away from the office / VM when they don't need it any longer. It's certainly not for everyone, but Regus seem to be doing ok.

vidarh2• 11 years ago

And for those who truly only need it now and again, it's great.

But time and time again I see infrastructure where people pay for these services for large amount of instances that are running continuously, blindly assuming that it's cheap because it's cloud. There's a bizarre level of price-blindness amongst certain subset of customers of Google Cloud and AWS that I've never seen anywhere else.

noelwelsh• 11 years ago

APIs, instant access, massive scale, no setup fees, and value-added services like DBs and message queues are all reasons to use a cloud provider.

jedberg• 11 years ago

Using the cloud as a rented datacenter is never a good idea. It will certainly cost you more.

If you're going to use the cloud you have to do it right, and that means auto-scaling and variable resource usage. Then this option will save you money.

lnanek2• 11 years ago

Google's cloud pricing is really terrible. Other competitors are intentionally driving prices down to dominate the market, so Google's prices kind of come across as a joke.

PeterisP• 11 years ago

If I want to run a processing job that takes a 1000 core-hours and have the results today, I can do that on their VMs for a couple dollars per run. Being able to do that on owned hardware or dedicated hosts would be orders of magnitude more expensive.

Similarly, if you're running hosted app that on most hours doesn't overload a single server, on peak hour fills three hosts, but on a large advertising event or accidental viral link takes fifty hosts for a day, and then goes back to normal - then you don't want to run it on VMs where you have to pay for them by month.

vidarh2• 11 years ago

That's true, but those are extreme niche use cases, and in most cases people have a substantial base load that they can run on dedicated servers for anything from 1/2 to 1/3 of the cost of AWS/Azure/Google (even with reserved instances and factoring out retainers for someone to handle ops issues).

Nothing stops you from mixing and matching dedicated servers with handling batch jobs and peaks with cloud servers. In fact, most data centre providers can offer the full range from unfurnished colo space to cloud offerings out of the same data centre these days - either directly or via partners hosted in their buildings. At least that's my experience.

tw042• 11 years ago

So... this is essentially first come, first served based on their description. And they say the preemptibles come from a smaller pool of resources. Seems odd to me they wouldn't have a number somewhere of how many are available and how many are in use given that it's a finite resource.

Tough to build a business model around a resource you can't even determine the availability of.

nightcracker• 11 years ago

Though to build a business model around a resource you can't even determine the availability of.

That's exactly why it's cheap: you're trading price for reliability.

tw042• 11 years ago

reliability != availability. I can build a business model around an unreliable, but available resource. I can't build a business model around an unavailable resource. They're two very different things, and the distinction is important.

boulos8• 11 years ago

You're right that it's first come first served, but the 24 hour time limit combined with natural churn means you aren't likely to be "locked out" for long periods of time.

h43k3r• 11 years ago

Now I am waiting to see a similar thing in Azure.

Preemptible VMs will prove to be very useful in fault tolerant Distributed Networks.

One more usecase that I can think off is Load Testing on a large scale ( in a distributed way ).

burner21926• 11 years ago

This is awesome. Thank you Google for another great addition to hassle-free cloud computing.

SEJeff• 11 years ago

And this is where a tool like mesos + aurora/marathon really really shines. The same can be said for the likes of kubernetes.

deadbunny• 11 years ago

Could certainly be interesting for cheaply expanding Mesos slaves to cope with a workload spike.

boulos8• 11 years ago

As we put in the docs, to try this out:

  gcloud compute instances create my-vm --preemptible --zone=us-central1-c

or just set the preemptible bool to true via the API (it's under scheduling). When you want to test how your system behaves on preemption, you can just do:

  gcloud compute instances stop my-vm --zone=us-central1-c

which will give you the same 30 second timeout as when you're preempted. Most OSes have a fairly standard set of things they do on shutdown that will at least send all your running processes a signal (via kill), but if you need to add your own you can inject it via the new shutdown script support (https://cloud.google.com/compute/docs/shutdownscript).

We tried to cover this in the docs (https://cloud.google.com/compute/docs/instances/preemptible) any feedback on that would be welcome!

Tegran• 11 years ago

The 1970s just called. They want their batch processing back. :-)

bitJericho• 11 years ago

At a penny per core per instance hour the cost is about 7 dollars a month. I can get a normal dedicated VM for cheaper... What's the benefit in this?

jpatokal2• 11 years ago

What's the benefit in this?

You're not buying a dedicated instance for a month, you're renting it as long as you need it. If you only need a core for an hour, your bill will be $0.01.

Introducing Preemptible VMs

Related Stories