Pinterest Cut Costs from $54 to $20 Per Hour With Automatic Shutdowns

http://highscalability.com/blog/2012/12/12/pinterest-cut-costs-from-54-to-20-per-hour-by-automatically.html

by autotravis • 14 years ago

197 71 14 years ago

dclusin• 14 years ago

$54 per hour for what? Is this simply the cost of running the instances? It seems to me like pinterest is throughput bound and the vast majority of their costs would be accrued via transit and storage charges.

aioprisan3• 14 years ago

I agree with your questions. I'd be interested what percent that signifies versus the total per hour bill for all services, not just CPU time, but bandwidth as well

jackowayed• 14 years ago

For now they really only saved $300k/yr, which is less than the cost of two engineers. Though I guess cutting a nontrivial expense that is likely to grow by nearly a factor of three is pretty great and probably worth investing in.

That said, this adds complexity to their systems with the only benefit being cost savings. Given that we can assume that no code is perfect, it's likely that at some point the auto-downscaling will cause an outage or period of slow responses, which could easily lead to lost usage and trust that costs them as much as they're saving on ops.

pdog• 14 years ago

Is cost savings really the only benefit? It seems to me that this is a better engineering outcome.

jspthrowaway2• 14 years ago

That depends on for who. For Amazon, perhaps.

sliverstorm• 14 years ago

IMO designing systems that can be powered on and off regularly (and quickly) is a good thing in itself; it encourages "proper" setup. I've found servers that have been on for months or years tend to need manual intervention after a reboot. When the machine could reboot every day, you can't have that.

In other words, it's just good design.

calpaterson• 14 years ago

Nitpicking: but it's not good "in itself", it just has other benefits.

jacques_chester2• 14 years ago

> IMO designing systems that can be powered on and off regularly (and quickly) is a good thing in itself

Despite the fact that, in theory, a mainframe should never go down, most dinosaur pens will power cycle them regularly just to see what happens if you come up from a cold start.

A mate of mine worked in a dino pen where they did this on Saturday evenings. He told amusing stories.

jedberg4• 14 years ago

> That said, this adds complexity to their systems with the only benefit being cost savings.

That's not the only benefit. They also get much better reliability. The systems scale down with load, but they also scale up. As their load increases their system scales up along with it, giving greater reliability during increased load. Since they have to architect for that, it makes scaling up in general easier.

azakai• 14 years ago

> For now they really only saved $300k/yr, which is less than the cost of two engineers.

It took them 2 weeks to implement. That means it's saved a lot more than it took to implement (2 weeks times a couple of engineers) already.

To use your metric, if they saved close to $300K/year, that means they could afford to add 2 engineers to their staff, which is very significant.

jbigelow76• 14 years ago

That's pretty interesting. I've been frustrated in the past by both AWS and GoGrid (and I'm sure every other cloud provider) that keep incurring VPS instance costs even when the instance is shut off. I understand that even if I'm not using the VPS the resources need to be kept in reserve (in theory), but the solution of destroying and reprovisioning instances sucks pretty bad, way to time consuming if you are dealing with only a handful and operationalizing it is not not worth it.

I'd love to move to a provider that let me provision an extra instance or two for either failover or testing/staging but not be charged for it if I wasn't running traffic to it.

EDIT: I stand corrected, I might have been thinking of Rackspace's cloud (can't remember what it's called now) instead of AWS. But I know for a fact I am right on GoGrid (and pretty sure Azure) because I have a long email chain arguing about charges for provisioned instances in off states.

james33• 14 years ago

Unless AWS has changed something since I last used it (which admittedly has been at least a year), they don't charge when an instance is off except for storage.

josegonzalez• 14 years ago

Well, not exactly true. If you pay a the Heavy Utilization instances, you are charged regardless of whether you even have the instance allocated.

zwily3• 14 years ago

AWS does not, and has not ever, charged for stopped instances (except for things like EBS volumes, which are billed separately).

jules• 14 years ago

Given the premium that Amazon charges, how does this compare to dedicated servers? And why are dedicated servers in the US so much more expensive than in Europe?

aioprisan3• 14 years ago

power is cheaper in Europe than in the US

PanMan• 14 years ago

Nope, it's the reverse, so that can't be the case. Some quick googling: http://en.wikipedia.org/wiki/Electricity_pricing US is between 8-17 cents. EU is 20-30+.

druiid• 14 years ago

Their engineer answered earlier in the thread. It was more along the lines of they had a small team and needed to scale up quickly (which I think is probably most of these EC2 stories you see, really). He also said that now their engineering team has more breathing room and considering dedicated/colo would be in the cards.

shimon_e• 14 years ago

Scale and operation. European operators that open up in America also provide cheap pricing. See OVH for example. Pinterest could easily get 80 servers by them for $5/hr.

RyanGWU825• 14 years ago

I'm Ryan Park -- I'm the Pinterest engineer quoted in the article. I'm happy to answer any questions about our setup.

Just to clarify, the auto-scaling was specifically for a pool of web application servers. At the time I gathered the numbers, there were 80 servers in that pool. In the last few months we've been moving toward a service-oriented architecture, and we've been able to use the same code to auto-scale the internal services. Of course it's not possible to auto-scale stateful servers like databases, but it's still saving us a considerable amount of money.

We implemented the auto-scaling in early 2012, so it's been in use for almost a year now. It only took about 2 weeks of engineering to build the system. It does need occasional maintenance, but it's still worth the effort given how much money it saves us.

ntoshev• 14 years ago

How do you decide whether to run a spot or on-demand instance to handle dynamic load? What happens if spot instance cost suddenly spikes?

Also, how do you use regions/availability zones?

RyanGWU825• 14 years ago

Right now we run everything in the US-East region, and we have all our services balanced across 4 availability zones. If there's a problem in a single AZ, it will affect every layer of our system, but only about 25% of the hosts in that layer. Some of our services are automatically resilient and can handle that easily. Others aren't so great, but we're working on more automatic failover.

When we need more servers for an auto-scaled service, we open spot requests and start on-demand instances at the same time. For most services, we want to run about 50% on-demand and 50% spot. We have a watchdog process that continually checks what's running. It launches more instances whenever there aren't enough, and terminates instances when there are too many. So if the spot price spikes and a bunch of our spot instances are shut down, the watchdog will launch replacement instances on-demand. It will also request more spot instances once the price has dropped back to normal. In reality we don't often run into spot capacity issues -- maybe once a month, and it's almost never apparent to our users.

I spoke about this in detail at AWS re:Invent last month, and the full talk is available online here: http://www.youtube.com/watch?v=73-G2zQ9sHU

achompas3• 14 years ago

Thanks!

How do your watchdog instances monitor other hosts? Do they track usage/load via snmpd, or something else?

Also, are they directly calling the EC2 API to launch or shut down hosts, or do you have an in-house deployment system?

RyanGWU825• 14 years ago

Our watchdog process calls the EC2 APIs directly to identify how many instances are running, which ones are spot instances, etc. Boto, the AWS client library for Python, makes that pretty easy. The watchdog isn't very sophisticated -- it just checks to make sure that the correct number of instances are running in each auto-scale group. Our application servers aren't very efficient in certain respects, so we don't trust metrics like usage/load to make auto-scaling decisions.

If I was doing it over again, I'd just use Amazon's auto-scaling features for all of this. At the time we built this, EC2's auto-scaling didn't support some of the features we needed. Since then, they've made it a lot easier to do things like set up a repeating schedule for auto-scaling, rather than using metrics.

We only have one EC2 AMI that we use for all of our servers. That AMI is pretty basic; it only does enough to connect to our Puppet configuration management servers. Puppet then configures the boxes as web servers (or databases, or...) and adds them to the appropriate load balancer.

aioprisan3• 14 years ago

it would be nice if you could share some of the logic that the watchdog uses in figuring out when hosts are shut down and others fired up

jedberg4• 14 years ago

> Of course it's not possible to auto-scale stateful servers like databases

That's not entirely true. :) It's just a lot harder to scale them elastically. At Netflix we've done some proof of concept work on elastically scaling Cassandra, although we don't have it in production yet. I think that is one of our goals for 2013.

thezilch3• 14 years ago

Theoretically, there's going to be a state of affairs where it's not possible -- that's physics. Maybe with multi-host tenant, so that we're not relying on AWS networking / control-plane screwing with what is going to be the most important part of that dance.

But this is about saving money; not being a tenant of multiple providers; not spending more than "2 weeks."

hnriot• 14 years ago

Id be happy enough if you could just keep Netflix running more often.

ericabiz• 14 years ago

Why would you not choose to go colo and save the money? Even with hiring an extra full-time person, going colo should drop your costs by at least half vs. Amazon.

RyanGWU825• 14 years ago

Our #1 requirement has been to keep up with the growth in traffic on the site. We've been growing so fast that there's literally no way we could have ordered and racked equipment fast enough. We were also a very small team -- a year ago there were only about a dozen people in the whole company. At this point we're much larger, which gives us room to consider more options like colo or multiple cloud providers.

AWS certainly feels pretty costly when you compare colo prices to the list price for on-demand instances. But one of the reasons I wanted to present our work is to show that you can use the cloud for a lot less than the list price. It takes work to buy reserved instances or run spot instances, but that does make it much more cost competitive.

samstave2• 14 years ago

I am not sure what you mean exactly.

By "going colo" they would need to layout all the upfront hardware costs. These would not be insignificant. They would then have all the operational overhead of maintaining all that gear.

You may have a higher monthly cost for AWS services - but without needing to buy any physical hardware, you have way less to worry about. Further - if you need to scale, it can be done in seconds rather than weeks/months given lead times for procurement, design time, implementation time (i.e. scheduling the install in the colo, coordinating the need for more space etc...)

This is just scratching the surface...

ck22• 14 years ago

Please make a way to authenticate incoming pinterest EC2 requests from the web vs other garbage traffic from EC2.

Or just simply publish your outgoing EC2 ip pool list.

We cannot completely block EC2 because of Pinterest and that's a bad situation.

kawsper• 14 years ago

Why do you need to block EC2?

pavs• 14 years ago

Can we have some stats?

Number of Servers? How much bandwidth per month? How much space needed and at what rate is it growing? Traffic stats (if possible)?

mcgwiz• 14 years ago

Do you use any Light or Medium Reserved? Or are they all Heavy?

trustfundbaby• 14 years ago

I'm interested in your db setup, specifically how do you replicate and locate them (regionally) to handle requests from the application instances, So that a request from Spain goes to the right server instead of trying to hit a db server all the way in Texas ... for example.

jcastro• 14 years ago

Out of curiosity what OS do you use? And do you make your own custom AMIs for various workloads?

WestCoastJustin• 14 years ago

Here is the AWS re: Invent STP 204: Pinterest Talks Rapid, Cost Effective Scaling on Amazon Web Services -- https://www.youtube.com/watch?v=73-G2zQ9sHU

jkat2• 14 years ago

In our investigation, we've found AWS to be around 5x more expensive. Being able to save 63% during off hours (or, let's just say, reducing AWS' bill an average of 30%) doesn't really seem to make much of a difference - and that's with paying a large amount upfront.

fbuilesv• 14 years ago

5x more expensive than what? Owning/leasing servers? Similar cloud providers?

jkat2• 14 years ago

More popular alternative. Either dedicated or collocating.

cperciva2• 14 years ago

If spot prices spike and spot instances are shut down, on-demand replacement instances are launched. Spot instances will be relaunched when the price goes back down.

I heard this at AWS re:invent, and thought I must have misunderstood. I'm still confused as to how this can possibly be a good strategy.

The pool of EC2 on-demand instances has a finite size -- it isn't magical -- and does hit its capacity limit from time to time. When there's high demand for EC2 instances -- say, when there's an outage in another AZ -- you're likely to see both spot prices going up and a lack of capacity in the on-demand pool. As a result, this strategy seems designed to only ask for on-demand instances at the times when they're least likely to be available.

jedberg4• 14 years ago

Spot instances come from both leftover on-demand instances as well as unused reserved instances. So it's quite possible to run out of on-demand and still have a low spot price.

cperciva2• 14 years ago

It's possible, sure... but only if the people who find that they can't launch on-demand instances don't think to try spot instances instead.

usaar333• 14 years ago

I deal with this a lot with my work at PiCloud and you very rarely see a lack of capacity in the on-demand pool (across every availability zone).

You actually see high spot prices due to what at first glance seems "irrational"; incredibly high bids on instance types, sometimes 3-4x over on-demand prices. I suspect such high bids are placed by customers of the spot instances who absolutely do not want their workload terminated early by Amazon and are willing to take the risk of paying more to run it to completion.

If you are a webapp though, like Pinterest, you don't have this desire. Hence, it makes sense to dynamically switch.

teeja• 14 years ago

I'm wondering if you did (or anyone has done) any cost effectiveness on using SSD's rather than HD's. (Or maybe that's an order-of-magnitude too low to consider?)

thezilch3• 14 years ago

Netflix has done such benchmarks [0]; TLDR: _their workload_ cut costs in half with hi1 instances.

[0] http://techblog.netflix.com/2012/07/benchmarking-high-perfor...

jcampbell1• 14 years ago

I can't imagine that SSDs would make any difference for web app servers. I am not familiar with app servers workload that uses a material amount of disk I/O.

That being said, if I built custom app servers, I'd use SSDs because the cost is small for a system that doesn't need much storage.

jacques_chester2• 14 years ago

Reminds me a lot of how some energy-intensive plants can spin up less-efficient / quick-start units during offpeak hours to squeeze a little extra production out.

And vice versa: most electrical utilities have slow-starting, efficient-as-possible turbines that never get turned off (baseload -- coal is most common), and a bunch of relatively inefficient but flexible turbines (usually natural gas).

eru• 14 years ago

Nuclear also makes for great baseload.

Wind or solar are similar in the sense, that you do not gain by turning them off, but their supply is not stable.

jacquesm4• 14 years ago

Nuclear is asymmetrical. It's very fast to shut down but horrendously slow to start back up again.

In case of sudden load drops where nuclear plants will be shut down for safety reasons this can cause availability to be affected for weeks afterwards.

brazzy• 14 years ago

> relatively inefficient but flexible turbines (usually natural gas).

Actually, natural gas plants are at least as efficient as coal. They're just more expensive, especially if you turn them on and off a lot, which is pretty bad for the lifetime of a lot of components.

hga• 14 years ago

Depends on the kind of plant, there's baseline gas plants that heat water to steam and thence to steam turbines, they're akin to coal plants, high capital costs, low operating costs. Then there are straight gas turbines, akin to the ones that power jet airplanes but more like the ones that power most of the US Navy's ships. These have lower capital costs but higher operating costs and are used for peaking.

Pinterest Cut Costs from $54 to $20 Per Hour With Automatic Shutdowns

Related Stories