Making Facebook’s software infrastructure more energy efficient with Autoscale

https://code.facebook.com/posts/816473015039157/making-facebook-s-software-infrastructure-more-energy-efficient-with-autoscale/

by otterley • 12 years ago

55 26 12 years ago

code.facebook.com

praseodym• 12 years ago

This is one of these things where virtualisation can help even more. For example, VMware can dynamically put servers in standby mode when demand is low and power them up again when needed: http://www.vmware.com/products/vsphere/features/drs-dpm

SEJeff2• 12 years ago

You know that would be trivial to do with bare metal and out of band management cards like Dell Dracs, IBM RSA cards, HP ILOs, or generic IPMI BMCs, right?

Virtualization doesn't really add much of anything for that specific problem other than increased context switching and slightly lower performance.

Disclaimer: building this type of thing (on bare metal) is a chunk of my day job. I see it as unbelievably trivial. In fact, the same ideas are behind Rackspace's "OnMetal" initiative:

http://www.rackspace.com/cloud/servers/onmetal/

wmf5• 12 years ago

If you want to power manage your pets, VMware makes sense. But yeah, there's a reason Facebook doesn't use virtualization.

devicenull2• 12 years ago

Getting IPMI controllers to do what you want is anything but trivial :)

acdha• 12 years ago

Disclaimer: building this type of thing (on bare metal) is a chunk of my day job. I see it as unbelievably trivial.

You do, I'm sure, because you've invested a lot of time learning it and building tooling around it. For anyone who isn't a full-time sysadmin, debugging all of the many and various quirks in management hardware is a major time sink versus scripting a VM server's API.

vicaya2• 12 years ago

"Virtualization doesn't really add much of anything for that specific problem other than increased context switching and slightly lower performance."

This is BS. What if you have 3 physical servers with 30% utilization? DRS can _seamlessly_ consolidate _arbitrary_ application VMs to one server and shutdown the rest. With bare metal, only certain specifically designed workloads (stateless web farms and some distributed systems etc.) can be moved easily.

It seems that dismissing virtualization out of sheer ignorance is a fad these days. Virtualization provides important hardware abstraction to a much wider variety of workloads.

bhouston• 12 years ago

Vfx companies have software that turn off renddr farm machines in periods of low load. I helped write software that did this backin 2006.

Basically, we found it was possible to shut down machines in periods of low load and then use "Wake On Lan" to start them up once load picked up again.

I am unsure if the on-off power cycling reduces machine longevity.

Might be something worth exploring at Facebook.

mschuster912• 12 years ago

I am unsure if the on-off power cycling reduces machine longevity.

Possibly the disks are the most vulnerable components. I wonder if it would be possible in software to shut down the CPUs and have only disks + network running...

donavanm• 12 years ago

A bit late to the party on this one. Check out cpu "c states" and things like intels "speedstep." Modern cpus can reduce/shutdown power to individual packages and cores. This can reduce power consumption from a hundred tdp to tens of tdp.

The downside is latency associated with changing state. Depending on the change it can be hundreds or thousands of micros to go through these states. On a server workload this can introduce huge latency outliers as a request blocks on a core to wake up.

wmf5• 12 years ago

In a diurnal cycle like Facebook's, you'd have one start-stop per day, which should be well within the rated specs of hard disks. A few years back I looked at the idea of treating disk lifetime as a resource and explicitly managing it: http://dx.doi.org/10.1109/MSST.2011.5937221

That's assuming that the servers have disks at all, which they probably shouldn't.

meatmanek• 12 years ago

You could probably get a similar effect by using HAProxy's "balance first" algorithm, which chooses the first available server with an available connection slot (as defined by maxconn). If you did this, you'd want to set maxconn pretty conservatively.

Making Facebook’s software infrastructure more energy efficient with Autoscale

Related Stories