Automating Datacenter Operations at Dropbox

https://blogs.dropbox.com/tech/2019/01/automating-datacenter-operations-at-dropbox/

by dmicher • 8 years ago

94 22 8 years ago

blogs.dropbox.com

nemothekid• 8 years ago

A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy) - which is interesting given most providers are writing new infrastructure code largely in Go (Spinnaker is another Python exception).

I'm guessing that Python is still heavily used inside Dropbox, but does anyone know if they have published any style guides or tooling to managing Python codebases at their scale?

amanzi• 8 years ago

Dropbox hired the creator of Python to help migrate their huge Python 2 codebase to Python 3. It would make sense for them to continue with their investment in Python.

lwf• 8 years ago

Dropbox sponsors most of the developers of http://mypy-lang.org/ , an optional static typing system for Python.

I've found it to be hugely useful in safely working in a large Python codebase :)

(NB: I work at Dropbox)

mroche• 8 years ago

Curious if you’ve ever tried Cython out. Not quite the same, but a semi-similar end goal to a degree. I started looking into the other day and it might provide some nice improvements to heavier applications.

jedberg• 8 years ago

Unless I'm mistaken, Guido still works for Dropbox. And as far as I know they still mostly do Python.

jensvdh• 8 years ago

Spinnaker is written in Java

jabl• 8 years ago

A theory based on the article is it seems Pirlo may be written in Python (going off the fact that is leverages SQLAlchemy)

Sounds like a reasonable guess.

which is interesting given most providers are writing new infrastructure code largely in Go

Dropbox apparently uses Go a lot, see e.g. https://about.sourcegraph.com/go/go-reliability-and-durabili...

They also use Rust for some performance-critical stuff, e.g. https://news.ycombinator.com/item?id=11282948

That being said, AFAIK originally Dropbox was written mostly in python. Probably there's still lots of that left.

talonx• 7 years ago

This is also borne out by the fact that they mention Celery.

danpalmer2• 8 years ago

which is interesting given most providers are writing new infrastructure code largely in Go

I'm not sure this is true. There's a lot of Java around, Scala is pretty popular too, in many industries C/C++ are the norm still for this sort of code.

Go might be the trend in open source infrastructure projects right now, but a significant amount of that is likely to be inertia from Docker and Kubernetes.

peterwwillis2• 8 years ago

"While there are some excellent job queue systems such as Celery, we didn’t need the whole feature set, nor the complexity of a third-party tool. Leveraging in-house primitives gave us more flexibility in the design and allows us to both develop and operate the Pirlo service with a very small group of SREs."

I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tools that support the software developers writing the business products.

"Using the network database and configuration tool developed by our Network Reliability Engineering (NRE) team,"

Another custom tool? Network inventory and config management tools do exist already...

"Rather than having engineers manually running tests using playbooks, Pirlo performed an automated sequential battery of tests that reduced the need for hands-on attention and concurrently increased diagnostic accuracy."

Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system.

mrbanks• 8 years ago

Developers at these big companies get bored & assume they know better than a mature open source solution so like to reinvent a perfectly good wheel basically.

inferiorhuman3• 8 years ago

Or they've actually used Jenkins (or any of these other suggested alternatives). I've used Jenkins personally and professionally.

Most recently I've been rebuilding my own CI stack and never really gave much thought to going back to Jenkins. So I've been asking around and one of the only common complaints I've heard so far is that getting the initial configuration done is painful and generally orthogonal to automation. Plus the documentation is atrocious.

No off-the-shelf product will be a perfect fit, but with CI software I was truly surprised at just how large the gaps were.

So, sure, if you're Dropbox and you want to automate everything Jenkins is almost certainly not the right tool for the job. If Dropbox already had a supported, mature in-house job queue system, why not use it?

Conversely at megacorp, they spent 4+ years claiming to work on deploying a Jenkins (CloudBees) cluster and still came up with bupkis. Our own internal job and message queuing systems were astoundingly bad (and support for internal tools was almost entirely forbidden).

At megacorp I absolutely decried any sort of home grown solution. But if Dropbox were to actually tackle the problems of internal testing and support and come up with mature solutions, why shouldn't they use them?

swish_bob2• 8 years ago

The irony of people saying essentially "Dropbox could have done this just by using open source tools" on hackernews, where people famously announced the Dropbox would fail because it was easy to reproduce using open source tools entertains me.

Maybe, just maybe, the engineers in question considered mature open source tools and decided they didn't serve their needs.

danpalmer2• 8 years ago

I don't know any more than this paragraph or two explains, but it sounds like NIH syndrome. If you chose to write your own solution just because a 3rd party one was complicated or expensive, you've underestimated the complexity and expense of developing and supporting new software. Not only do you have software developers developing your business products, but now you have software developers developing the IT tools that support the software developers writing the business products.

I think the part "Leveraging in-house primitives" is the important bit here. It sounds like they have an in-house task queue, so introducing another external one like Celery might add a lot of operational complexity rather than piggy backing on existing infrastructure. As for the reason for the existing infrastructure not being Celery, I suspect at Dropbox's scale tools like Celery start to break down.

Or you could, like, install Jenkins, write your tests, and do all this without writing your own distributed job queue system.

They likely haven't written their own job queue, they've just used an existing one. Again, the operational overhead of Jenkins is unfortunately very non trivial.

swish_bob2• 8 years ago

Again, the operational overhead of Jenkins is unfortunately very non trivial.

Not to mention the overhead of running Jenkins at the kind of scale they're talking about here. We're not talking about small numbers here.

chrisweekly• 8 years ago

> "in-house primitives"

^ perfect band name

Automating Datacenter Operations at Dropbox

Related Stories