Open-sourcing Facebook's internal branch of Thrift

https://code.facebook.com/posts/1468950976659943/under-the-hood-building-and-open-sourcing-fbthrift/

by jamesgpearce • 12 years ago

66 14 12 years ago

code.facebook.com

This is very cool but forking your open source project like this strikes me as very poor open source behavior. Now there's Thrift and FBThrift. It would have been much better to have integrated improvements back into the open source Thrift project incrementally.

djwatson24• 12 years ago

After thrift became an Apache project, internal Facebook employees found it harder to iterate using external tools than our internal github repos. Many of the changes depended on things that weren’t open source, or only recently became open source in folly. We do hope to merge as many changes as possible back upstream, but felt it was more important to get the code open source as quickly as possible. In addition, fbthrift is a dependency for several other projects Facebook wants to open source in the near future.

teacup502• 12 years ago

Wouldn't it have made more sense to contribute the changes first, rather than creating a fork of your own project?

This is just wasteful behavior. Why would anyone try to work with you to build community around this code? Why should anyone trust that you won't just abandon this fork's community and then come out with another full rewrite in a few years time from your silo?

StefanKarpinski3• 12 years ago

It sounds like a significant amount of the friction may be coming from the Apache Foundation's infrastructure and/or bureaucracy – or am I reading that wrong? It is hard to understand why Thrift isn't hosted publicly on GitHub – especially when Facebook is using GitHub Enterprise internally – unless the choice is political.

Many of the changes depended on things that weren’t open source, or only recently became open source in folly.

Is "folly" some sort of jargon I'm unfamiliar with here, or does this literally mean that some things were foolishly open sourced recently due to a lapses of good sense?

thrusong• 12 years ago

I hope one of those is Haystack. I've been dying to play around with it. It was originally supposed to be open sourced but they changed their mind.

klickverbot• 12 years ago

I wonder why they didn't publish the changes as one big monolithic commit on top of the Apache Git repo instead of obliterating all the history.

frisco• 12 years ago

I used to be all about Thrift. I first started using it back in 2008 and thought it was awesome for a long time. While it clearly works for Facebook (and many others), I've always had little problems with it. Connections silently fail (next request times out, following requests work again) if you don't make a call in long enough with some of the bindings. Generated servers sometimes leak memory. The type system is nice but creates all kinds of headaches when trying to reconcile it with something idiomatic in different languages. When many different services need to talk to each other it requires a lot of connections, each of which I came to view as a liability. Thrift's semantics are actually underspecified (http://twitter.github.io/scrooge/Symantics.html) so Twitter, who uses Thrift, came out with Scrooge, which is mostly compatible but not always.

My lesson was that Thrift is great if you have people who can look after it all the time. Thrift connections were never something you could forget about for a little while if you had a small team, though.

We've been moving away from Thrift towards a message queue-based system (using RabbitMQ) that lets us do everything Thrift does and more, and works every time. It used to be so nice to start a new service by writing the IDL and getting everyone to agree on a clean, well-bounded interface. Facebook forking their own project is kind of a last straw for my faith in Thrift, though. Oh well.

jamesaguilar• 12 years ago

30k qps? That seems pretty low even on a per core basis.

Open-sourcing Facebook's internal branch of Thrift

Related Stories