Scaling Mercurial at Facebook

https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/

by jordigh • 13 years ago

360 241 13 years ago

code.facebook.com

ngoldbaum3• 13 years ago

Mercurial has seriously improved over the past couple of years. If you tried mercurial a few years ago and were scared away due to speed or functionality issues, you might want to give it another shot.

JoshTriplett9• 13 years ago

Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does.

Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches? Does Mercurial provide built-in commands equivalent to "commit --amend", "rebase -i", and "add -p"?

ngoldbaum3• 13 years ago

Mercurial refers to git-like lightweight branches as bookmarks [1]. Branches are still permanent tags in the repository.

Commit --amend is now part of base mercurial. Rebase is available as an extension that is shipped with the client, like most advanced functionality. I don't know what add -p does.

[1] http://mercurial.selenic.com/wiki/Bookmarks/

mikegioia• 13 years ago

I work in several branches in the same file system directory. Branching in mercurial is as simple as "hg branch branchname", and switching branches is just "hg update branchname". As long as you make sure you commit everything you can easily work in one directory.

Merging changes across branches is easy too, as you only need to specify the revision number.

jordigh6• 13 years ago

if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does.

There is no real standard workflow. There are tools in place to build whatever you want. This is old, but remarkably still relevant:

http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-me...

If you want git-like branches, Mercurial bookmarks are close, but not quite the same. They have some intentional differences, which make sense and do not impede you, but will not make sense if you absolutely want them to behave like git branches.

"commit --amend", "rebase -i", and "add -p"?

hg commit --amend, hg histedit, hg record (but hg crecord is much nicer).

They're built-in, but you have to flip them on the last two with a config switch.

durin429• 13 years ago

Note that extensions that ship with Mercurial are exactly as supported as the tool itself, and come with the same kinds of backwards compatibility guarantees. It's not worth worrying about rebase, histedit, and record being extensions - if you want them, turn them on and use them.

k_bx6• 13 years ago

Why do you think mercurial branches are not lightweight? Which operation (create, close, push, pull) on branches is noticeably slower than git's?

Uchikoma2• 13 years ago

Let me translate: "if this tool is not exactly like Git, I will not use it".

masklinn6• 13 years ago

Does the standard branch workflow still expect you to have a separate repository and directory per branch?

You're probably confusing mercurial with bazaar, mercurial has always had branches (though they're not quite the same as git, mercurial's bookmarks are more closely related to git branches) and anonymous heads (contrary to git, an unnamed head is not stuck in limbo).

I don't care about plugins

That's stupid, mercurial is very much about plugins: there are dozens of official plugins shipped in a standard mercurial install.

the standard workflow

there is no such thing.

Does Mercurial provide built-in commands equivalent to "commit --amend", "rebase -i", and "add -p"?

All of them are provided in the base install, you just have to enable the corresponding extensions.

LukeHoersten• 13 years ago

I love the Mercurial community. We use Mercurial at work and I'm able to get instant support in IRC for any issue we have with an awesome signal/noise ratio. I'm glad Facebook is contributing back so much as well. My suspicion is that open source projects tend towards Git because of GitHub but I think a lot of companies who don't have the option of external code hosting lean towards Mercurial. All anecdotal observations of course ;-)

VladRussian24• 13 years ago

My suspicion is that open source projects tend towards Git because of GitHub but I think a lot of companies who don't have the option of external code hosting lean towards Mercurial.

GitHub is a consequence not the cause (ponder for a moment why there is no MercurialHub...) It is about ability to choose best source control tool for multi-versioned distributed concurrent development. Open source devs have such choice while corporate ones - no. FB choosing Mercurial tells a lot about environment there.

Timmmmbob2• 13 years ago

That makes no sense. Facebook specifically said they might have chosen git! They had the same choices as any developer except that a) they don't care about github, and b) they wanted to modify the VCS to support massive scale. Apparently out of git and mercurial, mercurial is easier to extend.

Uchikoma2• 13 years ago

90% of people choose Git because everyone else uses it. Noone has been fired for buying IBM.

ZenoArrow2• 13 years ago

There is a well used "MercurialHub", it's called BitBucket: https://bitbucket.org/

It supports Git now as well, but it was only for Mercurial use when it started.

hibbelig• 13 years ago

Isn't MercurialHub called Bitbucket?

xkarga00• 13 years ago

"Our engineers were comfortable with Git and we preferred to stay with a familiar tool, so we took a long, hard look at improving it to work at scale. After much deliberation, we concluded that Git's internals would be difficult to work with for an ambitious scaling project."

http://www.techradar.com/news/software/how-open-source-chang...

"And then Git itself wasn't working for us anymore because it wasn't scaling when we'd have an operating system release. So we ended up hiring most of the Git team - there's like only one or two core committers now for Git who don't work at Google,"

durin429• 13 years ago

Note that Facebook is trying to scale a single large repository, not an army of slightly smaller ones. It's a very different problem, and has to be solved in a very different way.

daleharvey• 13 years ago

I have seen presentations mentioning google development done as a single monolothic repository

http://www.infoq.com/presentations/Development-at-Google

Timmmmbob2• 13 years ago

Android is only split up into smaller repositories because git can't handle having it all in one.

Touche4• 13 years ago

We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository.

Pray tell?

kyrra2• 13 years ago

From the article:

Splitting it up would make large, atomic refactorings more difficult.

If you split the code base up into multiple projects, you need some sort of meta project to link them all together if you need to deploy it as a single entity. Doing a meta project isn't exactly easy, so it tends to be better to do a single project if everything is all tightly coupled still.

The project could be split up into smaller modules that are linked via shared common interfaces, but takes more time to maintain.

Touche4• 13 years ago

What is a wrong with submodules or subtree for a meta project? Or better yet, package management.

dudus• 13 years ago

On the other hand the Mercurial modularity was key for the migration.

georgemcbay• 13 years ago

I know nothing of the code Facebook is hosting, but one fairly obvious general benefit of having things in a single repository is that you can use relative paths to link dependent projects, like if you have a directory structure such as:

libs

libs/mylib

apps

apps/myapp

then myapp can reference library mylib via ../../libs/mylib

If all of these things are in separate repositories, all bets are off about how their paths relate to each other on the local filesystem, resulting in more complicated build procedures.

markcampbell• 13 years ago

There's a solution to this. Git submodules.

breck• 13 years ago

You can still have the benefits of relative links without a single repo if you create a dir at the root level.

For example, you can have a /facebook/ dir on every machine with the source, and all your subprojects can assume they are working off that, ie:

include /facebook/somemodule

yonran4• 13 years ago

I worked at Google (in a team using Perforce) and now work at a different company that uses multiple interdependent projects using Maven. Using a single monolithic codebase along with a build tool that statically builds everything at trunk has its advantages:

* You immediately get improvements from upstream projects without having to get them manually.

* You can unambiguously answer the question, What code am I using? with a single number. With multiple repositories, you have to list all the versions of each project that you are using.

* Easy API refactoring. You don’t have to worry about coordinating version number bumps across different repositories/dependency manifests when you make major changes to inter-project APIs. With a monolithic repository, you fix all callers of an API using a single code commit. No need to edit the version numbers in your pom files.

* Low cost to split a project into multiple separate projects. With multiple repositories or version numbers, you are reluctant to create new projects because the APIs will forever be harder to refactor (since you will have to worry about version numbers).

* No diamond dependency problem of 2 dependencies using a different version of a base project. Everyone is using the same version of base.

With a monolithic repository+build system, upstream callers are responsible to never make a commit that breaks downstream callers. I feel like it’s similar to the question of optimal currency areas. If your organization is growing in lock-step, then you can all happily share a single gold-standard repository with little friction. But if you can’t trust your upstream projects, you introduce versioning between the projects and have to deal with the mental burden of wondering whether to upgrade to the newest upstream project and whether you’re actually running the latest code.

Edit: added a couple more.

Touche4• 13 years ago

* You immediately get improvements from upstream projects without having to get them manually.

You also immediately get regressions. Not trying to be dismissive, but we fundamentally have different software philosophies if you think this point (which is the essence of most of your points) is a good thing that should be encouraged.

bluesnowmonkey• 13 years ago

Does this approach not isolate Google from the open source community? Most companies nowadays run on open source tools built from open source libraries written in open source languages. Version numbers and stable APIs are crucial.

exDM693• 13 years ago

* You can unambiguously answer the question, What code am I using? with a single number. With multiple repositories, you have to list all the versions of each project that you are using.

Git submodules may have a number of problems of its own, but it solves this one. There's always an unambiguous version number, which is the commit hash of the top repo. Every subrepository's commit hash is stored in the top repo and the top repo's version is an unambiguous version number of the entire code base.

I wish that more effort would be put in Git submodules, it's pretty much an afterthought addition but I've heard that there have been recent improvements and future improvements may be coming...

shoover2• 13 years ago

Nice list. What then are the mechanics of making a breaking change in a library, file format, network protocol, etc. in the single repo system?

mindjiver2• 13 years ago

At a previous job I migrated a quite large code base to git. IIRC it was at least 500k files and a couple of 10MLOCs. We had the same "scaling" issues Facebook mentioned here when trying to place all of this inside one repository. So we ended up going with submodules for this. Another idea was to perhaps enable re-use of repositories and/or disconnect old legacy code this way.

We did took a quick look at Mercurial but since lots of the upstream tools we used was using Git (linux, uboot, yocto, etc) it was an obvious choice. I seem to recall there being two hg extension that where of interest at the time (2010-ish), one to add inotify support and another to store large files outside the repository (hg-largefiles?).

Seem like Facebook's approach to the lstat(2) issue with watchman [1] is to use inotify on Linux. This has been discussed a couple of times for git as well but nothing has come of it so far [2].

[1] https://github.com/facebook/watchman [2] http://git.661346.n2.nabble.com/inotify-to-minimize-stat-cal...

natrius3• 13 years ago

Branches are my biggest pain point in Mercurial. What branching workflow does Facebook use?

k_bx6• 13 years ago

Why not use standard mercurial branches? At least you're able to say in which branch commit was done and draw a clean history for them.

natrius3• 13 years ago

I do use standard Mercurial branches, and I vastly prefer Git's model. I ask because there might be a better way to use them that I've overlooked.

durin429• 13 years ago

In what way? Have you looked at bookmarks?

natrius3• 13 years ago

I have. I'm wondering if they use bookmarks or named branches, and what strategies they've arrived at to work with either.

fhd23• 13 years ago

Same here. As outlined below, we tried to use bookmarks for feature branches a few times and it either doesn't work or we can't figure it out. We used permanent branches for feature branches for a while, but too many of those (even closed) would slow down the repository eventually.

Our coping strategy is MQ. It's not the same, but solves the problem pretty well IMO. MQ patches can be sent around and backed up, so not being able to push them to the server is not such a big issue. Takes a little upfront investment to figure out how they work though, Mozilla has a pretty good guide [1].

[1] https://developer.mozilla.org/en/docs/Mercurial_Queues

Crito11• 13 years ago

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch:

<old-comment> Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain point, with a large enough codebase (and, critically, enough throughput) you start to realize that you are about to hit a brick wall. With perforce, this starts to manifest with service brownouts.

With perforce, you can reasonably expect to run into this brick wall somewhere in the neighborhood of terabytes of metadata and dozens of transactions per second. That changes depending on what sort of beastly hardware you are willing to throw at your version control team.

Git of course hits a brick wall much sooner, somewhere around single-digit gigabytes of data (depending heavily on the average size of every object in the DAG), even ignoring throughput.

Perforce is probably good enough for Facebook in the present, but if you are a company that large and if you are forward looking, it becomes apparent that with existing version control technology, "one repo per company" is not a long term solution. Even "one repo per department".

You can split it even further, but what you realize is that you are developing infrastructure that allows you to use many repos (for instance your build servers and internal code search/browsing tools will now need to understand that concept) but you are losing many of the benefits of Perforce. While you are in the process of adapting your infrastructure to wrap its mind around many repositories, it makes sense to allow dev teams to really take advantage of this splitting. Develop infrastructure that allows Perforce and git repos to coexist in the company, allowing dev teams to spin up new git repos for their every project at will. Done properly, git allows you to create massively scalable systems that you can count on supporting your companies needs for the foreseeable future.

Smooth migration, migration that does not disrupt development, takes months (assuming the right initial conditions), so it is best to recognize the problem and start early, before service becomes disrupted.

If I understand Facebook's situation currently, they are still in the "try to make Mercurial scale" stage of denial, burning developer time and effort to push back that first brick wall (the same one that git hits, though mercurial hits it after git hits it but before perforce hits it...)

Here is a google presentation about extreme scaling with Perforce: http://www.perforce.com/sites/default/files/still-all-one-se...

An example of building multi-repo infrastructure for large projects with git is Android's repo: http://en.wikipedia.org/wiki/Repo_(script) Repo is just one example though; other, better, solutions are very possible.

</old-comment>

I guess the TL;DR here is that I think it is great that Facebook is making a single Mercurial repository work for their purposes right now, but I think they are kidding themselves if they think that is a long term solution. They are doing a pretty good job of making Mercurial scale like Perforce can scale, but that will only work for them for so long.

(In the above comment I talk about building a scalable system with git ("with" git, rather than making git itself scale), but the same can of course be done with mercurial instead of git. I don't mean this to be a comment suggesting that they should use git instead of mercurial.)

durin429• 13 years ago

An example of building multi-repo infrastructure for large projects with git is Android's repo: http://en.wikipedia.org/wiki/Repo_(script) Repo is just one example though; other, better, solutions are very possible.

I'm curious - do you know of any such better examples, or is this merely a theoretical "I feel like we could do better" statement?

Crito11• 13 years ago

I've worked with more pleasant systems, but they are not public and I don't feel comfortable talking about them publicly in much detail.

Basically though, repo is good for a single project ("Android", as a collective entity) but a company looking at working with many repos will want something designed with handling many unrelated projects at once. (For example, all the repos for "Android" and all the repos or "ChromeOS" should be able to live alongside each other in this system without any developer hassle, even though they may not have anything to do with each other).

Also repo is really only one part of the solution a company should look at. Properly done, the sort of system that I am talking about is really several systems that are all basically on the same page. Your build servers should understand how to get the code they need to build something, your internal code search/browsing tools should understand where code lives, etc. Re

Furthermore, allowing different types of repos to exist alongside each other is a good idea. That way each team can individually decide if they want to use Mercurial, Git, or even SVN.. To my knowledge, the repo script is only good for many git repositories.

pnathan2• 13 years ago

I participated in creating and open sourcing Guestrepo for hg. It is a very nice multirepo infrastructure, in my humble opinion. ;-)

https://bitbucket.org/selinc/guestrepo

sdesol• 13 years ago

My product is a read-only solution, which simplifies my technical requirements, but it basically de-emphasizes the repository, by making branches the focal point. If you have ever worked with UCM ClearCase by IBM, you'll understand that everything works at the stream/branch level. When you create a new UCM project, you construct it by picking points (baselines) from streams/branches, which can come from different pvobs (repositories). And UCM activities, which are like commits, can contain changes from different branches from different pvobs (repositories).

I personally think, if you want to take Git to the next level, you'll have to implement something like UCM ClearCase activities and projects. Commits should not be bound to a single repository and it should be very easy for people to say I want to create a product that uses starting points from branches x, y and z without having to think about what repo they belong to.

I have a couple examples that shows how my product de-emphasizes the repository. In the following example, you can see my Commits Finder tool, which lets you search for commits by branches.

http://ny.testdrive.gitsense.com/index?#pid=13&cid=20&trail=

As you can see, the search results shows commits from 10 different branches from 7 different repositories. My other example is my GitHub Pulls Finder tool which lets you search for pull requests by branches.

http://ny.testdrive.gitsense.com/index?#pid=24&cid=29&trail=

Here you can see pull requests from 6 different branches from 6 different repos.

Since my solution is read-only, it simplified things, but I don't think creating a nice layer on top of sub-modules would be be that technically challenging. And it's the direction I would personally go if I wanted to make Git more enterprise friendly.

VladRussian24• 13 years ago

With perforce, you can reasonably expect to run into this brick wall somewhere in the neighborhood of terabytes of metadata and dozens of transactions per second. That changes depending on what sort of beastly hardware you are willing to throw at your version control team.
Git of course hits a brick wall much sooner, somewhere around single-digit gigabytes of data (depending heavily on the average size of every object in the DAG), even ignoring throughput.

if you use Git the same way you'd use Perforce or Mercurial - one central server handling everything ... you'd better stay with Perforce or Mercurial. Corporate dev process is usually built around notion of "central" repo (and they equate it with "master") - just replacing Perforce server with Git server wouldn't change anything.

Crito11• 13 years ago

That is certain. If you are trying to run at a very large scale with a single repo, Perforce (or apparently Mercurial) is the way to go.

The problem is "how do you grow even further?" There isn't a perfect solution to that (yet), but I think the only existing workable (although absolutely imperfect) solution is splitting your codebase into many small repos.

Thankfully very few companies need to worry about this problem right now. A single Mercurial repo is apparently scaling for Facebook currently, and few companies will manage to stress Perforce.

ariwilson• 13 years ago

How do you import code from one repo into another?

This is the fundamental problem Facebook (and Google, Amazon, etc) are all trying to solve: how to share code company-wide. Sub-repos and making your tools hack around the repos doesn't solve that problem.

Crito11• 13 years ago

I think ultimately, past the scale that you can get with Perforce, that is a problem waiting to be solved by the next generation of VCS.

You can get all your many repos in one place, with all of your internal systems handling them nicely, but moving one file from one repo to another, in a way that history remains clear, is something of an unsolved problem.

cstejerean3• 13 years ago

You publish packages/libraries/gems/jars/whatever your language calls them, and a package manager. Needing to combine the actual source trees into a single repo like submodules or subtrees allow you to should be for really rare cases.

justinmk• 13 years ago

Facebook's problem is that they were trying to scale with Git improperly.

Here's the Feb 2012 thread with Facebook's Joshua Redstone regarding their experiments with git:

http://comments.gmane.org/gmane.comp.version-control.git/189...

In August 2012, they hinted they were near a solution:

http://www.quora.com/Facebook-Engineering/Has-Facebook-solve...

...and that teaser was just today updated with the remark, "Our Mercurial team has just posted the goods! Our problems are solved, mainly by not using Git."

pconf• 13 years ago

Please correct if I'm wrong but it looks like the decision to scale using HG instead of git was made on 2 points: 1) git maintainers basically said you should split repositories and left it at that, and 2) HG's cleaner code and abstractions made it easier to patch.

Fundamentally I like how this cleans up the design flaw of requiring history everywhere. 99% of the time when I clone or pull or diff or whatever I only care about HEAD. Why should I be forced to pull or store GBs of history or even MBs of metadata I can't use? Why not make leaving this data on the server optional? I can see how the decision to push history everywhere was made for simplicity but it doesn't reflect real world usage and clearly isn't scalable. Let's hope these history-option patches continue to be developed and make their way upstream. They certainly have my vote, not just as options but as defaults.

dubcanada• 13 years ago

I think they just want to modify Git and don't have any solid C developers that can make such things. So they turned to a python solution which is perfectly fine.

But webkit, and chromium, as well as other GIANT projects which as far as I know are larger then Facebook seem to work fine on Git.

eddieh• 13 years ago

WebKit uses Subversion (http://www.webkit.org/building/checkout.html). You can use Git with the WebKit repo, but it's git-svn doing the work...

khuey• 13 years ago

You really believe that Facebook couldn't find a competent C programmer to make some changes to git?

jaseemabid• 13 years ago

Its still one of the coolest and best paid companies out there! You really think they couldn't hire a few guys from core git team saying - "Dude, just improve git and we'll pay you?". You must be kidding

randartie3• 13 years ago

It's a bold assumption that those open source projects are larger than facebook, a company which employs 2000+ engineers.

webkit and chromium have about 6 or 7 million lines of code and < 200,000 commits.

Facebook 6 months ago had 62 million lines of code and > 1000 commits a day.

http://www.quora.com/Facebook-Engineering/How-many-lines-of-...

Also, there's a difference between those open source projects using large repos with git and Facebook wanting to increase developer efficiency. Developers sitting around waiting for a rebase doesn't really pay off, where as an open source project can get away with it.

Crito11• 13 years ago

The idea is that you should not have to modify git or mercurial. Instead of modifying git or mercurial to handle a single repo for your entire company, you should create a system that uses them to handle many repos in your company. You don't need C programmers to do this with git.

vishnugupta• 13 years ago

Amazon ran into this exact problem with Perforce. They threw hardware at the problem for a while but, fortunately, soon realized that they weren't getting anywhere with that. They were trying to work "with" git i.e., with one repo per service and were in migration phase when I left.

ibrahima2• 13 years ago

I get this impression every time I see a "look what neat scalability thing we did" post from Facebook Engineering. It's great that they're able to achieve such technical feats, but they refuse to acknowledge that maybe they're using the technology wrong. I'm reminded of the time they hacked the Dalvik VM on Android because apparently they had too many method names for the Dalvik VM to handle. http://jaxenter.com/facebook-s-completely-insane-dalvik-hack...

I mean, they run into a resource allocation problem for an application that is essentially a glorified web view, and they think "how can I hack up the VM to bypass this limitation?" Sounds like insanity to me.

randartie3• 13 years ago

I can't agree. Instead of taking everyone's advice of either switching to perforce (less user-friendly/frustrating repo) or splitting their repo into parts, facebook built a solution where they don't have to make either compromise.

Just because their solution is complex doesn't mean they're using the technology wrong. They set out their requirements and they met all of them. All their engineers are more efficient now and all this complexity happens behind the scenes.

MetaCosm• 13 years ago

It is insane, but it never comes all at once. I have been at a company that had this type of internal insanity, and it is a bit of a "boil the frog" thing.

Each individual step seems resonable and sane thing -- but when you look at the result of all those steps -- you stand back and just stare in awe at the horror you have created... it hits you just how far you have drifted.

In a recent meeting, we had a corporate redirection a bit -- and a co-worker simply asked the question "Is this a sane approach to what our actual problem is?" -- glad to be working with a crew who askes those kinds of questions.

weland• 13 years ago

I don't know if this is Facebook's case, but when I've seen stuff like that happening, it was because a bunch of (otherwise smart) developers were far too arrogant about the quality of their own work, too derogatory and, to some degree, too superficial about the quality of other programmers' work, and -- perhaps fatally -- too driven by the can-do-no-matter-what attitude that is so obnoxiously prevalent in today's corporate world, at the expense of common sense. Between the technical debt and the management pressure, I've seen a bunch of people refusing to admit it's their fault and cleverly save their asses (and earn a hefty bonus on one or two occasions) by doing something any sane programmer would have, in fact, fired them for.

CmonDev5• 13 years ago

What else do you expect from company which hires based on solving algorithmic problems?

moron4hire6• 13 years ago

"Our code base has grown organically and its internal dependencies are very complex."

That's a polite way of saying "we write shitty code without any sort of plan."

"Splitting it up would make large, atomic refactorings more difficult"

Actually, it's the other way around. Modularity tends to obviate the need for large, atomic refactorings.

And what, exactly, is the meaning of these graphs? This is leading me to believe that being a developer at Facebook is about quantity over quality.

ZenoArrow2• 13 years ago

To be honest, whilst we have no way to accurately determine whether the code is a mess without a chance to see it, the most surprising line of this article (in my opinion) was that the code base was larger than the Linux kernel. I'm not seeing anything on the front end that would warrant such complexity, guessing a large chunk of the code base is server code. Would be interested in reading a summary of the components of the Facebook code base.

plorkyeran• 13 years ago

I suspect that the kernel is one of the only things running on Facebook's servers that they didn't write from scratch. Alexandrescu has mentioned that a 1% speedup to HHVM saves FB about $100k per year, and at that sort of scale it's pretty easy for reinventing every wheel to make sense.

igravious2• 13 years ago

This rather surprised me as well. I tend to think of the Linux kernel as one of the larger single code-bases out there. Am I wrong?

enneff3• 13 years ago

Modularity tends to obviate the need for large, atomic refactorings.

"Tends to".

But when you're dealing with code at Facebook's scale, things that "tend not to happen" actually happen quite a lot. In fact, you must plan for them as a matter of course.

So yes, modularity is great, and I because I'm a nice guy I assume Facebook aren't a pack of idiots and that they're writing nice modular code. But even if that's the case, in an organization of Facebook's size you still need to make widespread, atomic refactorings on a regular basis.

I know this from experience, because I work at Google (much larger codebase than Facebook) on a low-level piece of our software stack. We face these issues regularly and while working in a single repo has its drawbacks, it also has real advantages.

moron4hire6• 13 years ago

Perhaps I don't understand the whole situation here. I hear "all of our code is in one repository" and I think "GMail and Google Maps are in the same repository, in the same repository with GoLang, in the same repository with AdWords."

The more I think about it, the more I think your post reveals a lack of maturity in our industry that lends credence to the pro-engineering-licensing argument that I've argued against many times on my own. That everyone can be so cavalier about this topic.

Because the fact that your companies are so large is EXACTLY why it makes no flipping sense that you're running gigantorepository. You have so many products, so many projects going on, that I just really have a hard time believing that it was disciplined software development that led to all of your code being so interdependent.

But the part that started getting under my skin was the fact that we aren't talking about Bob's Local Software Consultancy here. We're talking about two companies that touch the lives of hundreds of millions, perhaps even billions of people in the world.

If OpenStreetMap doesn't have their code in the same repository as Postgres, Linux, and DuckDuckGo, then there is no excuse for the Facebook Android App to be in the same repository as HHVM.

danielrhodes• 13 years ago

"We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Even at our current scale, we often make large changes throughout our code base, and having a single repository is useful for continuous modernization. Splitting it up would make large, atomic refactorings more difficult. On top of that, the idea that the scaling constraints of our source control system should dictate our code structure just doesn't sit well with us."

I remember reading about their Git troubles awhile ago, and I still don't buy this argument that it is better to have one large repository. One reason modularization is important is for the precise reason they are trying to get around it: removing the ability to make large scale changes easy and thus increasing reliability.

However, my understanding is that their desire to have one large repo is reflective of their their "move fast and break things" philosophy, which means not being afraid of making large scale changes. So I would be interested in hearing how they mitigate the obvious downsides given how many people they have committing to their codebase. It seems like you would just end up having to create constraints in other ways, so which constraints end up being the lesser of two evils?

zaphar6• 13 years ago

We use one repository at google.

We've found that "Removing the ability to make large scale changes easy and thus increasing reliability." isn't actually correct.

As an example, most of your codebase uses an RPC library. You discover an issue with that library that requires an API change which will reduce network usage fleetwide by an order of magnitude.

With a single repo it's easy to automate the API change everywhere run all the tests for the entire codebase and then submit the changes thus accomplishing an API change safely in weeks that might take months otherwise with higher risk.

Keep in mind that the API change equals real money in networking cost so time to ship is a very real factor.

It sounds like facebook also has a very real time to ship need for even core libraries.

igravious2• 13 years ago

What version control system do you use, or is it a secret? Perforce? Git? I've seen Linus's talk that he gave about Git at the GooglePlex so perhaps you use Git. If so, how have you not run into Facebook's scaling issues?

X-Istence• 13 years ago

You don't need a single repo to be able to run tests across all existing tools. If you have proper dependency management set up, you make the change, push it and a CI server goes off and builds it, then all dependent projects get rebuilt and tested...

novaleaf3• 13 years ago

thanks for this remark. i've been struggling with the choice of splitting into multi-repos, but have similar thoughts to what you just confirmed.

barrkel• 13 years ago

What you've got here is a different kind of tradeoff.

Normally, libraries are written under the assumption that clients cannot be modified or updated when the library changes. This brings in the concept of a breaking change, and a set of design constraints for versioning. For example, modifying interfaces becomes verboten, final methods start becoming preferable to virtual methods, implementation detail classes require decreased visibility, etc.

The advantage is that library producers are decoupled from consumers. Ideally the library is developed with care, breaking changes between major versions are minimized, and breakage due to implementation changes are minimal owing to lack of scope (literally) for clients to depend on implementation details.

But under the model you describe, you're leveraging the Theory of the Firm as much as possible - specifically, reducing the transaction costs of potentially updating clients of libraries simultaneously with the library itself.

The downside is the risk of unnecessary coupling between clients and libraries - the costs of a breaking change aren't so severe, so the incentive to avoid them is lessened, and so the abstraction boundaries between libraries is weakened. If the quality of engineers isn't kept high, or they don't know enough about how and why to minimize coupling, there's a risk of a kind of sclerosis that increases costs of change anyway.

ed_blackburn• 13 years ago

I wonder what they use at Microsoft. For their sake I hope they don't subject their own engineers to TFS.

mindjiver2• 13 years ago

I read somewhere(?) that they where running Perforce with some internal tooling built on top of it.

boyter• 13 years ago

I saw/heard that as well (hopefully someone can come up with the link). Things like Windows/Office apparently are in a custom perforce.

I believe that they do use TFS on a lot of the internal projects though and that Visual Studio is now done in TFS.

adamnemecek• 13 years ago

This is correct.

http://programmers.stackexchange.com/a/85891

jabits• 13 years ago

I bet you must have a lot of experience with TFS...

CmonDev5• 13 years ago

Does it support not checking-out files that were edited but did not change in the end or you still have to revert them manually?

com2kid3• 13 years ago

We eat our own dog food, most teams have moved over to TFS by now. I don't know about the larger orgs (Windows, Office) but for smaller groups TFS is how it is.

MS doesn't have a unified build environment, every team generally does their own thing. It has its pluses and minuses.

Shared code would occasionally be useful at Microsoft, but not as often as you'd think. Generally when relying on another team's code, it is preferred to take it as a binary drop when they do a product release, just like any other customer. This helps prevents needing to deal with churn in one's dependencies.

Pxtl3• 13 years ago

We eat our own dog food.

Except for that whole ".NET framework" thing.

agwa2• 13 years ago

For some context of why Facebook choose Hg over Git, here's the mailing list thread where Facebook initially reached out to the Git developers: http://thread.gmane.org/gmane.comp.version-control.git/18977...

novaleaf3• 13 years ago

I personally love mercurial (simpler than git), and have been a bit nervious over the last year or so with the mindshare shift to git.

So hearing about this (FB all in with hg) ensures that hg won't be falling behind... at least in the nearterm.

dgesang4• 13 years ago

Don't believe the hype. hg will be around for some time.

CmonDev5• 13 years ago

Don't follow the lemmings, choose your own path!

tshepang2• 13 years ago

I created this list to help me stop worrying: http://tshepang.net/major-projects-using-mercurial.

randartie3• 13 years ago

The discussion appears deleted on that thread

novaleaf3• 13 years ago

fyi, it was working about 8hrs ago. broken now. guess someone didn't like the inuendo ;)

hermanradtke• 13 years ago

That link appears dead. Here is another: http://git.661346.n2.nabble.com/Git-performance-results-on-a...

boklm0• 13 years ago

According to this page, Mercurial project leader is currently working for Facebook: http://mercurial.selenic.com/wiki/mpm

marmoute• 13 years ago

Facebook took the Mercurial path about 2 year ago, they hired some core devs one year later to secure and speed up development.

CmonDev5• 13 years ago

Google bought Git devs, Facebook bought Mercurial devs - wise choice for companies that even order custom CPUs.

jordigh6• 13 years ago

There are some hg devs at Google too. :-)

rottyguy• 13 years ago

Any comments on the build system being used for a monolithic hive of this size?

greatsuccess• 13 years ago

Partitioning is the answer. In a repo that size 99% of the history is useless to anyone. You wouldnt manage a database like this so why force SCM down this path?

If they used git with say only the last year of history in it they would be having zero issues.

ryanobjc• 13 years ago

While true, you only need 1% of the history, the problem is WHICH 1% and when?

Tracking down bugs in the history is a huge deal, and its often absolutely critical. And yes, regressions go past 1 year.

exDM693• 13 years ago

Partitioning may be the answer but this is a huge problem for corporations like Facebook (and the one I am working for). If things have been done with one giant repo from day one, splitting it is going to be a major engineering/political/social problem when there's thousands of engineers working on the code base and you can't just shut down business for the duration of the migration.

riddlemethat• 13 years ago

Can anyone explain to me how Mercurial is used at Facebook in conjunction with Murder (https://github.com/lg/murder)?

voidr• 13 years ago

They didn't really scale Mercurial, they basically took it and replaced a lot of it's functionality with remote services.

I understand their reasoning that technology shouldn't dictate the way they develop stuff, however I think splitting the project up and using submodules would have been a cleaner approach. If refactoring everything is something you do all the time, you might be doing it wrong.

cool-RR• 13 years ago

I wonder whether putting all the code on a ramdisk (with backup of course) is feasible. If so it might be a very cheap solution.

exDM693• 13 years ago

No, using a ramdisk these days usually makes things worse, not better. The reason is that the operating system already holds as much of the filesystem in caches (in RAM) as possible. So as long as you have enough RAM in your system, files will be cached and the result is better than using a ramdisk.

Scaling Mercurial at Facebook

Related Stories