Microsoft and GitHub team up to take Git virtual file system to macOS, Linux

https://arstechnica.com/gadgets/2017/11/microsoft-and-github-team-up-to-take-git-virtual-file-system-to-macos-linux/

by dmmalam • 9 years ago

796 218 9 years ago

hasenj4• 9 years ago

Moving Windows development to git sounds like a completely irrational decision from a technical standpoint that was driven mostly by marketing concerns.

Jonnax• 9 years ago

What did they use before and why is it better?

akie• 9 years ago

On what do you base that assumption?

Tuna-Fish• 9 years ago

Umm, why?

gmueckl4• 9 years ago

Because git is the latest fashion rage and by itself incapable of managing the size of the Windows code base. This is the ideal combination of reasons for corporate PHB to decide to use it.

Exhibit A: GVFS was solely invented as a hack to make git usable by the Windows team according to the Windows team blog.

(Yes, there is some sour grapes and sarcasm in this post)

chx• 9 years ago

Marketing concerns by whom? The nonexisting company behind git? Github when MS doesn't use github? Or you think MS is going to be able to drum up some business just because they use git? Please elaborate!

enjo• 9 years ago

While I don't agree with OP... you could make a case that switching to Git is a way for Microsoft to make itself more fashionable to the engineering community at large.

The marketing here is: it's cool to work at Microsoft again!

Analemma_2• 9 years ago

I worked at Microsoft before and after my team (not Windows) did the Source Depot -> git transition, and I can say that it was mostly for the better. Source Depot was pretty good as non-distributed VCSs go, but it doesn't hold a candle to git. Some specifics:

- Branching in SD sucked, the only way to work on multiple features at once was to either have multiple copies of the repo or constantly fiddle with "package files" that contained your changes, and the whole thing shit the bed if the server went down.

- All the new hires both from college and the industry all know git and there were an increasing number of them who just found it bizarre that Microsoft was still on a proprietary, non-distributed system.

- Higher-ups (correctly, IMO) decided that VSTS/TFS had to have git in order to remain competitive, and that we should be eating our own dogfood. That's partly a marketing concern but also a legitimate technical decision.

sctb• 9 years ago

Could you please try to comment more substantively? We've already asked you this.

https://news.ycombinator.com/newsguidelines.html

hasenj4• 9 years ago

We've already asked you this.

Am I on some kind of a black/gray list? Either that, or you specifically remember my name, which sounds odd.

scscsc• 9 years ago

The title is terrible. Here is an excerpt from the article that explains what's going on: "The company's solution was to develop Git Virtual File System (GVFS). With GVFS, a local replica of a Git repository is virtualized such that it contains metadata and only the source code files that have been explicitly retrieved. By eliminating the need to replicate every file (and, hence, check every file for modifications), both the disk footprint of the repository and the speed of working with it were greatly improved. Microsoft modified Git to handle this virtual file system. The client was altered so that it didn't needlessly try to access files that weren't available locally and a new transfer protocol was added for selectively retrieving individual files from a remote repository."

This style of working is needed in large code bases, where not all files are checked out on developer workstations (for performance or privacy reasons).

gmueckl4• 9 years ago

Wait... so they turned got into a centralized abomination and added access control to parts of the source tree (A feature that SVN had from the start but decentralized VCS cannot provide)? Is this correct?

WorldMaker5• 9 years ago

It's still distributed, just not entirely 100% distributed. It's like the difference in which IPFS nodes have which items pinned or how many seeders you have versus torrenters (and the subsequent file availability in that cluster) in BitTorrent.

Tarean• 9 years ago

Iirc google and facebook decided against using git because it didn't scale to stupidly enormous code bases.

Microsoft decided to stick with git but add non-strict fetching to it. On the plus side, still all the advantages of easy branching/merging and working offline if you touched the required files which distributed vcs's bring. But you still need to be connected to work with parts of the code base you haven't used before.

So I guess if you run the test suite for your current task first then you can work offline since all relevant files will be fetched?

coldtea3• 9 years ago

What do you think is more possible?

(a) the result of the work of a major Git player like GitHub, and a major software company like Microsoft, with tens of excellent engineers devoted on it, and that solves a real pain point they have, is a centralized abomination that merely replicates a feature SVN already had

(b) your description is a crude knee jerk reaction

jcoffland2• 9 years ago

Man, I get sick of the incessant title criticism on HN. I wish the quality of the title wasn't such a go to topic. Unless the title is grievously misrepresenting, let's just discuss the topic.

The_Hoff• 9 years ago

Maybe we can make this the go to reply on these comments.

jasonkostempski• 9 years ago

But then we all miss out on the title criticism criticisms.

doublerebel2• 9 years ago

Yes the latest crowd has created some memes for themselves. I agree we need to extinguish them for the sake of continued quality HN discussion. Including:

- X software package name reminds/confuses me of Y product with similar name

- X title is garbage. Here is the title I would write...

- I don't understand [basic concept available on Wikipedia]. Someone explain it to me (aka lazyweb)

- I'm not an expert on this, but [completely unqualified and uncited conjecture]

- I also once did X from TFA and [unrelated personal anecdote with no insight] (aka long form "me too" comment)

I come to HN so that I can read informed discussion from people who work in the field of TFA. To read discussion of basic topics between the uninformed, there is everywhere else on the internet.

thomastjeffery• 9 years ago

There is a [-] that you can click to minimize any thread you do not wish to read.

mikestew3• 9 years ago

I sometimes wonder if it isn't almost literal bike shedding: I have nothing to contribute on the topic it hand, but I'm compelled to contribute, so I'll go on about the title. Obviously there are some egregious examples that are correctly called out, in this case, yeah, I saw "the title is terrible" and quit reading.

j_s• 9 years ago

If you missed the discussion with details about how the technology actually works, rather than xplat: https://news.ycombinator.com/item?id=13559662 linked Q&A https://www.reddit.com/r/programming/comments/5rtlk0/git_vir...

An open-source cross-platform virtual file system API that was also fast (as opposed to FUSE) would be amazing.

jonstewart• 9 years ago

It would indeed, although FUSE has the virtue of simplicity, and it would be nice not to sacrifice that.

emveeoh• 9 years ago

Seems like Microsoft is trying to brand a cached proxy for Git...? This would make more sense as a paid service for those that need it. Possibly, a local appliance that a company would host internally. What am I missing...?

WorldMaker5• 9 years ago

It's a virtualization layer for the Git object database (all the hash-named files in a repository's .git folder) intended for sparsely checking out large git repositories (lots of commit history, huge trees of files, etc), and only downloading the objects you need as you need them.

It defines a "CDN protocol" for downloading those objects as needed (which Bitbucket and GitHub are both supporting in various alpha/beta stages), which is essentially a cache offered as a paid service to big enterprise projects, but the GVFS project also has to make sure that git operates as efficiently as possible with sparse object databases, and implements how those sparse object databases work at all (which to this point was not something git concerned itself with, and partly why the work is being done as a filesystem proxy using placeholder files on the user's machine).

The project has included work in making sure that git commands touch as few objects from the object database as they can to get their work done (minimizing downloads from a remote server).

signa11• 9 years ago

isn't this similar to what was done by rational (the company thst ibm bought iirc) as part of their clearcase product ? my experience with that was not so great unfortunately. it seemed very clunky to use and i think required an army of system-administration folks to run.

johnhenry2• 9 years ago

Naming things is among the hardest things for programmers... what would you title it?

em3rgent0rdr• 9 years ago

And good naming is useful to others. So I think we shouldn't discourage useful debates on how article titles could be improved. That helps us improve our naming skills.

tarruda• 9 years ago

Did I misunderstood the idea, or is GVFS essentially making git a non-distributed VCS?

WorldMaker5• 9 years ago

It's more that it provides more control on the centralized<->distributed spectrum. Git by default, yes, is fully distributed with every copy of a repo supposed to maintain full history, etc. GVFS allows you the option to offload some of that storage effort to servers you specify. Those servers can be distributed themselves (similar to CDNs, etc), so there's still distribution flexibility there.

You can think of it as giving you somewhat flexible control of the "torrenter/seeder ratio" in BitTorrent: how many complete copies of the repo are available/accessible at a given time.

mikepurvis• 9 years ago

I think once you're checked out, you're still distributed as far as being able to make and push commits which modify the files you have. But obviously someone can't clone files from you which you don't yourself have.

See previous articles which discussions:

https://blogs.msdn.microsoft.com/devops/2017/02/03/announcin...

https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-large...

srcmap• 9 years ago

From reading the first post "The largest Git repo on the planet" - "Windows code base" of "3.5M files", "repo of about 300GB", "4,000 engineers" - I assumed that "Windows code base" contains all the utilties, DLLs that included in windows release such as notepad, games, IE/Edge etc and not just the Windows KERNEL.

If so, the comparable code base to check against is Android AOSP, Ubuntu, RedHat or FreeBSD.

If that is true, I believe the source code base for Ubuntu/RedHat distribution with all the Apps likely be bigger compare to Windows in term of number files, source repo and number of engineers (open source developers for all the packages such as ff, chrome openoffice.)

Microsoft folks feel free to correct me here.

It seems that the existing git's process, dev model seems to work well for much bigger projects already by using different git repo for each apps.

Still not sure what pain point does the new GVFS solve.....

wildmusings4• 9 years ago

It’s just a pragmatic recognition that most files are going to stay unchanged relative to some “master” repository anyway. It maintains the same distributed semantics as vanilla git, but allows you to use less disk space if you choose to rely on the availability of that master repository.

acdha• 9 years ago

I've been thinking of it as more of making it on-demand — it doesn't sound like you lose the underlying distributed nature (e.g. from the description on https://blogs.msdn.microsoft.com/bharry/2017/02/03/scaling-g... it sounds like 'git push ssh://github.com/microsoft/windows.git' would work, although it might take longer to run than it does for you to be fired) but managed clients can choose to operate in a mode where they download everything on-demand rather than immediately.

That seems like a reasonable compromise for a workload which Git is otherwise completely unable to handle.

my123• 9 years ago

It actually takes a bit more than 24 hours... so yeah

snowwrestler• 9 years ago

It's worth asking: how many Github users are actually using git as a fully distributed version control system? The typical Github workflow is to treat the Github repo as a preferred upstream--which sort of centralizes things.

frabbit3• 9 years ago

   The typical Github workflow is to treat the Github repo as a preferred upstream

Is it typical? What metrics do you have to support that?

Speaking for myself I have only ever used triangular workflows: fork upstream; set local remotes to own fork; push to own fork; issue pull request; profit

https://felipec.wordpress.com/2014/05/11/git-triangular-work...

rtpg• 9 years ago

I think a common open source workflow is to have a forked GitHub repo and working off of this.

So in the end you have your local repo, your fork, and the organization repo. During a pull request process third parties might make pull requests to your fork to try and fix things

alkonaut3• 9 years ago

Depends on what you mean by distributed. If a repo contains 2 projects pA and pB and I’m only involved in the former while the other project has a 2GB binary asset with 1000 historical revisions, then I’m happy to just be distributed wrt my part of the repo.

To put it another way: A server with multiple repos on it is a central server with respect to the repos you don’t clone! This is the same, but for parts of repos.

jwilk3• 9 years ago

For the avoidance of doubt, GitHub is not upstream of Git.

tomxor2• 9 years ago

yeah this was really confusing in the article everywhere it mentions up-streaming it refers to "git" only then later mentions how GitHub is accepting it's modifications... so not entirely sure which it is.

TL;DR pretty sure it's not touching git source at all.

Looking though the GVFS readme: https://github.com/Microsoft/GVFS it appears that it actually wraps git, which would make sense because in that case they would have to explicitly add support to each git host and provide individual platform ports of GVFS (all of which would be unnecessary if it was actually upstreamed into git).

So that's good... git is one of my favourite open source tools and I would really hate for M$ to start polluting it. I don't care if GVFS is a good idea or not I just don't trust the fuckers and they will always deserve that suspicion.

hdhzy3• 9 years ago

No, they needed to do some modifications to the git suite of tools. Generally git expects all objects to be on disk and Microsoft wanted to have sparse checkouts of files in a specific revision.

Not really polluting but rather having some objects be fetched only on demand.

Source: https://blogs.msdn.microsoft.com/devops/2017/02/03/announcin...

In addition to the GVFS sources, we’ve also made some changes to Git to allow it to work well on a GVFS-backed repo, and those sources are available at https://github.com/Microsoft/git.

For the record as far as I understand GVFS the article is correctly using git vs Github.

crankylinuxuser• 9 years ago

Does this mean we have to fdisk/format if we forget the commands to correct a broken git repo?

https://xkcd.com/1597/

I seriously did that first learning Git. And there's plenty of niches and side-cases that I'm still not quite sure of. Going from client-server to distributed has a level of complexity that usually isn't discussed until you implement.

EDIT: And further understanding is, this provides a GIT filesystem based connection so that one can work on a multi-TB repo without downloading everything locally.

This seems to be the result of choosing to have all the software in one OMG-sized repo, rather than 1 project/repo. And evidently they need a "keep on server" for this. Makes me wonder more why they even went with Git or a distributed model at all. This seems more like they 'screwed up and now have tons of bandaids'.

0xfeba• 9 years ago

Yep, I've done that.

aeorgnoieang3• 9 years ago

Lookup the history of GVFS and why Microsoft chose Git. There are interesting reads about it.

There's also lots of interesting reads about 'monorepos' and their pros and cons. Note tho that Google and Facebook, and lots of other companies, use mono-repos. It's not just Microsoft and, not surprisingly, they've all made (more or less) reasoned, thoughtful decisions taking into account lots of factors that almost no one else in the world would ever think to do.

chungy• 9 years ago

Anyone else get superconfused and think about the GNOME Virtual Filesystem?

sam0x172• 9 years ago

https://github.com/Microsoft/GVFS/issues/7

awalton• 9 years ago

Having worked on GVFS (the GNOME one) in the past... every single time.

Maybe GNOME should start trademarking project names too :/...

Nelkins• 9 years ago

I wonder if this is a precursor to open-sourcing Windows development...

gmueckl4• 9 years ago

No. Open Source and walled gardens don't mix. MS wants to turn Windows into the later, obviously.

sam0x172• 9 years ago

And they are still calling it GVFS -- a name that is already used by a prominent linux technology. https://github.com/Microsoft/GVFS/issues/7

zellyn• 9 years ago

Every single hacker news discussion these days seems to have a complaint about reusing a name.

Every name is taken. There is nothing to be done about it. It's okay. I mean, it's sad, and terrible, but it's also fine.

camgunz• 9 years ago

They are definitely not all taken

jcoffland2• 9 years ago

I can't help thinking, there is more to this than is discussed in the article. This tech can also be used to give developers selective access to a git repo. It lets Microsoft and other commercial software vendors get on the git bandwagon with out actually sharing all the code with all the developers. Each git repo is a complete copy because git was designed to support Open-Source. In Open-Source there are no hidden parts. But that's a hard pill for corps like Microsoft to swallow. This looks more like a solution to meet developer demands to use git and also satisfy the demands of management to maintain control of their team's code based.

Sevii• 9 years ago

Got vfs is more motivated by Microsoft wanting to use git as a monorepo. I think they have around 2k windows devs on one git tree with this vfs.

philipov5• 9 years ago

Are there advantages to using a monorepo rather than having a proper package manager? Seems like the problems they're trying to solve would be better solved by not insisting on using a monorepo.

binthere• 9 years ago

As stated in the article, git is also challenging for game development because of the large asset files. Git has limitations and they are part of the reasons a lot of companies and even indie game devs didn't jump into it (Jonathan Blow, John Carmack, being some of the critics). It's more about speed, simplicity, and efficiency rather than "Corp" vs "Open Source" conspiracy.

KallDrexx2• 9 years ago

Git's issues with game dev isn't just because of large file size though. Binary files are hard to control in a DVCS because if two people spend 24 hours to modify a binary asset at the same time you can't just merge the changes together like with source code. Someone has to manually go in and replicate both changes together.

muricula• 9 years ago

It's not about maintaining control -- downloading any one of those files is as easy as running `touch` on them. It's about only checking out which pieces you actually need to build the component you care about. That way I don't need to download the whole IE source to build the NT kernel. (I just checked code into windows yesterday)

paradite• 9 years ago

Uhm... the author seems to be confused about the relationship between Github and git.

GitHub doesn't develop git, it's a service built on top of git. Any modifications to git has nothing to do with GitHub. GFVS is a GitHub project, not part of git.

aeorgnoieang3• 9 years ago

GitHub doesn't develop git

I'd be surprised if none of their employees have contributed to Git. I didn't interpret anything in the article as saying that GitHub develops Git, in its entirety (or even largely).

GFVS is a GitHub project

GFVS is a Microsoft project and GitHub seems to be contributing. And the GFVS developers have been submitting, successfully, their changes to Git itself upstream to the actual Git project. So it is becoming a part of Git.

jwilk3• 9 years ago

I didn't interpret anything in the article as saying that GitHub develops Git

"Microsoft [...] wanted to get these modifications accepted upstream and integrated into the standard Git client.

That plan appears to be going well. Yesterday, the company announced that GitHub was adopting its modifications and that the two would be working together to bring suitable clients to macOS and Linux."

This hints that Git upstream = GitHub. I mean, why mention GitHub at all if they aren't upstream? The rest of the article doesn't explain GitHub's role in this story either.

jwilk3• 9 years ago

I'd be surprised if none of their employees have contributed to Git.

In git.git, there are 7 commits with @github.com authors, last one from 2014.

(I hope it just means GitHub folks are trying to blend into the crowd by using personal addresses, or something...)

js2• 9 years ago

I'd be surprised if none of their employees have contributed to Git.

Jeff King (peff) has worked for GitHub for a long time:

https://github.com/git/git/graphs/contributors

Michael Haggerty (mhagger) is also a GitHub employee.

So, yes.

Mraedis• 9 years ago

Small comment, it's GVFS, not GFVS,. FileSystem, FS always at the back.

Yen3• 9 years ago

GitHub may not be the primary contributor to git, but they're certainly a significant user of it, and they _have_ contributed patches to upstream git.

(example, the '768 conflict' bug here: https://githubengineering.com/move-fast/)

dfabulich• 9 years ago

Microsoft and GitHub are also working to bring similar capabilities to other platforms, with macOS coming first, and later Linux. The obvious way to do this on both systems is to use FUSE, an infrastructure for building file systems that run in user mode rather than kernel mode (desirable because user-mode development is easier and safer than kernel mode). However, the companies have discovered that FUSE isn't fast enough for this—a lesson Dropbox also learned when developing a similar capability, Project Infinite. Currently, the companies believe that tapping into a macOS extensibility mechanism called Kauth (or KAuth) will be the best way forward.

I'd love to read more technical details on this. How can Kauth support a virtual filesystem?

tankenmate2• 9 years ago

"Git wasn't designed for such vast numbers of developers—more than 20,000 actively working on the codebase. Also, Git wasn't designed for a codebase that was so large, either in terms of the number of files and version history for each file, or in terms of sheer size, coming in at more than 300GB. When using standard Git, working with the source repository was unacceptably slow. Common operations (such as checking which files have been modified) would take multiple minutes."

These sentences in some parts gloss over the details, and in others it is flat out wrong. Git was designed for tens of thousands of developers (the Linux kernel), it was designed for huge numbers of files (but large numbers of files works fine on Linux due to the dentry cache, it sucks on Windows because they don't have a cache that has the same behaviour as the Linux dentry cache). Admittedly it is slow for files that are large in size, but it was designed for source code; what sane developer would have source files that are hundreds of MB in size?

That said, monorepos with several dozen software projects can be very slow due to O(n) functions such as git blame and the like. It's true that git wasn't designed for large numbers of projects in one repo that are only indirectly shared by library code and the like.

Unfortunately that was not written in the article.

godzillabrennus• 9 years ago

Your comment reminds me that Microsoft should really double down on replacing NTFS. Even Apple has moved off HFS+ with their latest OS.

sekh60• 9 years ago

They unfortunately removed the ability to create ReFS disks from Windows 10 Pro and lower. Hopefully they reverse that decision soon. That said I'm not sure how stable it was considered.

alkonaut3• 9 years ago

I develop a desktop app with lots of test documents (test inputs, outputs) that are tied to specific revisions in order to work with the test suite that uses them. None are huge but there are many and some are frequently changed (2GB working copy, 15gb repo, 100k commits). They could be put in some (very) complex hierarchy on a network share, but having them in vcs is so convenient. If rev 9999 changes the code so that the expected output testfileA differs then you update the expected output in the same commit. If you don’t have the file and you want to be able to clone+build+test any revision then you still need the results for 9998 also on the share.

This scenario is nicely handled in svn, mercurial, tfvc and similar. If this lets git do it then it looks like a great idea.

Game developers with lots of churn in huge binary assets also have the same problem, but probably even worse.

mkobit• 9 years ago

Those kinds of use cases are what Git LFS [1] was created to help with. The major Git vendors support it (Bitbucket, GitHub, GitLab). I'm not as certain about the open source implementations.

[1] https://git-lfs.github.com/

erikbye• 9 years ago

Game devs just use Perforce (for binaries, or everything). Sometimes Git in combination.

Permit• 9 years ago

These sentences in some parts gloss over the details, and in others it is flat out wrong. Git was designed for tens of thousands of developers (the Linux kernel),

Are there really ten thousand active developers on the Linux Kernel?

According to [1] there were ~ 2,000 over a 15 month period. It's an order of magnitude difference that seems like it would have an impact.

[1] https://arstechnica.com/information-technology/2015/02/linux...

derefr• 9 years ago

And even then, the kernel doesn’t follow anything like the approach to distributed development that your average Github project does. The kernel consists of a bunch of maintainers who each have their own intake branches, to which other people submit PRs or patches as that maintainer wills; and then the mainline kernel, which only takes contributions via patches from one of the maintainers.

The nearest thing you could compare the development model to is the set of base packages for the BSDs: each subsystem/package has its own upstream development (like the Linux subsystems with their maintainers); and then Linus serves as the release engineer, taking updates to those packages and merging them into the release. (I compare to the BSDs specifically because Linux kernel subsystems, like non-cross-platform BSD base-system packages, have no other life outside of their potential integration into the release. People aren’t using the upstreams directly, other than in development of the new feature.)

tedivm• 9 years ago

The article above was actually wrong and has been updated to show that Microsoft has 3,000 active developers instead of the original 20,000 that is quoted here. So it's only a difference of 50%, not a full order of magnitude.

wildmusings4• 9 years ago

The Windows repo is massive, at least 100GB. Consider that the component equivalent to Linux, the NT kernel, is but a small part of the whole thing. And it is indeed a single large project, where every commit has to leave the entirety of Windows in a good state. So if you change some widely used function, you need to fix every single caller in all of Windows. Also, the build system evolves along with the product, so it is checked into the repository as well. The right version of the build system is whatever is in the timestamp you are building. So there’s your seemingly elusive good reason for keeping large binaries in the repo. And while I worked there, the object files from the nightly build were also checked in, so you could rebuild mostly-unchanged components quickly.

The alternative to doing it this way is to introduce version management between components, which is totally unnecessary complexity. Windows is one product and all of the components (or subsets thereof) ship together. As far as internal-only interfaces are concerned, version N does not need to work with N-1 or N+1, only N. That is a massively simplifying assumption that makes a lot of things much easier.

There is a reason why all the big tech companies operate this way. Microsoft, Google, and Facebook, all have one giant repository with their entire product.

dingo_bat2• 9 years ago

There is a reason why all the big tech companies operate this way. Microsoft, Google, and Facebook, all have one giant repository with their entire product.

Cool, but that doesn't explain why Office and Windows need to share the same repo. Ditto for gmail and chrome os. Ditto for fb messenger and whatever face recognition algorithm they use.

InclinedPlane3• 9 years ago

I worked at a game company for a while and one of the interesting things there was that there really wasn't a good way around checking in binaries to the repo. First off, a lot of your "source" assets are properly in binary form (images, for example), because that was the most compact form for them to exist, by orders of magnitude, and also because trying to generate them from some other non-binary representation would add tremendous overhead to development work. On top of that, building the entire product from "source" might involve tens to hundreds of thousands of hours of CPU/GPU time. So you just didn't do that. Instead you built everything in parallel and checked the results in somewhere, that would then get bundled together into the final product. There are ways around doing things this way, for sure, but given that you can just go pay a fairly insignificant amount of dollars to some company and get a VCS that won't choke.

jsmthrowaway• 9 years ago

Tens of thousands of developers pushing changes is not the same as the Linux development process, at all. The key is in the word “actively.” Even Perforce couldn’t keep up with that type of load in a similar situation with Google’s monorepo, forcing them to develop a Perforce-compatible alternative.

It’s tough to scale VCS to very large teams, regardless of the software. No need to stand up for git.

perfmode• 9 years ago

IMHO it is a major failure that git doesn’t scale for org-wide mono repos. Software development process should not be dictated by limitations in the VCS. It surprises me that others are so willing to settle.

_s• 9 years ago

Would you be willing to elaborate on the pros/cons of mono repos vs what git encourages? It seems to just be the wrong fit for a given use case, but works very well others.

jonhohle2• 9 years ago

Is there a VCS that does?

When I was at Amazon, Perforce couldn't scale (it had already been partitioned into a few depots, but those had reached their reasonable capacity). Amazon moved to a per "module" git-repo system. It was convenient to push across multiple repos simultaneously, but it wasn't a deal breaker when we lost it (I never heard of any workflows being completely unworkable following the transition).

jlgaddis• 9 years ago

IMHO it is a major failure that git doesn’t scale for org-wide mono repos.

git was (initially) written by Torvalds to serve his own purpose -- he needed a new VCS for Linux. Let's say that it "doesn't scale for org-wide mono repos". Unless that was a design goal (and I'm pretty sure it wasn't), how is that a "failure"? It apparently works just fine for the kernel developers.

Yet because it didn't work well for Microsoft's 300 GB worth of tens millions of lines of code it is somehow a "failure"?

Apparently Microsoft's own TFS is a failure too, then, because it seems that it wasn't up to the task either.

hasenj4• 9 years ago

What I find more surprising is the length they went through to make git work for them.

It would probably have been better if they had assigned a team of highly skilled engineers to design a distributed source control from scratch specifically to solve the problems faced by the windows team.

After all, `git` itself was developed for Linux because Linus was not satisfied with any of the existing solutions.

geofft4• 9 years ago

in others it is flat out wrong. Git was designed for tens of thousands of developers (the Linux kernel),

This is easy to check, and the article is right. The Linux kernel does not have tens of thousands of active developers. Over a little more than the last year:

    titan:~/src/linux geofft$ git log v4.8..v4.14 --format='%aE' | sort -u | wc -l
    4857

If you look at just the most recent release cycle:

    titan:~/src/linux geofft$ git log v4.13..v4.14 --format='%aE' | sort -u | wc -l
    1839

Alternatively, if you look over the last ~year, but at people who have more than five commits (chosen arbitrarily):

    titan:~/src/linux geofft$ git log v4.8..v4.14 --format='%aE' | sort | uniq -c | awk '$1 > 5' | wc -l
    1814

A codebase with 20,000 actively working on the code base is at least an order of magnitude more than the Linux kernel. If they're committing patches every single day, probably more than that. I have, last I checked, four commits in Linux: two for a summer internship in 2009 and two doc fixes. That certainly contributes to the scale of Linux the kernel in the sense of "there are so many people working on it!", but not in the sense of being an "active developer" of the git repo. I was fine sending in my patches to a mailing list and hearing back at some arbitrarily later point, not triggering any CI, not asking anyone else to collaborate on my work, etc. That doesn't work if you have 20,000 engineers on the same codebase writing code every day.

Admittedly it is slow for files that are large in size, but it was designed for source code; what sane developer would have source files that are hundreds of MB in size?

Sometimes that's the best way to get things done? The best Debian workflow I've ever worked with (and I've worked with a lot) involved actually committing binary .debs to SVN alongside their source code, because that meant that the SVN revision number was the single source of truth. There wasn't some external artifact system, nor was there a risk of picking up the wrong packages from an apt repo when rebuilding an older SVN revision.

I'm not defending this as pretty. But I will defend this as sane. It got the job done reliably and let me work on actually shipping the product and not yak-shaving workflows.

hitekker• 9 years ago

+1 for the research and the distinction between sanity and purity.

opmac• 9 years ago

Good stuff. Came into this thread expecting the top comment to be a know-it-all answer trying to "debunk" the article, as so commonly seen on HN. This is a perfect, well-researched response.

pfranz2• 9 years ago

actually committing binary .debs to SVN alongside their source code, because that meant that the SVN revision number was the single source of truth

This is totally armchair-devops, but it sounds like a more ideal case would be CI that runs based on your commit. In my experience, developers get lazy and are likely to commit binaries that dont match the code or miss some dependencies or steps.

I completely agree with your point, using the tools as intended isn't always the best way to get things done. Video games are an easy example, code is often dependent and intermixed with binary assets and you want those to be versioned together.

kelnos• 9 years ago

I think this is all missing the point. Do they want to store, say, the entirety of the base Windows OS in a single git repo? If that's the case, sure, git isn't a great fit for their use case, and they should either use something else, or find a way (as they're doing?) to change git to fit their needs.

I can't see MS having tens of thousands of developers working on any single component of Windows, so I'm guessing they _do_ want a giant monorepo. If they were to break it up into separate repos per OS component, I doubt they'd have scaling issues (of course, breaking things up introduces coordination and dependency issues).

exikyut2• 9 years ago

Question: what's the `sort -u` for? You're piping to `wc -l`.

trisimix2• 9 years ago

Dude teach me how to pipe liKe you

sambf• 9 years ago

The article was modified to "— more than 3,000 actively working on the codebase".

ethomson9• 9 years ago

I agree that these sentences gloss over the details - and rightly so: it's an article in Ars, after all, not a journal article. We (Microsoft) have published a lot[1] about our journey to put Enterprise-scale codebases, like the Windows codebase, into Git. I think you'll find that they expand upon some of the things that are glossed over.

And I would encourage you to read them, so I won't belabor the details here, but to give you just one example: git blame is not the issue that we're most concerned about. The basic, highest priority functionality in Git - things like git add - are the the issues we're most concerned about.

git add is O(n) on the number of files in the repository. When you run `add`, git reads the index, modifies the entry (or entries) that you're changing, and writes the index back out.

The Windows repository has about 3.5 million files[2]. Running `git add` - when we started thinking about moving Windows into Git - took minutes.

Can you imagine running `git add` and having it take 10 minutes?

Now obviously there are some inefficiencies here - there's quadratic operations and the like that went in assuming that nobody would ever put 3.5 million files into a Git repository. And we've been cutting those out over time[3].

Thankfully, Git does have some functionality to support very large repositories - using shallow clones, sparse checkouts and the like. We've added narrow clones, to only download portions of the tree, and automation to handle this automatically without user intervention.

That's the scaling work that we're doing with GVFS. And these changes bring the P80 time for git add down to 2.5 seconds[4]. We've been contributing these changes back to git itself, and we're thrilled to work with industry partners like GitHub who are also interested.

Sorry to go on - version control is a passion of mine, as it is for many of us working on Visual Studio Team Services. Your conclusion is very much correct: Git wasn't designed for this from the beginning. Thankfully, software isn't immutable, so we're scaling it up.

[1] http://gvfs.io/

[2] https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-large...

[3] https://blogs.msdn.microsoft.com/devops/2017/05/30/optimizin...

[4] https://blogs.msdn.microsoft.com/devops/2017/11/15/updates-t...

jopsen• 9 years ago

Hmm, some part of me wouldn't mind, a FUSE setup where: /repository/github.com/<org>/<repo>/ is automatically cloned out from github when accessed the first time :)

It would be pretty cool to use as a $GOPATH. Or just for making drive-by github contributions super easy.

zimbatm• 9 years ago

I built https://github.com/zimbatm/h to satisfy the same need but it's a command-line tool, not a FUSE filesystem. In some way I prefer it because transparent network access doesn't work really well with failure modes.

Usage: `h <owner>/<repo>` clones the repo, if it doesn't exist, and takes you there.

This and hub are integral to my developer workflow:

    $ h some/repo
    $ git checkout -b some-branch
    [hack and commit]
    $ hub fork
    $ git push -u zimbatm
    $ hub pull-request
    Done

hinkley2• 9 years ago

It's all fun and games until someone like me tries to run `find` on the parent directory...

silentOpen• 9 years ago

Children would be lazily loaded and would not appear in directory listing of the parent directory. `find` would search your existing local checkouts only and the file system would still be POSIX.

rokob• 9 years ago

Interesting to see them try to fix Git when Facebook considered it (due to similar constraints) and decided to go with Hg instead: https://code.facebook.com/posts/218678814984400/scaling-merc...

jng• 9 years ago

Microsoft is used to implementing its own filesystems, so it probably feels more natural for them to do that than it does for Facebook.

glandium• 9 years ago

They decided to go with mercurial because it was possible to monkey patch its guts from extensions written in python. They're now talking about implementing parts in rust, which, ironically, would have prevented them from doing what they originally chose mercurial for.

masklinn2• 9 years ago

They decided to go with mercurial because it was possible to monkey patch

The ability to extend, swap and replace deep into the stack was always a core component of Mercurial, hence having to enable extensions for things many users would consider core features e.g. terminal colors or graph log

Facebook didn't hack around like monkeys, they built extensions. And when they could not do it as extensions, they upstreamed improvements to the core.

The alternative would have been to fork the codebase entirely.

Monkey patching the internals would have been significantly less maintainable: having used it as a library, I can tell you that none of the internal stuff is considered stable and pretty major components will change between point releases (I was using the diff-parsing and patch-application routines for something else, the API changed basically every minor release, forking would at least give you a heads-up conflict when upstream changed, monkey-patching would either blow up at runtime or not go through the patch anymore)

They're now talking about implementing parts in rust, which, ironically, would have prevented them from doing what they originally chose mercurial for.

No.

ethomson9• 9 years ago

Microsoft PM here. It's important to note that Hg also had similar constraints. So Facebook was in a position of having to fix Git or Hg to suit their needs. It turns out that for a variety of factors, Hg was the better choice for them to hack on.

Microsoft took the same look and went the opposite direction, and decided that Git was easier for us to hack on. This was a no brainer for us. We build multiple tools that host peoples Git repositories (TFS and VSTS) and have Git repositories as deployment mechanisms for Azure. We have several contributors to tools in the Git ecosystem on staff, including people contributing to core git, and the maintainer for Git for Windows, libgit2 and LibGit2Sharp. But we have comparatively little institutional knowledge of Hg.

That post you linked is awesome. Facebook has done a lot of really impressive work scaling Hg and a lot of the lessons learned at Facebook and implementations in Hg are very similar to what we've done with Git.

This video from Durham Goode on Facebook's version control team (from Git Merge, amusingly) is also awesome: https://m.youtube.com/watch?v=gOVD-DrUpwQ

normaljoe• 9 years ago

It's really good to see Microsoft join in the open source community. I switched to Mac over a decade ago largely because of the Microsoft only mentality back then. Really makes me consider switching back.

Koshkin• 9 years ago

To me it looked like one of the points of Git was that one could work on anything from the entire repository locally, i.e., for example, without having an internet connection, e.g. when on an airplane. (I think having one huge repository is the problem here.)

jaccarmac3• 9 years ago

Well, Git was indeed created to serve Linux kernel development. So decentralized access for the people writing patches all over the world was useful. However, when companies started using source control for their monorepos which developers access through a work network or VPN decentralization takes a backseat to business concerns. Different use cases and all that.

secstate• 9 years ago

I think the problem there, is that you then basically wind up with Git on the backend and an SVN-like layer over the front. Which seems a bit silly. SVN is a very robust project that works really well. Why not add git-like porcelain to SVN if you're gonna force devs to have an network connection so they don't have to download a giant git working directory.

Smart people are working on all this, so I'm sure there are reasons, but in the all the instances where I've had to interact with a monorepo, it was because the tech debt was too high to pay off to break it apart, not because it was better.

And if you're indebted to the point where you have no point in paying it off, you damn well better have leveraged assets against it (i.e. a cash cow of a business, like MS Word or Facebook)

dragonwriter• 9 years ago

But there are source control systems designed around that centralized use case. Lots of them. One of which Microsoft owns.

Why spend so much effort bashing a distributed peg into a centralized hole?

jeanmichelx• 9 years ago

This is only tangentially related, but is there any standard advice to get git on a remote file system (ceph) to run fast with a long history?

gumby• 9 years ago

A couple of thoughts as the implementation downloads mainly placeholders:

that mechanism can also be combined with permissions on the server side so contractor X who is fixing a driver need only have access to the driver and some supporting code while still checking into the monorepo.

It also means that every developer’s laptop won’t have a full copy of the repo (so if said laptop is lost, the risk is more contained)

And it’s similar to Part of LFS, bigfiles and other “blob” add-o s to git...could see in future a way to put .doc files and the like into repos.

drdrey• 9 years ago

Sounds a bit similar to Google's Piper (https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...) that lets you check out a portion of the monorepo, no?

Myrmornis• 9 years ago

The open source, free GitHub hosting doesn't need the scaling work Microsoft has done

They might want to fix that sentence. It reads as saying that Github is open source. Perhaps it should read

The free GitHub hosting that is typically used by open source projects doesn't need the scaling work Microsoft has done

exabrial• 9 years ago

This is kinda funny, it undoes one of the original pillars of why git was made

nerdponx2• 9 years ago

It's a little surprising to me that these companies use Git at all. What's the appeal of Git really? Is it just the "familiarity factor", and is that really a reason to pour so many resources into it?

dingo_bat2• 9 years ago

I think almost all the people they hire nowadays would be familiar with git and not something else. So it would be a lot of time savings in training if you switch to git instead of training every newbie to your own peculiar source control mechanism.

nerdponx2• 9 years ago

Sure, but the CLI is the CLI. I can't imagine there's no way to replicate the Git CLI without rewriting the internals. Surely the basics of branch-and-merge will be the same, right? I doubt anyone would miss the dangerously overloaded `checkout` subcommand, or wrestling with rebase conflicts.

OOPMan• 9 years ago

Personally, I imagine low-cost in-place branching is useful to most people doing serious work...?

mg74• 9 years ago

"The biggest complexity is that Git has a very conservative approach to compatibility, requiring that repositories remain compatible across versions."

This, above all else. That I never, or anyone else on our team, ever has to worry about wether they are running an "older version" of git is such a win.

gear54rus• 9 years ago

Big problem. Just update your machines. Am I supposed to miss great features because of your admin laziness?

It has to fail gracefully on version mismatch, other than that it's just a bad decision.

xyzzy_plugh• 9 years ago

It's not really a surprise that monorepo advocates struggle here. Compatibility through versioning and stable APIs is challenging and requires discipline.

Making wide-spread changes to all clients of your code when you make changes that break things doesn't require discipline.

iceman2654• 9 years ago

Does anyone else think this is another "Embrace, extend, and extinguish" move from MS? I don't know enough about the changes make that judgement.

m12k• 9 years ago

It seems that lazy-downloading files isn't all that useful if you need to compile all the source code. Does anyone know which build tools can be used to avoid having to open most of the source files - maybe some centralized build cache?

Microsoft and GitHub team up to take Git virtual file system to macOS, Linux

Related Stories