Supercharging the Git Commit Graph

https://blogs.msdn.microsoft.com/devops/2018/06/25/supercharging-the-git-commit-graph/

by ethomson • 8 years ago

227 82 8 years ago

blogs.msdn.microsoft.com

slobotron• 8 years ago

Interesting tidbit:

The developers making Microsoft Windows use Git

xkcdefgh• 8 years ago

Yeah, i believe it was a huge effort some time ago to migrate windows to git

LyalinDotCom2• 8 years ago

Yep we use Git to build windows! here is the story that was told about our efforts.

https://arstechnica.com/gadgets/2018/03/building-windows-4-m...

01100011• 8 years ago

Kinda surprising. I know everyone seems to love git these days, but I find it's really better suited towards distributed and/or smaller projects. After feeling the pain of a megarepo system at work, I'm pushing to switch to a monorepo(well, like 4 repos instead of 200). git sort of sucks for monorepos.

Also, even after learning a fair amount of git, I still find I spend a noticeable amount of time dealing with it. I don't remember spending a lot of time on any of the VCS systems I've used in the past. They just stayed out of my way and let do my thing.

rtcoolaid• 8 years ago

You are actually right and down voters have no clue (and this is what I hate about HN, if you don't comment then don't downvoye)

To answer your question though - Microsoft has a lot of git extensions that they are slowly submitting upstream. Hence git is usable for their megarepo.

Look up GVFS on GitHub, very cool work.

Same for Twitter (they are not submitting upstream though as far as I know)

Similar for fb/mercurial

adrianN4• 8 years ago

I do a lot of things that aren't really possible in older VCS with git, which is why I spent more time with it.

pdpi• 8 years ago

git sort of sucks for monorepos.

Microsoft works around several of the issues there by using GVFS. Also, at Microsoft scale, everything "sort of sucks", there's just no silver bullets. You take one of the least bad options and put all the effort you can towards making it work as well as you can.

MaxBarraclough• 8 years ago

I find it's really better suited towards distributed and/or smaller projects

But it was specifically designed for the Linux kernel.

yoklov2• 8 years ago

Dang, this made `git log` inside the repo I use at work (which is enormous with a stupid number of commits) nearly instantaneous. Great work.

`git status` still takes over a second for me though, oh well...

fanf2• 8 years ago

Have you tried the fsmonitor hook feature that was added in git 2.17? https://blog.github.com/2018-04-05-git-217-released/

yoklov2• 8 years ago

Yep, as well as the untracked cache. I also was using a split index for a while, but it didn’t play nicely with some tools...

erikb• 8 years ago

Everybody who seriously uses git will also want to see the graph somehow. Here's how I do it: https://github.com/erikbgithub/dot-files/blob/master/.gitcon...

bluebluetimes• 8 years ago

To enable the commit-graph feature in your repository, run git config core.commitGraph true. Then, you can update your commit-graph file by running

‘git show-ref -s | git commit-graph write —stdin-commits'

how do you automatically update this?

rakoo9• 8 years ago

It's not perfect but doing it in a pre-push hook can be useful. Unfortunately there's no post-receive hook on client-side, which could have been useful in this situation...

masklinn5• 8 years ago

It's not perfect but doing it in a pre-push hook can be useful.

That's completely unnecessary and way too frequent. Some where else (reddit I think?) the authors noted that they'd like to have it run alongside GCs in the next version. So running it as pre-auto-gc is a better idea.

stolee3• 8 years ago

We are working to make this run automatically in the future. https://public-inbox.org/git/20180627132447.142473-1-dstolee...

You don't need to run this with every commit, but maybe once after a big fetch or right after a new clone.

pmarin• 8 years ago

This is basically what Fossil call timeline or I'm missing something?

https://www.fossil-scm.org/index.html/timeline

masklinn5• 8 years ago

It's not a UI feature, the UI feature has existed forever (it's the log and log graph). This is a cache for the graph so that it does not have to be rebuilt every time it's displayed.

rakoo9• 8 years ago

Yet another file in the .git directory. The work is impressive and certainly helpful, but I can already hear Fossil proponents say "just use SQLite", which is getting more and more true.

Aissen2• 8 years ago

I had to bite… The sqlite repository has only 20795 commits since May 2000 at the time of this writing: https://www.sqlite.org/src/timeline?udc=1&ss=m&n=100000&y=ci

This is the amount of commits that goes into Linux every ~5months.

Has anyone done any meaningful performance comparison between fossil and git?

adrianN4• 8 years ago

The number of commits is not a very meaningful metric.

reificator• 8 years ago

How does the number of commits in the SQLite repository make it unsuitable as alternate store for git metadata?

superflyguy• 8 years ago

Use sqllite instead of git? Or git should use sqllite? If the latter then one problem is you'd need to keep your own fork forever as they don't accept patches. I'm not sure if that's a price worth paying to reduce the number of files git uses. Why is this a problem for you, anyway?

girvo• 8 years ago

They mean to use SQLite instead of the .git folder and file structures, I believe.

rakoo9• 8 years ago

It's not a problem for me, because all I see is the different commands that _use_ the underlying infrastructure. It's more about the design that was chosen: if you want to speed up things with git you have to implement specific logic in application code that will write a file and will need to update it periodically to keep it up-to-date, instead of using a querying engine made specifically for this purpose.

I don't see git ever changing its file format, but I do see another tool that imports everything from git and gives you a read-only sqlite db where you can do whatever you want, including displaying a graph quickly as the post advertises.

rhencke• 8 years ago

I love SQLite, and Fossil is very cool, but I don't see the fundamental difference between Git adding another file in the .git directory, and Fossil adding another table or index in the SQLite database.

rakoo9• 8 years ago

This is about having a single, unified interface for all operations. This is all explained in great details by SQLite itself at https://sqlite.org/appfileformat.html

farresito2• 8 years ago

To save people some time, an alias for your .gitconfig

create-graph = "!f() { git show-ref -s | git commit-graph write --stdin-commits ; } ; f"

masklinn5• 8 years ago

How does it save time, given this command will rarely be run, and next version will automatically run it on GC? Instead of copy/pasting a command you now have to copy-paste that same command with more gunk added and run it separately.

farresito2• 8 years ago

You have to run it on every repository you want this in, don't you? At least, that's what I understood.

wscott2• 8 years ago

"Before I joined Microsoft, I was a mathematician working in computational graph theory. I spent years thinking about graphs every day, so it was a habit that was hard to break. "

As a former BitKeeper developer, this is a key person for Microsoft to have on hand to improve git. BitKeeper got about 10X faster after it was used on the Linux kernel and many of the key performance wins were due to better graph traversal algorithms. Rick was a wizard at that sort of thing. The other was memory layout optimizations and caching. (my contribution)

So Rick made sure we walked the graph as little as necessary and I made sure the graph had an extremely compact representation containing as little information as possible and then once a target commit is found it is looked up in another store.

However, this quote was disturbing. "There are a few Git features that don’t work well with the commit-graph, such as shallow clones, replace-objects, and commit grafts. If you never use any of those features, then you should have no problems!" The joy of not having to write commercial software! Reading between the lines it appears they changed the default output order for 'git log' or some internal API and then didn't bother to fix the cases that depending on the old order.

stolee3• 8 years ago

Sorry for the worrying note about the experimental feature. One issue when working in open source is that contributors don't have control over the release cycle, and review requires smaller series than having the feature be delivered all at once.

These interactions with grafts, replace-objects, and shallow clones are one reason 2.18 does not create and manage this file automatically. The commit-graph file works by representing the commit relationships in a new file, and if that file exists, we treat that as the truth. The commit grafts, replace-objects, and shallow clones use another set of special files to track commits whose parents have been modified in special ways. If you would like to see our progress on integrating these features together, please see this thread on the Git mailing list: https://public-inbox.org/git/20180531174024.124488-1-dstolee...

wscott2• 8 years ago

One issue when working in open source is that contributors don't have control over the release cycle, and review requires smaller series than having the feature be delivered all at once.

Yes, my reply was unnecessarily disparaging. Overall this looks like a cool feature. Perhaps a stopgap solution if for those commands to just delete your cache. But then your repository will get mysteriously slower. I just need to think of this as a technology demonstration.

tomfloyer• 8 years ago

I wish i had those amazing algorithm skills.

xvilka• 8 years ago

Would be nice to have support of it in tig too.

yebyen6• 8 years ago

One of the best things I ever did for my git usage was installing this ridiculous thing in my .gitconfig aliases:

    [alias]
        l = log --date-order --date=iso --graph --full-history --all --pretty=format:'%x08%x09%C(red)%h %C(cyan)%ad%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08 %C(bold blue)%aN%C(reset)%C(bold yellow)%d %C(reset)%s'

It lets me type 'git l' and get this kind of commit graph:

    *       8264cf2a 2018-06-28 yebyen (origin/dev, dev) ...
    |\
    | *     df6e4c49 blabla commit messages
    | *     195d8924 commit messages
    |/
    | *     96e74a5c (branch-label) commit message
    | |\
    | | *   b2786c07 blabla
    | | *   85721288 blabla

This is semi-related but not nearly as interesting from a technical perspective. But it's one of the missing features in Git, to visually help people understand why it's helpful to rebase occasionally, and keep your commit history clean.

I think I found it here[1]

[1] https://stackoverflow.com/a/16735971/661659

WorldMaker7• 8 years ago

I had a similar alias for a while at a past job, but realized I was just using `gitk` anyway, and `gitk` is installed by default everywhere and doesn't need an alias setup, so it's also especially easier to teach to junior developers.

Also, I don't encourage rebasing, especially to junior developers. I realize everyone has different preferences, but a messy graph is useful and there are other tools like `git log --first-parent` for getting "clean" baselines from the graph. The great thing about using a graph in the first place is there are a lot of traversal options and you don't have to micromanage where the trees are to still see the forest.

yebyen6• 8 years ago

Yes! I agree with you mostly, except that we try to keep from having junior developers for too long. Nobody is really a junior developer, and everyone should get a turn as release manager. The main benefit of this is that people can see directly how their work style impacts the release engineering process, if they do know exactly what that process involves and actually get a turn at it. (You have to stub your own toe in order to know how bad it hurts.)

Our team is actually really small, and we like to make sure everyone knows about the hassles involved in putting together a release and doing a complete code review when it's needed. The main reason I encourage rebasing is because it helps avoid a messy graph, and a messy graph makes it immediately much harder to do a rebase across any number of merges, or any non-trivial span of time.

So in other words, I like to selfishly expect the other developers on my team to do rebases at the appropriate times, in order to preserve my own capability to easily do rebases when needed. (We also learned last week that git rebase has a --preserve-merges option that I feel foolish to not have known about sooner. We've wiped out so many merge commits unnecessarily.)

We treat the master branch as carved in stone like anyone should, but other branches should clearly flow their merges only one way (features into releases, or features into environments), and if a branch hasn't actually been included in a release tag, it's considered fair game for rebasing. It helps us to prevent our three developers' sometimes too many concurrent trains of thought from resulting in equally too many confusing merges, or an intractable number of HEADs to manage and organize into releases.

One of the places we struggle is that we're not all-the-way onboard with CI/CD processes, but we do the best we can so that whenever support for that kind of thing materializes, we will mostly not need to change our processes at all and can obtain the benefits of that kind of tooling as quickly as possible.

linkmotif2• 8 years ago

Rebasing is fundamental to a proficient local workflow. Interactive rebasing with git rebase -p lets you craft and recraft your commits easily until they are readable and tell a story. The sooner a person can rebase, the sooner they can use Git. It’s not really possibly to proficiently or pleasantly use Git without rebasing.

claytonjy• 8 years ago

I wonder if any of these folks will have a hand in improving the Github commit graph, now that Microsoft owns it?

As great as Github is, I've never understood why they still have such a hard-to-read, horizontal commit graph while competitors like Stash/bitbucket/gitlab have all had beautiful vertical graphs (like shown in this article) for as long as I can remember. I think this is especially valuable for newbies who are less inclined to get similar viz at the command line, but still useful for vets when they (inevitably) end up in weird branch situations.

s2g• 8 years ago

This is very cool.

Makes me sad I'll never do anything like this.

Supercharging the Git Commit Graph

Related Stories