Show HN: Fast `git status` (git core.fsmonitor in Rust)

https://github.com/jgavris/rs-git-fsmonitor

by jgavris • 8 years ago

52 18 8 years ago

github.com

bluejekyll2• 8 years ago

How does this differ from the standard fsmonitor?

This appears to mainly be a lightweight wrapper of watchman.

jgavris7• 8 years ago

In my benchmarks it's about twice as fast as the reference Perl script monitor. Also just a quick exploration for me into Rust.

https://github.com/git/git/blob/v2.16.0/templates/hooks--fsm...

bluejekyll2• 8 years ago

Thanks for the link. Yes, that makes sense. It’s also less code. :)

Edit: well, maybe just fewer comments... ha

glandium• 8 years ago

You should make it clearer that the benchmark is perl fsmonitor vs. rust fsmonitor. Without reading the comments here, I would have thought it was without fsmonitor vs. with.

sigi45• 8 years ago

At first i thought i don't understand what your little rust code is doing, than i got it.

Its a watchman wrapper written in rust.

But in that article you linked, there is mentioned that git ships with a sample watchman hook already?!

jgavris7• 8 years ago

It is not enabled by default yet, since it depends on an external tool to provide repo file update events. Git has one bundled that uses Facebook's watchman (with homebrew if you've installed it like so), but you have to configure each repo (or globally) to enable it. I was just curious how much faster a Rust implementation could be over a Perl script, and wanted to learn a bit of Rust.

Phrodo_00• 8 years ago

I wonder how much faster it would be to use the OS' inotify equivalent (if Linux, inotify itself) directly from rust rather than using an external tool.

pitaj• 8 years ago

I wonder what kind of gains could be seen by using a fully rust-native bundle with something like notify [1]

[1] https://crates.io/crates/notify

avar2• 8 years ago

This is not at all comparable to what the OP submitted and would be at least a many thousands of lines worth of effort.

What watchman is doing is doing inotify-as-a-service for you, in order for any program to use the Linux inotify API it needs to be running constantly so it can keep consuming events from the kernel, and in the case of the use case watchman fulfills, maintain an internal database of how the current state differs from various times in the past.

There's lots of very nasty edge cases you need to deal with. E.g. if you don't consume the kernel events fast enough (or the buffer is too small) new events will be dropped.

You'll get told about this by the kernel, but now the only way to re-build your in-memory representation to match reality is to recursively walk the FS with readdir(), stat() etc, but at the same time consume new inotify events and update your in-memory structures while doing both.

Needless to say this is a lot more complex than just using watchman off the shelf.

It's also pointless to optimize this area, watchman usually returns in 2-3 milliseconds even on huge trees, but because of bugs / misfeatures in the current git-side implementation it can take hundreds of milliseconds until we make sense of all of that. This is because git doesn't really "know" about watchman/inotify, it just uses it as a replacement for walking the FS, but on big repositories other stuff can still be expensive.

To OP: I highly encourage you to submit this to the Git mailing list for comment. I've been involved in this area (helped write the Perl hook) and there's people who'd be interested in having this be e.g. a part of git's contrib/ directory so it can be compiled with git.

jgavris7• 8 years ago

Thanks for the explanation, indeed watchman does all the heavy lifting! I will submit it to the mailing list, I'd love to see it in git core.

avar2• 8 years ago

If you're optimizing for speed use BSER instead of JSON to talk to watchman: https://facebook.github.io/watchman/docs/bser.html

jgavris7• 8 years ago

Fantastic suggestion! I will give this a go soon.

https://github.com/facebook/watchman/tree/master/rust/serde_...

benpeart• 8 years ago

Very cool! I wrote the original (with help) in Perl as it has the advantage of being installed with git but I love the speed advantage of your Rust implementation. Also helps shows the advantage of making core.fsmonitor a pluggable interface.

mwean• 8 years ago

What about using entr[1] instead? I've had great success using it for rebuilding ctags on file changes.

[1] https://github.com/clibs/entr

jgavris7• 8 years ago

I have a 'git aware' bash prompt, so my primary use case is making sure my prompt is ready with the git status really fast.

ufmace• 8 years ago

Cool idea, but I tried it on one of my team's bigger repos, and it's actually slower than stock.

Show HN: Fast `git status` (git core.fsmonitor in Rust)

Related Stories