Git Internals, Techniques, and Rewriting History

https://blog.isquaredsoftware.com/presentations/2019-03-git-internals-rewrite/#/

by mxschumacher • 7 years ago

162 58 7 years ago

blog.isquaredsoftware.com

acemarke12• 7 years ago

Oh hey, that's my post! Glad someone found it interesting enough to submit.

Don't expect this to hit critical mass by now, but happy to answer questions on the rewriting thing if anyone has any.

funkattack2• 7 years ago

Slide #39 seems to use a graph from https://marklodato.github.io/visual-git-guide/index-en.html#... , that is published under "Attribution-NonCommercial-ShareAlike 3.0 United States (CC BY-NC-SA 3.0 US)". This should be credited appropriately and your content should be shared alike.

acemarke12• 7 years ago

Yeah, I grabbed a bunch of images from various sources for the slides. Unfortunately, I was in a hurry and didn't record all the places I got them originally.

I'll try to find time to go through and attribute things. Thanks for the heads-up!

jeanmichelx2• 7 years ago

Cool post, i wish I could do something similar at work.

I'm wondering why you went through the pain of making an HTTP server to process JS files, instead of say putting the temporary file over a ramdisk and running the formatters in parallel over multiple files. Plus, even if it's slow, you only have to run it a few times right?

acemarke12• 7 years ago

A RAM disk never even crossed my mind :) I know they exist and have used one in the past, but it didn't occur to me as a possible tool. Also, I was doing this on Windows, and I don't even know if you _can_ set up a RAM disk on Windows. (edit and of course as I say that, someone literally just linked a Windows RAM disk FS driver upthread.)

And no, I was re-running this _dozens_ of times to get it right, especially when I was in the "run -> transform fails on broken JS code -> debug -> write regex -> rerun" portion.

dahart2• 7 years ago

Reading the approach you took for reformatting code, I was curious whether after you’d done it, it still seemed worth using the python+libgit2 approach? Do you know how long the shell approach would have taken without using libgit2? Did it save time overall to use the lower level approach, including the presumably extra coding time? The code formatter was probably the limiting run-time factor, no? When I’ve used filter-branch and BFG in the past, I had to run the whole batch multiple times before I got it correct, and before I was sure the result was safe... did you have to do that, run the filter multiple times? Would you take the same approach if you had to do it again from scratch? I’m curious if I should invest in this approach next time it comes up for me.

acemarke12• 7 years ago

Yes, it was definitely worth it, for multiple reasons:

- I'm a lot more comfortable with Python than I am with shell scripting

- The Python+libgit2 approach _absolutely_ saved time. Being able to iterate through commits in 4-ish hours vs 24 hours for a complete run was huge, and yes, I did a lot of partial runs to test how the process was going.

- Yes, the code rewriting portion ended up as the bulk of the time. I could do a no-op iteration of the commit history in about 30 minutes, and the final processing was about 4.5 hours.

If I ever needed to do this again, yes, I'd use basically the same approach, just with the advantage of applying the knowledge I learned right away.

dang• 7 years ago

We changed the URL from https://blog.isquaredsoftware.com/2019/10/presentation-hooks... to the slides which have the content.

acemarke12• 7 years ago

Note that the original post links to an earlier article I wrote that goes into a lot further detail on the actual rewrite process:

https://blog.isquaredsoftware.com/2018/11/git-js-history-rew...

samwestdev• 7 years ago

any video archive of the presentation?

acemarke12• 7 years ago

No, I only gave this as an internal brown bag talk at work.

ridiculous_fish2• 7 years ago

What are the best practices for extending git? Every example seems to be a shells script that calls out to git; is there a better approach?

saagarjha• 7 years ago

What kind of functionality are you trying to add? For something simply, shelling out to git is probably OK; if you're writing something more complicated many projects rely on libgit2. For really weird stuff you can probably just directly go and edit objects inside of .git, as long as you're careful.

ridiculous_fish2• 7 years ago

I would like to extend git such that you can check out a commit, amend it, rebase all descendants onto the amended commit, and update any branches accordingly. Effectively like `rebase -i` but not modal.

This requires keeping some state around. libgit2 is a nice pointer, I'll check that out.

(hg users will recognize this as "restacking.")

pvinis• 7 years ago

Extending how? Are you looking for automation or new commands or something else?

bloopernova• 7 years ago

You can add a section called alias to your ~/.gitconfig

    [alias]
        lol = !git --no-pager log --graph --decorate --abbrev-commit --all --date=local --pretty=format:\"%C(auto)%h%d %C(blue)%an %C(green)%cd %C(red)%GG %C(reset)%s\"

Then you can run "git lol" and get that cool output. That's one way to "extend" git.

hnrodey• 7 years ago

git aliases

custom shell scripts

use a Git library in your language of choice and write something

emmelaich• 7 years ago

You can just put any executable in your PATH with name of git-WHATEVER.

e.g.

    $ echo echo hello world > ~/bin/git-hello-world
    $ chmod +x !$
    $ git hello-world
    hello world

You'll even get typo recognition:

    $ git hello-worl
    git: 'hello-worl' is not a git command. See 'git --help'.

    The most similar command is
       hello-world

mederel• 7 years ago

The slides don't display well on mobile device

rudiv• 7 years ago

Seconded; the text does not wrap in portrait orientation on small breakpoints and trying to scroll trigger the progression of slides.

acemarke12• 7 years ago

Sorry, mobile formatting was not something I was concerned about when I was putting these together. I presented them from my laptop, so that was all that mattered at the time.

Phrenzy• 7 years ago

Do I sense an upcoming fork? ;)

pikzel• 7 years ago

Would have been nice if we could scroll through the pages and not having to use the arrow keys.

mettamage2• 7 years ago

I find the diagram on page 15 to get an overall feel/flow for basic git usage a very good diagram. If I'd still be teaching at a coding school I'd hand it out to my students.

rinchik2• 7 years ago

Isn't a bit off? It shows "git diff" as a diff between staging and workspace when it should be between workspace and local repo.

mettamage2• 7 years ago

Whoops, I think you're right.

I skimmed over git diff, as when I use git diff I constantly re-figure out for what it precisely applies again. I just care to remember "it diffs".

pjc503• 7 years ago

Not quite: if you stage something, it no longer appears in "git diff" and you have to use "git diff --cached" to show it. One of those things which I trip over occasionally.

"git diff --cached" = staging vs. HEAD

"git diff" = workspace vs. staging (but when nothing is staged, staging == HEAD)

(I think?)

ethomson• 7 years ago

No, this slide is correct. git diff (without any arguments) shows you what you haven't staged. In other words, the different between the working folder (which here is called "workspace") and the index.

darekkay• 7 years ago

I always found "rewriting history" misleading. Git is immutable by default, so you cannot "rewrite" the history. In fact, you are creating an "alternate history" (which the article mentions). While the difference appears subtle, it takes away the fear of users who never use any "altering" commands because they think they might lose their changes or mess up something.

chrismorgan• 7 years ago

The most common scenario is that the original history is immediately discarded, by virtue of no longer having any refs pointing to it—though it lingers in the reflog for a while before being garbage collected. In that situation, I think it’s quite reasonable to call it rewriting history, because the original history no longer exists; for the term “alternate history” to have much meaning, the original must still exist, to be compared to.

I’m looking at the long-term picture here, rather than how you interact with it when in the process of rewriting history.

Clarifying that you’re replacing history rather than modifying it is useful, but I certainly have no beef with the expression “rewriting history”.

dahart2• 7 years ago

You’re right in the case of rebase, and I like the point about making alternate histories, but I don’t mind the word ‘rewriting’ because the net effect is to modify the history that ‘master’ points to. I assume the downvotes are due to the article’s main point being about true history rewriting; using filter-branch and BFG isn’t creating an alternate, it’s truly re—writing.

Anyway, personally, I’d pick on the other word sometimes, and suggest we don’t call it rewriting ‘history’ when rebasing local changes before push. Maybe it’d be better to define “history” to mean things that are pushed and shared with others, and call commits that haven’t been shared something like “local changes”. Might be moot in this case since the article was mainly about rewriting the entire repo, so “history” is totally appropriate, but still perhaps nice to use terminology to distinguish between commits that have been shared from commits that haven’t, since that is the line across which one should be more careful with any kind of rewrite.

pjc503• 7 years ago

I love how the first real slide goes straight into the fake manpage generator: https://git-man-page-generator.lokaltog.net/

"App Repo Size Issues:" yes, this is the Achilles heel. By being a full distributed system where every client has to carry a full copy of the history of everything, some common practices become unsustainable. I've had two employers that checked build artefacts into SVN, for example: do that with git and the repo becomes unusably large very quickly. Vendoring dependencies, a useful practice if your project is slow-moving, will also bloat the repo. They should be using "shallow clone" for Jenkins, but even that can be surprisingly large.

I've also been through the "apply BFG to repo" phase (very time-consuming, and blocks all commits while you're doing it!)

New idea: run the formatter against every commit in the history, so that it looked as if the code was "always formatted the right way".

This is actually brilliant, for the reason they give - keeping "credit" assigned correctly to original commits.

Determined it was okay if older commits were potentially "broken", as long as the latest commit runs and has all of our changes as of late 2018

I'm less OK with this, as the chances of an automatically introduced horrible bug which you can't trace seem rather high and you've wrecked any chance of using git-bisect! But if the "tip" is the only supported released version, I suppose it's less critical.

It's also interesting that most of the speed issues are addressed by re-architecting to avoid syncing to disk. If there was an easy Windows RAMdisk this might have made almost as much of a difference.

acemarke12• 7 years ago

Agreed on all points.

We happen to be in a situation where we have very long development cycles, and also rarely need to go back and rebuild older versions. So, in our case we could get away with this.

And yes, the speed improvements came from a combination of iterating over Git commits in a single Python process vs running Git as an external command, and avoiding touching disk for everything except reading and writing blobs and commits.

jacobush• 7 years ago

But there is an easy RAMdisk for Windows! http://www.ltr-data.se/opencode.html/#ImDisk

aequitas3• 7 years ago

My collegae told me I'm the guy with his phone number in the git.txt file, don't know if that's a compliment?

But whenever I try to explain Git to someone I try to step away from a computer and just work out the problem on a whiteboard. Using post-it's for branches/HEAD/tags and marker to write down the commits and the commit-tree. Preferrably permanent marker to reflect the permanent nature of commits with regards to branches (eg: rebasing keeps the original commits around). Also I ditch files in favor of a picture of a cat there changes are attributes like added body parts or toys since most people like cats better than files.

Taking it away from a computer really helps to reduce the complexity involved and thinking about what manipulations you make to the Git 'database' visually really helps understanding the concepts imho. Just going through it step by step as if you where the Git binary making the changes, and after a while the Git commands turn from hard to remember trivia to tools in your toolbox.

pjc503• 7 years ago

Conversely, this is a massive indictment of the git user interface.

People used to talk about "CASE tools": Computer Aided Software Engineering, by analogy with the CAD tools that replaced physical drawing tables once the technology got good enough. I joke that certain tools are "computer impaired software engineering", and git can certainly belong in that category at times.

What we have built is a tool with a good internal model but a user interface that it's easier for you to not use a computer, to step away to the whiteboard, in order to work out what to do.

We don't tolerate this for, say, word processors; people don't go to a piece of paper to lay out their headings and only then work out the incantations necessary to achieve that result.

(Has anyone attempted to make a "transparent porcelain" GUI which represents the "intuitive" internals in an actually explanatory way? The success of which could be measured by the reduced number of mistakes and amount of apprehension experienced by non-expert users?)

aequitas3• 7 years ago

Very true. But the word processor example is a little scewed. It depends on the audience and the value they want to get from the tool, take Latex for example which is widely used writing scripts, being far away from WYSIWYG.

Have a look at Gitless[0]. It's a layer on top op Git which simplifies/rearranges some of the concepts and commands of Git while still manipulating the Git DB in the same way (you can switch back and forth between Git and Gitless seamlessly).

[0] https://gitless.com/

mschaef• 7 years ago

We don't tolerate this for, say, word processors; people don't go to a piece of paper to lay out their headings and only then work out the incantations necessary to achieve that result.

There's a reasonably significant set of people that do this sort of thing. The backlash against word processors in favor of simpler focused writing tools or even simpler word processors is what I'd offer as evidence. Scott Hanselmann's video series on "How to REALLY use Word." is another example. (Word has offered powerful stylesheet features since at least 1991 and very few people actually use them, even those that could benefit.)

intarga• 7 years ago

I don't think it's an indictment since it's a clear tradeoff. Yes git isn't intuitive or self explanatory for newcomers, but that's only because it's optimised to be maximally efficient for experienced users.

To me, this is a sensible decision. The target users of git will use it day in and day out for decades, in this case it makes sense to prioritise experienced usage. For onboarding new users there are plenty of good tutorials, and I think in the long run it makes sense to put the legwork in with those, rather than switch to a more intuitive but less efficient porcelain.

theamk2• 7 years ago

We certainly tolerate it for word processors, it is called LaTeX, and a majority of CS papers are written in it. It is pretty complex, and you sometimes need a help of outside expert, but I cannot imagine using Word instead for complex documents. One you get past the learning curve you can be much more productive.

Hm.. replace a few words and you will get a paragraph about git.

rsp1984• 7 years ago

Safari user here. For some reason the navigation arrows are missing and I can't get past the first page...

acemarke12• 7 years ago

Sorry! I built it using the Spectacle web slides toolkit, and have only looked at it in Chrome and Firefox because that's what I use myself.

dreamcompiler3• 7 years ago

Impossible to resize and read on my phone. It amazes me when a developer writes a presentation about a developer tool using a goopy JS-dependent infrastructure that apparently has never been tested on mobile.

acemarke12• 7 years ago

As I said down-thread, my only immediate concern when I put the presentation together was presenting it from my own laptop. I appreciate that folks are wanting to read it on mobile, but I've got a lot of stuff on my plate (primarily work around maintaining Redux), and modifying my slides to work better on mobile has not been a priority.

dreamcompiler3• 7 years ago

Fair enough. Sorry about not reading before commenting.

emilfihlman• 7 years ago

Note: the presentation has ui flaws and does _not_ fit on a 3:2 screen, so you'll have to zoom to about 80% for it to be visible. This is apparent on slide 8.

Git Internals, Techniques, and Rewriting History

Related Stories