A Git catastrophe cleaned up

http://blog.plover.com/prog/git-tastrophe.html

by asymmetric • 10 years ago

274 260 10 years ago

blog.plover.com

Why not just revert the offending commit? It would be a valid blip in history as mistake made and corrected.

OJFord2• 10 years ago

It occurred to me while I was writing this that it would probably have worked to make one commit on master to remove the back-end files again, and then rebase the entire topic branch onto that commit. But I didn't think of it at the time. And it's not as good as what I did do, which left the history as clean as was possible at that point.

barrkel• 10 years ago

He could have made the reversion commit on a local branch (branched from master), then rebased interactively to reorder / squash commits.

It presupposes that the commits are clean and well-separated though.

soft_dev_person2• 10 years ago

I read that, but it doesn't read like he is aware of the option to revert a commit directly. More that he would remove them manually and commit that.

Little difference, but if "as clean as possible" was the goal, I think that would be the cleanest.

[edit] added the little words that my keyboard seems to filter out.

rwmj• 10 years ago

Another thing is not to fear the non-fast-forward commit. Sure you'll need to tell the other developers (apparently "a hundred" in this case, but more often just a handful of people in the same room) that they have to reset back a few commits before pulling, but that's not the end of the world.

bluehex• 10 years ago

Not only that but the failure case when a developer pulls before receiving the memo isn't that bad. Git tells you that the branch was force updated and working out what to do from there is usually pretty easy.

latortuga• 10 years ago

Admittedly I did not read the whole post, but I agree that this sounds like the right answer. He made a mistake on master by merging a partial topic branch. The answer is to revert the mistake (git revert abc123) and then either rebase+merge or just merge the whole topic branch. I also don't agree with his conclusion that his way left the history in a better state - reverting the commit actually represents the mistake that was made.

OskarS• 10 years ago

Yeah, that was my instinct as well. The problem was that the first user did that kooky way of bringing the 18 files over with `git checkout`. That's a terrible idea, both because it lead to the catastrophe here, but also because you lose all history of those changes, they appear brand new in that commit (which makes `git bisect` and `git blame` much less useful).

No, just `git revert` that terrible commit, then do a regular sensible merge/rebase. Solves all problems in, like, two commands.

guard-of-terra2• 10 years ago

Two fun things about git: It is deterministic, and it doesn't delete anything (readily).

This means you can't really have a catastrophe.

Just git reflog your way out.

tomp• 10 years ago

That's a bit of an overstatement. You can't have a catastrophe with stuff already committed (but it's very easy to delete in-progress work).

guard-of-terra2• 10 years ago

Only if you never commit and then do checkout -f / reset --hard. Why would you be doing that?

Otherwise it's still in reflog. Anything you ever commited is just there.

duiker101• 10 years ago

If you are committing so rarely that you can have a CATASTROPHE with your in-progress work, maybe you should try breaking the work in smaller parts and committing more often

gpderetta• 10 years ago

well unless you haven't pushed in a while and you rm -rf your home directory. Of course that has never happened to me.

On a completely unrelated topic, extundelete is a great piece of software.

qwertyuiop924• 10 years ago

the reflog is GCed, IIRC. So if you wait long enough, and commit long enough, that history will be gone.

lomnakkus2• 10 years ago

The reflog GC is time-based -- it defaults to retaining entries 90 days.

rplst84• 10 years ago

This is the right answer. I think for whatever reason, software programmers are afraid to learn how git works. They'll spend countless hours learning how their other tools work - but not the one that keeps and protects changes to their work. It's maddening. The worst part is - it's really not that difficult to go into the .git directory and look at how the files are structured and learn it. Once you do, a lot of using git becomes second nature.

kwe• 10 years ago

This is the video that I keep recommending to new people who want to know how it works on a technical level. I've always had amazing/euphoric feedback to that recommendation.

https://www.youtube.com/watch?v=ZDR433b0HJY

CJefferson4• 10 years ago

I've had occasional data-loss from git.

The problem is when you realise someone badly rebased, or forced a push, a while ago and now you have to try to find where the stuff was lost.

mcherm• 10 years ago

The author concludes by saying:

I think I've written before that this profusion of solutions is the sign of a well-designed system. The tools and concepts are powerful, and can be combined in many ways to solve many problems that the designers didn't foresee.

I disagree. I consider this to be a failure of Git. The set of different options (normal merge, rebase, filter-branch, etc) is complex and not cleanly orthogonal which makes for a very messy "mental model". Even experienced experts would have difficulty finding the clear, simple way to solve this problem and those less experienced would have little chance of proceeding cleanly.

I really wish some tool other than Git had "won" the version-control race; I honestly believe Git to be the worst of the contenders in the most recent generation of version control systems (albeit better than the previous generation in important ways).

stonecodegump• 10 years ago

I agree. I know there is a lot to Git and you can pretty much do everything. But whenever such discussions come up about how many options there are and how amazing it is, I just glaze over and completely fail to concentrate. I don't think I have the brain power to actually master Git. Luckily I work in a 2 man team, so rarely do I need to do anything particularly complex. I have my workflow and I do what I can not to stray from it.

ipioxu15• 10 years ago

Git is certainly the best of the version control systems to date, and I'm happy that it won. Experts have no problems with issues like this.

The described failure mode also can occur in any existing version control system --- the user is essentially did

cp -f ../mybranch/somefile-new.c somefile.c; hg commit somefile.c

The history in mybranch touching somefile.c is now partly useless, since all the patches are already applied. Of course, the branch can still be merged, if the conflicts are resolved --- the conflict resolution is then pretty much doing the above command again.

mtrn• 10 years ago

Git is certainly the best of the version control systems to date, and I'm happy that it won.

Me too.

What I found was that people do not recognize what tool git really is. People do not read the manuals and count on their world knowledge acquired from years of svn or other centralized systems. It's not the best sign, that you have to really dig git to be able to use it properly. But once you understood the core ideas, you can basically derive a lot of the workflows yourself, in a very principled and clear manner. Also: once you get a bit more familiar with the data model, one almost cannot not appreciate its beauty, I believe.

warbiscuit6• 10 years ago

Honestly, I'm in the exact opposite camp - I'm glad mercurial is still alive and kicking, since I've found my experience with it (and tortoisehg) to be superior in a number of ways. Fewer cross platform issues, a much cleaner commandline interface, can hop over to a much more powerful gui if needed -- I generally use it even when I'm interacting with a git repo.

That said, I'm far from an expert in git. What do you see as it's advantages? (outside of the larger mindshare, obviously :)

deanCommie• 10 years ago

Allow me to disagree with your disagreement.

Git fails often for the "basic" use cases. I won't lie. The number of options is intimidating for a beginner and one can easily get themselves into trouble. This is why there are Git guides and tutorials and UIs and they are all ultimately unsatisfying when you're a beginner.

However the use case discussed in this article is NOT a beginner use case. It is an advanced case and the fact that Git is an advanced tool that supports it (both natively with git-filter-branch, as the author found out, and ad-hoc-ly as was their original plan is a testament to the power of the system.

Tell me how you would better achieve the requirements that were the premise for this article in a different VCS?

jasoncchild• 10 years ago

I agree with your disagreement. Furthermore, I'm curious how this edge case could have been mitigated using an alternative tool? most of my VCS experience is with git, and a very little SVN.

ksk2• 10 years ago

What is advanced about a partial checkin?

ap222133• 10 years ago

DVCS is fundamentally a simple concept. And, every use case should be possible with a series of elemental operations. Any 'advanced' use cases can and should just be 'macros'.

pawadu2• 10 years ago

I consider this to be a failure of Git. The set of different options (normal merge, rebase, filter-branch, etc) is complex and not cleanly orthogonal which makes for a very messy "mental model".

Allow me to disagree... Does the CVS or SVN way of doing rebase and filter-branch work with your "mental model"? Well, there is your answer.

I really wish some tool other than Git had "won" the version-control race; I honestly believe Git to be the worst of the contenders in the most recent generation of version control systems

Not to sound harsh, but why would I want to use an inferior tool just because you don't get git?

irishsultan• 10 years ago

I disagree with the person you're replying too, but CVS and SVN are not the current generation of version control systems (and CVS is perhaps not even the previous generation), so it's unclear why you bring them up.

If you want to compare, compare to darcs, mercurial, bazaar, monotone, ... plenty of choice without going back to svn (let alone cvs).

nolemurs3• 10 years ago

I see what you mean about the messy mental model, and I do 100% agree that in a well designed system most problems should have one obvious best solution. That said, I'm not sure that's a fair criticism of git.

For instance, in this case, as mentioned below, the author of the article should have just reverted the commit where he checked out the files, then done a normal merge. If you're thinking of everything at the commit level, it's pretty much the obvious thing to do. The first obvious thought is to just backtrack (as mentioned in the article), then once you realize that's off the table (since the code is out in the wild) the very next obvious idea is to revert. Revert is what you do when you can't just undo. It's not even as if his commit involved any merges, so the revert is super easy.

I guess I just don't agree that experts would have a hard time finding the simple solution - it was the first thought that popped into my head, and I read the article mostly looking for the explanation for why he didn't just do it.

While git does provide near complete control, you almost never need to use more than the basics, and it doesn't have to be confusing.

EDIT: Just to make very clear why I think the solution is obvious, the first thought that I think will occur to almost anyone is, "if I can just bring the master branch back to its prior state, then I can merge like usual". The beauty of git is if you can form a coherent idea like "bring the master branch back to its prior state" then you can probably do it, and it probably isn't hard. The mistake the author made was to have that thought, and then instead go off down a rabbit hole, and throw out the original simple idea.

jdmichal2• 10 years ago

If you revert, how do you guarantee that the final result is identical to the previously deployed and tested version?

artpepper• 10 years ago

But how do you revert a commit if there were subsequent commits that changed the same files? (If I'm reading the original post correctly.)

stephenboyd• 10 years ago

I think there's an opportunity for a simpler-to-use alternative CLI Git client. Something more user-friendly than the /usr/bin/git program we're talking about and more developer-friendly than the existing Git GUI apps.

VLM3• 10 years ago

Another parallel idea to consider is procedure and policy.

The cleanup after the foot-shot was pretty gory, but I'm not sure with disciplined git-flow that its possible to replicate that foot-shot, and if you can't shoot yourself in the foot then the goriness of the cleanup is a non-issue.

A third level up is you've got two devs carefully not coordinating and working together on code that closely related to each other, aside from the git-splosion that is not generally a recipe for success. Like, refactor the obviously independent multiple concepts (obviously independent because non-cooperating devs are working separately in parallel on them) in those files into multiple files and then there aren't merges to fight.

Drdrdrq• 10 years ago

There is gitless [0][1], haven't used it though.

[0] http://gitless.com/

[1] https://news.ycombinator.com/item?id=12621837

justinlaster17• 10 years ago

The set of different options (normal merge, rebase, filter-branch, etc) is complex and not cleanly orthogonal

All of those commands you just listed are clearly defined and serve different purposes. In other cases, most commands or command flags simply help the user do something in an automated fashion that you could do by typing each specific command out (i.e., git pull --rebase). Git's core tooling is mainly centered around graph manipulation, everything else has been added due to convenience. And quite frankly I've never really understood the criticism people levy at git for being "complicated."

It's stupidly simple at the end of the day, especially compared to the hodgepodge of features that other VC's provide, all of which don't really work together and are designed to cover some gap in what should be handled in the core design of the VC (I'm looking at you, Mercurial branches and bookmarks). All tools require you to be familiar with them. There's no getting around that.

Even experienced experts would have difficulty finding the clear, simple way to solve this problem

I'm not sure how you can possibly say that, the answer is pretty obvious to a lot of us reading this article. This just feels like baseless whining.

There is a litany of literature out there that could have easily pointed the author to a well defined, accepted, and documented solution.

darkerside• 10 years ago

most commands or command flags simply help the user do something in an automated fashion that you could do by typing each specific command out (i.e., git pull --rebase)

A tangent perhaps, but would you mind expanding on this? I use `git pull --rebase` frequently, and I'm not sure how this fits into what you described. Is it the specific command, or the automated shortcut? And what's the counterpart?

Just trying to check my mental model.

blktiger• 10 years ago

Mercurial bookmarks are equivalent to git branches, but Mercurial branches have no equivalent in git. In Mercurial, branches give you a way to track the branches a commit is a part of which is good for long-running branches (I think of things like master, maybe a long running refactor or major feature branch) while bookmarks are great for the more normal workflow of small, focused, feature branching. Neither of those are designed to cover a gap in the core design, they are a _feature_ of the core design that git doesn't even have.

avmich• 10 years ago

It's stupidly simple at the end of the day

That reminds me a joke. On a particularly convoluted math lecture the professor writes on the whiteboard and explains at the same time: "from here we can state that, obviously..." He suddenly stops, slowly turns the head down, keeps mumbling "so we can obviously...". He slowly walks in the room, trying to catch the thought, then leaves the room... Some minutes pass. Then he comes back, with head up, widely open eyes and confidently continues - "Yes, this is completely obvious - so from here we can easily get the following conclusion."

Git has a lot of nice explanations. However it's either a particularly edgy case, or all those great explanations somehow fail to easily conduct those core "stupidly simple" ideas forward to actual Git beginners. At least to some of those beginners - not an insignificant number.

CJefferson4• 10 years ago

My problem with git is that it has a simple model, with a horrible GUI on top.

For example, what does the following command do?

    git checkout a/b

I've had confused users manage to produce 3 different outcomes from this command:

* Get a clean copy of the file a/b from the current branch

* Checkout the branch b from remote a

* Checkout the local branch 'a/b'

By confusing where you can use particular types of arguments, I've found students will often end up in these situations (in particular lots seem to end up with local branches called 'remote/branchname', or files called 'remote/branchname'), which tends to lead to massive confusion.

Life would be much better if git was a more direct mapping to it's underlying model, and didn't try to be so clever about guessing what users wanted.

VLM3• 10 years ago

Git is just getting in the way of the classic old philosophical discussion of there should only be one correct way to think vs a more multicultural the more ways there are to think and adapt and overcome, the better.

The technical discussion of git innards and workflow just get in the way of the core philosophical discussion, and the innards would fall into place if the philosophical question were answered (as if philosophical questions are ever truly answered, LOL)

A classic example of the philosophical argument is Perl vs Python. There's infinite ways to express yourself vs there is only one way that works.

Now its very politically incorrect to say you like Perl in 2016 and I suspect Git is going to be in big trouble soon enough for the same philosophical reason.

I would be interested in hearing what is the philosophical equivalent of Python for source code revision control. I mean... people can't be seriously suggesting going back to RCS/CVS/SVN/Hg, can they? Or I guess they can? Maybe some closed source silo I'm unfamiliar with? Maybe no revision control at all and go back to tgz files with dates embedded in the filename?

A classic problem with limiting opportunity, limiting the range of thoughts and ideas, is finding a way to do it without impacting productivity or quality. In that way "use something simpler and enormously less capable" would be considered cheating the philosophical problem.

bad_user• 10 years ago

You're trying to find too much reasoning.

Nowadays many people hate Python as well and I'm speaking as an ex-Python developer that now hates Python.

And do you know what's ironic about Python? Because the language excluded certain features, it led to multiple non-orthogonal features that solve very specific use-cases and ways of doing things in its ecosystem, with hacks upon hacks meant to implement ideas from other ecosystems, most of them slightly broken.

The problem with language simplicity is that it is misunderstood. Simplicity in the language implementation is often correlated with complexity for the developer, being what happens when you limit what can be expressed.

Amezarak2• 10 years ago

I mean... people can't be seriously suggesting going back to RCS/CVS/SVN/Hg, can they?

...What's wrong with Mercurial? Or is there another Hg source control?

Personally, I started using Mercurial when I can and Git when I have to; I can't imagine someone considering Mercurial as backward even if they prefer Git.

kasey_junk2• 10 years ago

A) I've used both hg & git extensively and muuuuch prefer hg.

B) I find it crazy that people are so reductionist when it comes to source control. There are tons of systems out there beyond the rcs/cvs/svn tree that have explored the space immensely. If all you are comparing to is those then clearly you'd think git was the only choice.

SixSigma• 10 years ago

I consider this to be a failure of Git.

Surely it's a failure of source code as streams of text.

If each function was versioned with dependencies then this problem would be eliminated.

Replaced by other problems.

theptip• 10 years ago

This is close to theological discussion territory, and I can see both sides of the argument, so I'm just going to mention that there is an interesting paper[1] from a couple years ago that uses a rigorous concept design process to attempt to simplify Git, and results in a VCS that does indeed require a simpler mental model (Gitless).

Personally I'm not going to abandon Git and all the tooling and community it comes with just for a simpler mental model, given that I have a moderately good handle on the (mainline) features of git -- and therein lies the problem. (Although note, Gitless does work on top of Git, so you should be able to use it with Github by the looks of things [2]).

I've linked Adrian Colyer's summary of the paper to make it easier to digest:

[1] https://blog.acolyer.org/2016/10/24/whats-wrong-with-git-a-c... [2]: http://gitless.com/#vs

ASalazarMX• 10 years ago

I'll just leave this here http://xkcd.com/1597/

AstralStorm3• 10 years ago

This could be written about any tool. Especially bad with Perforce and Subversion - which do not really support branching. They say they do, but then the merging is a huge pain.

Sir_Cmpwn7• 10 years ago

Remember that git was written by and written for kernel hackers. Most of the people I've found having trouble with git are as far from kernel hackers as you can get and most who don't are quite a bit closer.

As a kernel hacker myself, I think Git's interface is superb and intuitive.

emmelaich• 10 years ago

I don't think Junio Hamano (aka gitster) has done much kernel work at all.

But point taken.

wilde• 10 years ago

Luckily there are a bunch of tools that wrap the git primitives to present more friendly UIs. I spend most of my source control time in SourceTree at this point.

agentgt2• 10 years ago

I have to say I am fan of both Mercurial and Git particularly because I'm a crusty old developer who came from some god awful VCS/SCM like PVCS, CVS, RCS, Source safe. I remember first using Perforce for the first time and thinking dear god it is expensive but totally worth it (this was before git).

Really what I think needs improvement is better diff/merging interfaces. There really isn't platform independent one that is opensource and free.

Right now I use p4merge because it works fairly well on macOS but I still think it sucks (it doesn't have keybindings and yeah I have tried vim and emacs diff ... they are ok but not as easy to visualize diffing).

voidr• 10 years ago

I would be interested to know that in your view, which VCS would have solved this problem better assuming they would have done the same thing. It seems to me that they chose the worst possible option each time they hit a problem.

mjd• 10 years ago

The Reddit discussion of this, though brief, was interesting and to the point.

https://www.reddit.com/r/git/comments/5i3mpz/another_git_cat...

coolgeek• 10 years ago

This is the author of the blog post we're all discussing

scarface742• 10 years ago

I think everyone is overlooking the main point.

He said that one guy was only making changes to either the front end or back end code.

They should have been two separate repos. one for the front end code and one for the back end code.

icebraining• 10 years ago

That's a huge discussion in itself: https://news.ycombinator.com/item?id=10007654

scarface742• 10 years ago

That was a great discussion. As someone said below, I agree if it is independently deployable, it should be in its own repo.

If you need to share code between repos you need to create versioned packages - in the c# world - Nuget.

mnarayan013• 10 years ago

If you break up your repos whenever two random people work on disjoint file sets within a branch, you're going to end up with a lot of repos.

Raphmedia• 10 years ago

Where is the line? Would templates files and view-models be in the front-end repo?

Ajedi322• 10 years ago

I'd say the line is that, if you can independently deploy just a single part of a repository, that part should probably be in a separate repo.

In the article, the author stated that "X decided to merge and deploy just the back-end changes". To me, that suggests that perhaps the backend is an independent component entirely decoupled from the front end.

At the very least I think you should avoid making changes to multiple independently deployable parts in the same commit.

msvalkon2• 10 years ago

While an unorthodox merge strategy was used, this is what happens when you hole up in a topic branch for a long time. I bet this would've been easier had they merged smaller commits or PR's to master constantly. If one is afraid of deploying unfinished features, don't make them functional until they are ready. Tie them together once finished. Or did I miss something here?

sangnoir• 10 years ago

I'm also very surprised - I thought it was standard to merge master (target upstream branch) into the topic branch frequently (daily seems reasonable). I know some people do not appreciate "merged <branch> into <other branch>" commits in their history, but that is a small price to pay IMO.

kosinus• 10 years ago

Or rebase. Don't hide merge fixes in merge commits, keep your changes relevant when read in context of current master.

Unless you're working on a topic branch together with another dev, but I find that rarely happens in practice.

roblabla• 10 years ago

Well, I routinely `git rebase upstream/master` to keep my git history clean. It has the same effect as a merge from master, but keeps everything tidy.

draw_down4• 10 years ago

It's a very small price to pay! I can't believe how many people hate this practice because of the merge commits. It's extremely useful and not doing it makes git usage that much worse.

swalsh• 10 years ago

If master is currently in production, a good way to do frequent deploys for code that's "not quite ready yet" is to use a feature toggle. This allows you to do partial feature deploys, but block any code paths from using it. Plus when it's time to start directing traffic towards that code path you can turn it off, if you find a bug you missed during testing.

MatekCopatek• 10 years ago

This only works for isolated features. As soon as it's some kind of refactoring or a feature that affects a lot of existing code, it's not viable anymore.

Taniwha• 10 years ago

while I don't use git a lot (I use p4 for work) I think that this is often a lot of pain - the trick is not to get to far from main, continually touching up your working branch, from main as you work so that when the time comes the final merge is essentially already done (again I'm not a heavy duty git user, maybe this is hard)

I'm also not a big fan of the everyone has their own branch world view, I worked on a big project (a chip design project) where we essentially lost track of what we were building because of this. I'm much more keen on requiring people to stay close to main and eating their own dogfood

hellofunk2• 10 years ago

The more I read stuff like this, the more I wonder how many problems would just go away for so many people if they used Mercurial instead.

qznc• 10 years ago

How would this problem go away with Mercurial?

hellofunk2• 10 years ago

It's much harder to shoot yourself in the foot with Mercurial.

Which is why so many developers and companies prefer it:

https://news.ycombinator.com/item?id=10228399

https://news.ycombinator.com/item?id=9465084

hackerboos• 10 years ago

Why is Mercurial better than git? It's my understanding that you can still rewrite history with Mercurial.

pjmlp• 10 years ago

Most of the Mercurial use cases fit better with the metal model that people coming from SVN and similar, are used to.

It is only when one wants to do advance tricks that he/she needs to enter "Git-like" land.

krupan3• 10 years ago

Mercurial keeps track of whether you have pushed a commit or not, and if you have pushed it, it won't let you rewrite it without doing a manual override. This feature is called, phases.

zedpm• 10 years ago

You get new problems with Mercurial, like almost every tool out there barfing when it encounters multiple heads in a repo. Don't have multiple heads in a repo, you say? Not a viable option if you want to use bookmarks as lightweight branches.

luos• 10 years ago

So if I get this then he just added the files to master, then tried to work on the topic branch?

That really seems weird. I don't think it's git's fault. Also he could have done git merge master --accept-theirs if he really wanted some kind of history but I guess it would be worthless.

forrestthewoods2• 10 years ago

I like Perforce. It may not be perfect. But it's idiot proof.

"Days since gitastrophe" is a common phrase. There is no Perforce equivalent. You can't blow your leg off. There aren't thousands of "Perforce made easy" blog posts because it's actually easy. There are no "fixing my p4 repo" tales because it never breaks.

Thanks Perforce.

taesis• 10 years ago

In my personal experience, life (or, er, people) tend(s) to find a way to mess up any sufficiently complex, shared resource. In other words, I'm happy that it never breaks for you :3

Re. Perforce made easy, I'd suggest hitting up stack overflow or the p4 forums -- the community is much smaller than git's (i.e. practically the entire open source community vs. game developers), so there are fewer people available to write such blog posts (but enough to complain, evidently). I have a number of company internal-blog posts that explain Perforce to fellow newbs, but they can't be made public due to contracts and etc.

reitanqild4• 10 years ago

Sorry for ruining the party (seriously).

I like Perforce. It may not be perfect. But it's idiot proof.

That's because it is almost impossible to do anything, good or bad with it. ;-)

"Days since gitastrophe" is a common phrase.

Never heard it before.

There is no Perforce equivalent. You can't blow your leg off.

The equivalent of a hand saw: you will have a harder time cutting off your leg with a hand saw than with a chain saw.

There aren't thousands of "Perforce made easy" blog posts because it's actually easy. There are no "fixing my p4 repo" tales because it never breaks.

Or because nobody uses it and those who do keeps the knowledge for themselves;-)

Seriously: I used perforce for 18 months and I tried to understand why people loved it but no luck. Git isn't exactly perfect but compared to "checking out" a file before even editing it or not being able to commit without access to the server (IIRC) git is perfect. Oh, and not being able to check out a file that someone else has checked out, so if a dev goes on holiday without "checking in" the files first then you have to hack a bit to get ready for work again.

The best argument however for git is that even the perforce guys have integrated with it.

Thanks Perforce.

forrestthewoods2• 10 years ago

Game development is probably >95% Perforce. Has been for over a decade. Will be for the next decade.

AstralStorm3• 10 years ago

Bonus points when you get conflicts or try to use their terrible Swarm review system which relies on shelves.

Also branching which hides previous history before branch out for no reason. No real branch merge functionality. Automatic merge being inferior to what Git has.

And more...

Diederich• 10 years ago

I worked at a company that used Perforce for a few years. We never used the p4 command-line tool because over the years people had written wrappers around all of the sub-commands to make things easier.

The wrapper command-line tool was about 10,000 lines of Perl, and called 'p5'. Good times.

fpig• 10 years ago

I don't understand the problem here, why didn't he just do a merge and resolve the 18 conflicts by using the version of the file from master?

And the problem wasn't in checkout-add-commit, that is a trivial issue, the WTF here is producing 406 new commits in a branch without ever thinking of merging master back into it or rebasing on master, to avoid having a giant merge later.

lmm3• 10 years ago

The next day he wanted to go ahead and merge the front-end changes, but he found himself in “a bit of a pickle”. The merge didn't go forward cleanly, perhaps because of other changes that had been made to master in the meantime. And trying to rebase the branch onto the new master was a complete failure. Many of those 406 commits included various edits to the 18 back-end files that no longer made sense now that the finished versions of those files were in the master branch he was trying to rebase onto.

Can one not instead merge master into the feature branch?

ep103• 10 years ago

ding ding ding. The original dev should have made a new feature branch off master. Then they should have dropped the 18 files in, or if the 18 files had a commit history, cherry picked those commits into the new feature branch.

The next day, when he was ready for the rest of the changes, merge those into the feature branch.

These devs were trying to be clever, and got bit by it :)

KISS

antocv3• 10 years ago

I also dont see why he didnt just git revert to the parent commit, the commit before the other guy changed 18 files seemingly randomly.

Then rebased featurebranch on that, and thats it. So sure the history would contain the mistake, so what.

cyberpunk9• 10 years ago

I'm with ya; I don't see the evil in just force pushing head back to where it was before though -- do you really need 800 useless commits in the history when you're going to just be adding the code back in sanely later anyway?

Reverting them seems a bit more risky to be than just rebuilding it properly -- you'll need to write some scripts to make sure you got all the commits and then the whole dev team is going to end up pushing commits which unrevert your reverts just so you can then rebase the feature branch..

Sometimes push -f is the best option ;)

aji• 10 years ago

Agreed. He has the following to say:

The easiest strategy in a case like this is usually to back in time: If the problem was caused by the unorthodox checkout-add-commit, then reset master to the point before that happened and try doing it a different way. That strategy wasn't available because X had already published the master with his back-end files, and a hundred other programmers had copies of them.

So resetting the commit isn't an option, but what about reverting it? It seems like the obvious alternative to me, I'm not sure why he didn't mention it.

peff• 10 years ago

The simplest solution is:

    # try the merge; you'll get conflicts on those files
    git merge topic

    # discard the versions from the topic branch;
    # you know you already merged those changes in
    # the funny "git checkout commit", so any differences
    # are due to changes on master.
    git checkout --ours new-file-{1,18}

    # now you are free to fix up any real conflicts
    # and resolve the merge
    git commit

This has the advantage of representing the true history. You had two lines of development (the original topic, and the "squashed" history created for deployment), and the merge shows them coming together and choosing the deployment-side content.

SonicSoul• 10 years ago

hmm this is certainly not simple to me! what's new-file-{1,18} for ?

antocv3• 10 years ago

Its the names of files, I believe.

so git checkout --ours {name_of_file1,name_of_file2,and_so_on}

The {} is standard bash expansion, you can try it like mkdir hello{one,two,three}, so parent poster went with articles use of new-file as the name of the files that were affected.

cyberpunk9• 10 years ago

Yeah; I know it sounds evil and this shouldn't be done but why not just make a new branch from the borked master, force push master back to sanity (pre-massive merge), have everyone else cherry pick and push their commits and then just rebase new branch and fix the conflicts in there as you should and merge normally?

btym• 10 years ago

Because it's selfish. Your time isn't more valuable than that of the developers already working against that branch (and as stated in the article, there were hundreds). It's your job to merge your changes cleanly.

draw_down4• 10 years ago

I hate this shit. Rebasing always causes conflicts and dealing with them is such a giant pain. I get that in this case the designer really brought the pain on themselves but I wish using git didn't require this sort of surgery periodically, which in my experience it does.

aninhumer3• 10 years ago

As opposed to what? Conflicts are an inevitable consequence of multiple people working on the same code at the same time, and git is as good or better than the alternatives for handling that.

draw_down4• 10 years ago

In my experience- as opposed to merging, which just works, but nerds get all persnickety about because they hate merge commits. I understand the concern I suppose, but working with `git rebase` just seems so unnecessarily painful. It's really awful.

kazinator3• 10 years ago

X decided to merge and deploy just the back-end changes, and then, once that was done and appeared successful, to merge the remaining front-end changes.
"What should X have done in the first place to avoid the pickle?"

0. (Of course, not develop a 406 patch changeset and then have to pick it apart. Make smaller pushes, frequently.)

1. Create a topic branch right there at the tip where the 406 changes are locally committed.

2. Then use git's interactive rebase to rewrite this branch such that just the back-end commits are picked first, followed by the front end.

3. Make a back-end topic branch from the last back-end commit and test that. If it's cool, master can be rebased to that and pushed to origin/master upstream.

4. Test remaining front-end changes, rebase master to them, push.

Also:

3. a) If back-end changes need fixing, fix them on the back-end-topic branch. Then rebase the original topic to the back end topic to pick up these changes "under" it. (I.e. replay the front end over the new back end, and install as new front end).

sirclueless• 10 years ago

What to do with commits that touch both back-end and front-end code in step 2.? This amounts to manual run of git-filter-branch, which is a feature the author was unaware of. Without that tool this task is daunting. If the author is willing to manually rewrite potentially hundreds of commits that touch back-end code, then there are a lot of straightforward solutions. We're looking for something more elegant.

kazinator3• 10 years ago

I have dealt with that sort of thing, and roughly as follows. Having somehow identified these commits, I would mark them "edit" in the interactive rebase workflow. Then when such a commit is applied and the interactive rebase stops on it, I would use "git reset --patch HEAD^" to undo some of its changes in the index (while keeping the working tree the same). Then do a "git commit --amend" to overwrite the commit with pruned one. At that point, all the rejected changes appear as local modifications. Another "git commit -a" creates a new commit out of them; so the original became two. If the commit needs to become three you can just "git commit --patch" twice. Or just reset the whole original commit to HEAD^ and do several "git commit --patch" ops. (Use -c <original sha> to re-use the commit message as a basis for the new commit messages).

Typically I will do a rebase pass through the commit stack to do nothing but these splits in that pass. Then when things are separated out, do another interactive rebase to do the re-ordering and squashing.

forgottenpass• 10 years ago

2. Then use git's interactive rebase to rewrite this branch such that just the back-end commits are picked first, followed by the front end.

Probably how I'd do it, but I know it's not a popular workflow. The only project that I've seen publicly talk about expecting a changeset to rewritten from "history of fiddling around" into "series of incremental improvements to a codebase" is the linux kernel.

It's more work for every developer, and digs into more advance git operations, but it really helps keep the tree clean. Most projects don't have a velocity that requires such well-groomed changes, but this is an example of how failing that can make the history ugly and changes more difficult to work with.

For those interested why/how you'd rewrite a patch series: https://www.kernel.org/doc/html/latest/development-process/5... The 406 changes probably compress to less than 20.

azernik• 10 years ago

My ex-employer (Meraki) was VERY insistent on people cleaning up their histories. It was very good not just for keeping a clean tree, but also for code quality - once you have code changes in neatly-separated commits with discrete chunks of functionality, any leftover test code or unrelated changes pop out immediately to both the original coder and the reviewer.

By the way, this also helps with the issue this developer faced because part of the problem seems to have been that the backend and frontend changes were mixed in the same commits (hence the ugly hack for committing). Doing this kind of history-cleaning as you go makes it much easier to manipulate the order of committing changes to master.

rurban• 10 years ago

400 commits not cleanly applying? Not a big deal. I routinely merge 1000-2000 commits and rebase 30 active branches to that also. The solution is git rerere. It stores all the resolved merge resolutions forever, and cp or rb apply cleanly then, without any trouble. Eg https://medium.com/@porteneuve/fix-conflicts-only-once-with-...

gragas• 10 years ago

and published the changes to `master`

-_-

cousin_it• 10 years ago

In a corporate environment, I think I prefer a simpler workflow with a plain old centralized VCS and without using any branches at all. As code gets written, each commit goes on the trunk behind a feature flag (which you need anyway). That way each commit can benefit from continuous builds and testing, and other people can notice problems early. Branches would only be used for releases.

I've worked like that for years on some pretty big projects, and it never caused complicated problems like in the OP. The only caveat is that you need a strong safety net against breaking the trunk (lots of tests, mandatory code review, etc.)

krupan3• 10 years ago

People in the discussion here saying, somewhat blindly it appears to me, that using mercurial would have avoided this mess. I'm a huge mercurial fan and have dealt with some tricky situations similar to this (but never this situation exactly) and I'm not so sure how mercurial would have handled it. The best I can say is that I've never known even the most adventurous mercurial user to use checkout (actually revert in mercurial) on individual files in that way. Is that something git people do more often?

Aside from that, it'd be fun to see how mecurial handles this, but I'm not sure from reading the original post if I could exactly reproduce it.

Mercurial would let you do the checkout (revert) trick that started it all. I can imagine it causing merge conflicts as described. Mercurial does let you specify how to resolve merge conflicts for the whole merge, or you can tell it not to resolve conflicts at all and then you can run hg resolve on a file-per-file (or glob of files) basis and tell it to pick default (equivalent of master) for the files you want. I didn't quite follow the git way of doing this he described with .gitattributes, but using hg resolve sounds easier (but neither are things a non-expert user of either tool would know).

In the end some other solutions were proposed. I would not recommend using checkout (revert in mercurial) either. I don't know of a filter-branch equivalent in mercurial, but that sounds like a cool way to deal with this. In mercurial I probably would have reached for graft (equivalent of git's cherry pick), which isn't very different from git.

marcinkuzminski2• 10 years ago

IMHO it's odd approach to the problem. I'd rather ask the author (who knows best in this case) to split this into two separate parts that can be nicely merged.

isaac_is_goat• 10 years ago

Why wouldn't they reintegrate the mainline development branch with the new branch if it was so long lived? And/Or have the new code behind a feature flag so you could potentially have it deployed but disabled? So many ways this could have been avoided with some basic forward thinking...

SadWebDeveloper• 10 years ago

The problem with Git is that everyone is trying to use Git as central repository rather than distributed as if it were SVN, personally i blame GitHub for promoting among the new developers the wrong tool for the job causing all this unnecessary drama. Git is the best version control system if and only if the project has a good leader checking everyone merges before commiting and letting everyone knows who is working on what and what parts will affect.

cyberpunk9• 10 years ago

But I couldn't think of anything, so I asked Rik Signes. Rik immediately said that X should have used git-filter-branch to separate the 406 commits into two branches, branch A with just the changes to the 18 back-end files and branch B with just the changes to the other files. (The two branches together would have had more than 406 commits, since a commit that changed both back-end and front-end files would be represented in both branches.) Then he would have had no trouble landing branch A on master and, after it was deployed, landing branch B.

Well. Okay. That's a technical solution and it'd work, it's probably no less time consuming than going and fixing the code in a new branch and merging cleanly (every time I end up needing to filter branch stuff I have to RTFM on it, and it takes ages) -- this problem is NOT a technical one though; it's a process one.

Why are you landing 400 commits in one go? Half of those were on files which then start causing merge conflicts for your team and wasted a huge amount of your time?

Use feature flags, fix your conflicts in branch, don't merge anything into master unless it's using the 'merge' button on github/gitlab/gogs/whatever. And really think/discuss/roundtable about how you're introducing features because it sounds like this is running away from you a bit here..

It doesn't need to be this complex, and these kinds of messes can't really be put on the tools -- although git certainly makes it easy to set a lot of things on fire..

edibleEnergy• 10 years ago

this problem is NOT a technical one though; it's a process one.

No, the cause of the problem was a process one, but now it's become a technical problem.

That isn't really helpful when someone has already created a git nightmare like what's described here.

cyberpunk9• 10 years ago

Hmm, I wasn't trying to come off that way, although I wasn't trying to be helpful with the specifics of their git nightmare since happily they sorted it out anyway: I was only trying to suggest that the only way you "clean up" from a nightmare such as that is to start looking at what seems to be an obviously fragile way of dealing with their code since that's the real problem..

Unfucking the repo, while mildly technically interesting should really be the smaller of the two lessons learned from such a mess... No?

bowmessage• 10 years ago

Worth noting that cherry-pick takes commit ID ranges in the form xxxxxx..yyyyyy, it may have simplified the driver.

Thanks for the tip re: --keep-redundant-commits!

jdonaldson• 10 years ago

Reading this was like watching a traffic accident in slow motion. I could hear myself yelling at the author as if he were a student driver:

"Use filter-branch!!! Use filter-branch!!! NOOOOOOOO NOT merge union with manual deletes!!!"

But... he went and did it anyways. Honestly, reading back through commit logs, you always find the part where the driver runs off the road, plows through a clearly marked gate, runs on a train track for a mile or two, then merges back onto the main street, carrying part of a mailbox and a deer carcass.

You can fault git if you want, but it seems like some of these cases just arrive naturally no matter what cvs is used. It would be great to have a "git education" repo that contains situations just like these to work through... sort of a "drivers ed" for managing a repo.

chetanahuja• 10 years ago

I humbly (re)submit this for your consideration https://git-man-page-generator.lokaltog.net/

guomanmin• 10 years ago

Participate in Atlassian Research

My name is Angela and I do research for Atlassian. I’m kicking off a round of discussions with people who use Git tools. Ideally, I’d like to talk to people that sit on a team of 3 or more. If this is you, I would love to talk to you about your experience with <using> Git tools, or just some of the pain points that are keeping you up at night when doing your jobs.

We’ll just need 30 mins of your time, and as a token of my thanks to those that participate, I’d like to offer a US$50 Amazon gift voucher.

If you’re interested, just shoot me an email with your availability over the next few weeks and we can set up a time to chat for 30 minutes. Please also include your timezone so we can schedule a suitable time (as I’m located in San Francisco). Hope to talk to you soon!

Cheers,   Angela Guo aguo@atlassian.com

A Git catastrophe cleaned up

Related Stories