SQLAlchemy Migrated from Mercurial to Git

http://www.sqlalchemy.org/blog/2013/05/25/sqlalchemy-migrated-to-git/

by dbader • 13 years ago

114 58 13 years ago

octo_t• 13 years ago

> Git manages the size of the repository more efficiently; while the Mercurial repository has been approaching 50M in size, the Git repository is only 17M.

17MB vs 50MB - almost a third of the size. That is definitely quite impressive.

micampe• 13 years ago

I’m not the biggest git fan and I still think it’s right choice, but how is this even a reason to switch? Unless you go in (older) Subversion ridiculousness how are 40MB of disk space even a concern? They even listed it first. At best it would be Oh, and we saved 40MB, the size of two raw pictures from a digital camera.

myth_drannon• 13 years ago

I would assume people cloning the repository on slower connections.

zzzeek3• 13 years ago

it's the time it takes to clone. Also with git I need to clone a lot less since I can create local feature branches that I can delete if they are abandoned.

edit: quick speed test, git clone from my server = 17.4 seconds, hg clone from the same server's hg repo = 25.4 seconds

bobz• 13 years ago

When you consider that distributed repositories are going to be cloned dozens, hundreds, or thousands of times, it starts to add up.

Even if this larger project could handle that bandwidth, it is a significant factor for smaller projects, or larger projects like github. Meaning more of a chance that git remains the dominant choice.

etanol3• 13 years ago

The size issues of the SQLAlchemy repository come from the way Mercurial handles copies and renames.

I prefer Mercurial because is much easier to use but this file rename issue always make me feel uncomfortable when reorganizing code.

These days I'm giving Fossil a try, which still is easier to use than Git and the repository size sits between Git and Mercurial.

tonfa• 13 years ago

If that was an actual issue (I doubt it is), they could have done a oneoff repack server side (reordering changesets to optimize compression).

Edit: I think the third point would have been enough, network effect for free software is important. Size (reorder your repo if it matters), branching (use bookmarks, not named branch unless you know what your are doing), and history rewriting (use evolve) are dubious points.

SEJeff• 13 years ago

Very smart move. Some time ago Riak did the same thing. Their rationale was more detailed, but came to the same conclusion: http://basho.com/a-few-more-details-on-why-we-switched-to-gi...

etanol3• 13 years ago

Too bad there are no comments on the blog:

    SQLAlchemy's issue repository will remain hosted on Trac;
    while a Git repository can be mirrored in any number of
    places, an issue repository cannot (for now! Can someone
    please create a distributed issue tracker? Should be
    pretty doable, though getting Github/Bitbucket to use it,
    not so much...), so SQLAlchemy's long history of issue
    discussion remains maintained directly by the project.

http://fossil-scm.org !!!

zzzeek3• 13 years ago

uh yikes! http://fossil-scm.org/index.html/dir?ci=601c15421a4a5ca5&...

although good point, I see they're doing distributed. will take a look at how they approach that.

etanol3• 13 years ago

I think you can customize most parts of the Web UI:

http://www.fossil-scm.org/fossil/wiki?name=Cookbook#css

qznc2• 13 years ago

There are tools like Bugs Everywhere [0], which mix well with distributed VCS. However, the question is if distributed bug tracking makes sense.

[0] http://bugseverywhere.org/

travisb• 13 years ago

There are many different projects which implement distributed bug tracking[0], some better than others. The question of how much distributed bug tracking makes sense depends strongly on the structure of the project. If the project is developer heavy (such as developers responding to user help requests) it can work well, but if the project has several strata of developers and support people and users then it might not make as much sense.

[0] http://travisbrown.ca/blog.html#TooMuchAboutDistributedBugTr...

voltagex_• 13 years ago

Trac is pretty bad. Can you remove yourself off a subscribed issue yet?

trosenbaum• 13 years ago

Would something like Kiln Harmony work for storing the main upstream repository?

https://secure.fogcreek.com/kiln/

It seems that if KH translates seamlessly between HG and Git that a project could accept pushes from both, right?

I assume since KH is not open source / free that this would not be an acceptable method for maintaining the main repo, or that the translation would introduce interesting wrinkles, but it does seem like one could potentially have his cake and eat it too.

qznc2• 13 years ago

Thanks to hg-git, Mercurial users should have no problem using the git repository.

gecko• 13 years ago

hg-git and Kiln Harmony serve different, though overlapping, purposes.

hg-git allows people who know both Git and Mercurial, but who prefer Mercurial, to work with Git repositories from Mercurial. No effort is made to hide the Git model, and no effort is made to ensure that the Mercurial repository generated from the Git one is idempotent (i.e., a given Git repo will always generate the same Mercurial repo) or can round-trip (i.e., I can trivially craft repos that hg-git can work with just fine, but where pushing to a bare Git repo will result in a different Git repository from the original). The benefit to this model is that the Git users don't need to do anything different, ever, and no one needs to know that you were using Mercurial, ever.

Kiln Harmony does something different: it's designed to let a team use whatever tools they want. This means that Mercurial users don't have to learn the Git model, Git users don't have to learn the Mercurial model, and generally, everything "just works" in that situation. Doing that requires a lot more processing power than hg-git requires, which is part of why we only offer it as a hosted solution for the moment. It also works best when your central repository is a Kiln Harmony repository.

I obviously think that's a fine trade-off, but if part of the main motivation of SQLAlchemy was GitHub, then Kiln Harmony probably isn't a good solution.

zalew2• 13 years ago

> Largely due to the popularity of Github, Git has achieved a much higher userbase, to the degree where we regularly have users requesting us to move to Git so they can provide pull requests.

I am too a Mercurial refugee and prefer Git, but how is that even a reason? People don't want to contribute if it's on a Mercurial repo?? Python seems to be doing fine on HG.

Meanwhile, thanks for your amazing work on SqlAlchemy(&co), zzzeek!

gamache2• 13 years ago

I think what those users mean is that they want the project on Github. There's no concept of "pull requests" within Git.

Anyway, most Git users aren't Mercurial refugees; I wager we're Subversion refugees and kids.

zalew2• 13 years ago

> There's no concept of "pull requests" within Git.

http://git-scm.com/book/ch5-2.html "run the git request-pull command and e-mail the output to the project maintainer manually."

http://www.wired.com/wiredenterprise/2012/05/torvalds_github... "Git comes with a nice pull-request generation module, but GitHub instead decided to replace it with their own totally inferior version."

> I think what those users mean is that they want the project on Github.

that would be even worse. luckily as zzzeek explained below, it wasn't the case.

k_bx2• 13 years ago

Could someone please explain me an argument about removing branch in git vs closing it in mercurial? As I understand, when you "remove" branch in git, just as when you close it in mercurial, it doesn't delete commits from tree either, since it can lead to disaster (if those commits were pushed already). So someone else could actually start new branch from commit which is in branch you just "deleted". Am I wrong here?

0x0• 13 years ago

If there are branches with commits that aren't referenced elsewhere, i believe deleting the branch will eventually cause the commits to be "garbage collected". So you can delete abandoned experiments and keep a clean house. On the other hand, commits that are referenced elsewhere (perhaps merged into a main branch) will stay. But you could still delete the branch it was merged from, and avoid having the list of branches grow huge.

jedbrown2• 13 years ago

You can delete commits from Hg using 'strip' (enabled via the 'mq' extension). But this is strict deletion (as opposed to marking it for gc after a grace period) and you can't push an "unreference this commit", so you need admin access on the remote to strip a commit there. If you push branches to bitbucket for review, the abandoned commits stick around as anonymous heads until you strip them via the web interface, but they still stick around for anyone that pulled from you. The Hg community is working on a system [1] for dealing with this, but it's still not ready for general use and I find it significantly more complicated than Git's model.

[1] http://mercurial.selenic.com/wiki/ChangesetEvolution

plorkyeran2• 13 years ago

> So someone else could actually start new branch from commit which is in branch you just "deleted"

Well yes, of course. They could also start a new branch from any other commit that they happen have in their local repository. Branches aren't really 'things' in git. They're just a pointer to an arbitrary commit that automatically gets updated when you make new commits. It's rarely a useful thing to do, but you don't even have to be on a branch to make commits.

csense2• 13 years ago

> Am I wrong here?

It sounds like you may be a little fuzzy on how Git works.

> when you "remove" branch in git...it doesn't delete commits from tree...since it can lead to disaster

When you delete a branch, only commits and filesystem states that are no longer accessible from any other branch, tag, index, or stash become eligible for garbage collection. So in normal usage [1] [2], you shouldn't ever be able to delete the contents of any commit that exists in the history of any named commit.

A commit that is newly eligible for garbage collection will not actually be garbage collected during a grace period, I believe one week. During this time the commit is still accessible by commit hash [3].

Branches are kind of like variables in a garbage-collected language, and commits are like objects in that language. An object is eligible for GC if and only if there are no variables that point to it. Making new variables that point to the same object is a really lightweight operation since most of the data's shared. Unlike many OO languages, objects in Git are immutable, which means git can safely collapse identical copies of the same object to a single instance (an optimization which git is quite aggressive about).

> it can lead to disaster (if those commits were pushed already)

If you want to cause a remote repository to delete a branch, look at the --mirror and --delete options for the push subcommand. AFAIK, pushing the deletion of a branch has the same effect as if you deleted the branch locally on the machine you're pushing the deletion to.

> someone else could actually start new branch from commit which is in branch you just "deleted"

If another developer has merely fetched the deleted branch, their remote tracking branch will be deleted when they fetch again. If your colleague has created an actual local branch with the changes you deleted (as opposed to a remote-tracking branch), when or whether your colleague deletes that branch is entirely under his/her control [4].

[1] Using low-level git subcommands to force git to delete objects which are still referenced by other objects is abnormal usage.

[2] Corrupting the object database in your .git directory through unclean shutdown, hard drive failure, or vindictive hex editing is also abnormal usage.

[3] If you neglected to write down the commit hash before you deleted the branch, you can likely find it in the reflog. The reflog is a history of the commit hash at the tip of each branch. Alternatively, you can use git subcommands to browse the object database and find orphaned commits.

[4] At this point, git has no technological way [5] for anyone but your colleague to delete the local branch from your colleague's machine. Socially, of course, you can use project management techniques to encourage your colleague to delete the branch (e.g., if the official maintainers announce that the deleted branch is considered obsolete and will not support it or accept changes based on the version of the project in the deleted branch, that can be a compelling reason for people to stop using the copies that are floating around. Or your colleague's boss can tell him to delete it or be fired.)

[5] If you have git push access to your colleague's repository, or you have shell access to your colleague's user account, you can delete their branches. Footnote [4] is referring to the usual case where your colleague's machine is a private box where you have no access.

k_bx2• 13 years ago

Thanks for your answer. I really was never aware of git's GC mechanisms before, that's why I was sure that commits still hold inside a tree. And you perfectly answered my question (well, at least parts that I understood all details about, I clearly see I'd need to read more some time later).

I still find mercurial's branching model "the right thing" in terms of tree showing development history inside a branch, or overall (multi-branch) history-review (where you clearly see which commit was made in which branch), but I now see that it's really nice from repository-cleanup perspective to have features git has, to deal with non-needed branches and commits (via different mechanisms).

qu4z-22• 13 years ago

A commit isn't really "in" a branch in git in the same way as it is in hg. In hg, each commit "belongs" to a branch (I'm remembering correctly here, right?). In git, a "branch" is basically a variable pointing at a specific commit. A commit can be part of one or many branches (or even none).

ezquerra2• 13 years ago

Every project has the right to chose the VCS that they feel is right for them. However I find the reasons they give for the switch are pretty weak except for the last one (github is more popular).

1. Repository size: Such size differences are uncommon, and mostly likely due to the fact that the git repo was a brand new repo created from scratch by converting the original hg repo. If they repacked the mercurial repo it would probably be greatly reduced in size. They could also do:

    hg convert --config "extensions.convert=" --branchsort sqlalchemy sqlalchemy-smaller

Which results in a reduction of the repository size from 45 MB in the original to 24 MB in the converted repo. This is not as good as git but quite close IMHO.

2. This is not correct: Mercurial bookmarks _used_ to be an extension (a long time ago!) but they have been built into mercurial for a long time now. They are basically equivalent to git branches. So nowadays this is just a process issue. If using named branches is a problem just tell your devs to not use them and use bookmarks instead!

3. Mercurial's rebase and mq extensions are part of core mercurial. They are packaged and distributed with mercurial, their test suites are part of the mercurial test suite and they are supported and developed by the core mercurial developers. The only difference between a core command and one of the official extensions is that these must be manually enabled to be user (by adding a line to the mercurial config file). This is a good thing! This makes it harder to mess up your repo if you do not know what you are doing. If you are not knowledgeable enough to find and edit the mercurial config file you should definitely not be using rebase or collapsing your changesets.

The last reason on the other hand is one that I can understand and which I think has merit. Is it worth the hassle of changing their DVCS system? Perhaps. However it bothers me when legitimate _social_ reasons to switch from mercurial to git are supported by not quite as clear cut technical reasons such as the ones put forward in this blog post.

zzzeek3• 13 years ago

> If using named branches is a problem just tell your devs to not use them and use bookmarks instead!

but they do, because all the devs know git and are only using mercurial because they want to send you a pull request, and now their feature branch is forever.

Mecurial is unforgiving of mistakes - once in the repo, they are forever.

As for MQ, I've really tried to understand it, but it's just profoundly awkward IMHO. Mercurial's main benefit is that it's much easier to use and understand than git, so if I'm going to need to fire up the extra brainpower, I might as well learn git where these advanced patterns are extremely commonplace, and are part of the core functionality, not as a series of optional extensions that have been bolted on over the years. I've almost never seen anyone using MQ or rebase with mercurial (or even bookmarks for that matter).

I've tried carefully in my post to express that my points are all based on an overlap of social with technical issues. I am aware that technically, hg has answers for all these things. But these answers don't pan out in the real social world, for better or for worse. If git didn't exist, I'm sure there would be a much larger knowledge share for mercurial, and I was even betting on that in the beginning (certainly the "easier" tool will be more popular?) but it hasn't turned out that way.

rbehrends4• 13 years ago

> As for MQ, I've really tried to understand it, but it's just profoundly awkward IMHO. Mercurial's main benefit is that it's much easier to use and understand than git, so if I'm going to need to fire up the extra brainpower, I might as well learn git where these advanced patterns are extremely commonplace. I've almost never seen anyone using MQ or rebase with mercurial.

I believe the GP's reference to mq is for the strip command (to delete unwanted revisions), which for some odd reason that I've never understood is part of the mq extension.

ezquerra2• 13 years ago

If I want to contribute to a project I don't expect them to accommodate my workflow. I expect to be asked to comply with their workflow.

Let's say that I were the maintainer of a project in which we did not want to use named branches. If I got a pull request from someone that had created a named branch I would just kindly ask the contributor to remake his pull request, using bookmarks rather than a named branch.

The fact that they made a mistake _on their repo_ does not mean that that same mistake must end up on _my_ repo.As I said this is just a social problem.

You say that mercurial is unforgiving of mistakes. I say that mercurial makes sure that your _shared_ history is not lost. You can edit unshared history as much as you want.

As for MQ... I would say that it is an acquired taste :-> If you don't like it there are some good alternatives, such as histedit (also a built-in extension) and the new evolve extension which is still a work in progress but which already lets you safely edit _shared_ history.

dlitz4• 13 years ago

> Mercurial's main benefit is that it's much easier to use and understand than git

That's really only true if you're a beginner and you weren't properly taught how git actually works. Fundamentally, git's data model is really simple and transparent. The commands are somewhat arbitrary, but no moreso than Vim's or Emacs's. As with Vim and Emacs, you don't need to know most of the commands anyway, except for convenience.

Ziomislaw• 13 years ago

TL;DR - we didn't read the HG manual, so we had to change as we did't know how to use its features.

qu4z-22• 13 years ago

Or because none of their contributors who only installed hg to submit a patch had read the entire hg manual.

develop74• 13 years ago

Of course, all they had to read and follow was "contributing to sqlalchemy" manual.

develop74• 13 years ago

1. What a huge saving — 0,002¢ per repo copy. Don't spend it all in one place.

2. Bookmarks is core feature since March 2011. This basically means those who pushed "move to Git" decision, haven't checked if anything was updated in Mercurial (and with bookmarks in particular) since then at least. This is pretty understandable — it is known that only software gets updated is one you're paying attention to, others don't.

3. As ezquerra mentioned, those extensions Mike is probably referring to, are shipped with Mercurial, and are enabled with only single line added to .hgrc. Not mentioning those extensions do not _emulate_ Git features, they _reproduce_ it.

4. Only reason makes sense.

SQLAlchemy Migrated from Mercurial to Git

Related Stories