JS Git: working git clones for offline development in the browser

http://www.kickstarter.com/projects/creationix/js-git

by bpierre • 13 years ago

114 68 13 years ago

ori_b• 13 years ago

I'm confused about what problem this is trying to solve. Can someone enlighten me?

creationix15• 13 years ago

The main problem it solves is to allow offline development in sandboxed environments like ChromeBooks and Windows RT devices (or even iPads if you had a BT keyboard).

shawndumas• 13 years ago

My 11yo has a ChromeBook and uses Cloud9 IDE to do all his coding at home. When he's on the go with mom (home-schooling at the park) he can't code; sadness...

If he had a git clone in a sandbox he could keep coding; happiness...

backed!

MatthewPhillips6• 13 years ago

Web apps do not have read/write access to the file system. So let's say you wanted to write a text editor web app. You can't modify a user's local files so you are forced to store their data on your servers. You are forced to create some UI for integration with GitHub, Bitbucket, etc. You are forced to add a lot of stuff Sublime and TextMate don't have to worry about... you are forced to build an IDE.

This would improve a lot of that kind of thing.

ldng• 13 years ago

IMHO, that's definitely something Github should have been working on for its paying customers.

nwienert2• 13 years ago

I don't think this page explains well the use cases for this. It alludes to kids programming, but then switches over to explaining how things are sandboxed, why the developer is experienced, etc.

My question is what is the net benefit of having this? What does it enable us to do? Why should I back it?

wesbos• 13 years ago

I think he is alluding to programming on an iPad and other locked down devices. It needs to happen in the browser as lower level access isn't an option.

jackmaney3• 13 years ago

This may be a bit of a naive question, but why would anyone want to code on a tablet? Touch keyboards are horribly inefficient and are prettymuch impossible to use for touch typing.

creationix15• 13 years ago

The main thing it allows is developing offline en environments like chromebooks (like the new Pixel). I worked on Cloud9IDE for a year and it was a great experience as long as you were online with a fast connection. With this library, html5 apps will finally be able to do the full developer lifecycle. They can clone from github to the browser's local file storage when online, work offline using an editor like ace or codemirror, and then when they are online again, they can push their changes back to github. I'll implement branching, merging, diffing, and as many other awesome common tasks from git as possible.

I've wanted this for years and I'm super excited to finally be able to create it when this kickstarter finishes!

nwienert2• 13 years ago

Thank you. This sounds really useful. I'd suggest adding something along these lines to the project!

sirclueless• 13 years ago

This sounds like useful information to have on the KickStarter page. You sort of obliquely reference this stuff in the first few paragraphs, but it would be great if you fenced off a whole section referring to the kinds of tools that could use this (with links to Ace/Cloud9) and talk about the reasons why they can't do this right now and how cool it will be after you implement this and you can use them offline.

If you want to jump on a particular recent buzzword hype bandwagon, you can talk about "Always-online requirements" and how you can fix that.

bsimpson• 13 years ago

Any plans for Brackets integration?

city41• 13 years ago

When working in this way, how do you run the code? I imagine anything involving a backend would be tough, but would it be possible to run purely client side code when offline?

tjholowaychuk• 13 years ago

Good luck man! Sounds like a fun project

AaronO• 13 years ago

Nice project. I've been working on Dulwich (a pure python implementation of the Git protocol) as well as a higher level abstraction. I would definitely be interested in contributing to JS Git.

MatthewPhillips6• 13 years ago

Will Git source compile with clang? If so emscripten is a possibility.

TheMakeA• 13 years ago

In the past I investigated compiling libgit2 with emscripten. Didn't spend a lot of time on it but I wonder if that could work.

creationix15• 13 years ago

I will explore this as one of the possibilities. I'm not dead set of reimplementing everything myself. If using an existing JS library and finishing it or compiling an existing C library using emscripten is the best, then I'll do that..

The point is I'll spend the proper time to evaluate all the options and fill in any gaps needed to have a solid solution.

zodiac• 13 years ago

fwiw I managed to compile libgit2 with emscripten - https://github.com/zodiac/libgit2

espadrine• 13 years ago

How will it work on non-Filesystem API browsers (that is, all of them except Google Chrome and… BB10)?

Are there plans?

http://caniuse.com/#feat=filesystem

MatthewPhillips6• 13 years ago

IndexedDB can store blobs. There's even a FileSystem API polyfill that uses IndexedDB: http://ericbidelman.tumblr.com/post/21649963613/idb-filesyst...

creationix15• 13 years ago

Thanks Matthew, that's the plan. Also since the target audience is developers, I don't have to support ancient browsers. This isn't a news site for the general public, it's a developer library.

artagnon2• 13 years ago

A pure Javascript implementation of git?

0. Why is this a good idea? Have all the desktops and laptops died, and are people forced to use Mozilla OS for development? I'd vastly prefer native applications on iPad, Windows RT, Android and other such platforms (if I suddenly went crazy and decided to do development on them). And a PNaCl application on Chromebook, if that's what you're specifically targeting.

1. Why won't it die like git.js did? [1] Why aren't you working on extending git.js?

2. Where can I see a technical roadmap? You're planning to deflate, applying deltas, huge packfiles with 4G objects in Javascript? What performance guarantees can you make?

3. Git is not a dumb book-keeper where you just checkout, make changes, and checkin. How are you going to replicate what makes git so awesome: `rebase -i`, `filter-branch`, `rerere`, pickaxe, and so many other features?

[1] https://github.com/danlucraft/git.js

creationix15• 13 years ago

I want this, I know many people who want this. That's good enough in my book to create it. If enough people believe in the idea enough to fund me, then it's a good idea.

I know this is a hard project, that's why I haven't done it yet in my free time. I realize that this will require my full-time attention. But click through the links of my other projects in the kickstarter. I'm not new to hard problems and low-level coding in JavaScript.

I don't have a detailed roadmap and I can't make performance guarantees because I don't know yet. But I do know that I'll do whatever is possible within the constraints of a web browser. Again, I'm not new to these constraints. I believe the web is finally mature enough to pull something like this off.

rektide4• 13 years ago

0. is a solved problem: this Kickstarter is for a library, not an application. There are already native libraries for git, so go ahead and go crazy and get busy developing, via libgit2, egit, ngit, objective-git, or any of the others.

You come off as tyrannically dogmatic without having any real use case- you're poo poo'ing a web platform without any expressed reason for wanting a native stuff aside from it being native. -1. (And, humor, the native stuff exists.)

artagnon2• 13 years ago

Don't mistake me for some old timer hammering away on his BSD box. I love the web, but I don't go around farting rainbows and claiming that Javascript is the be all and end all of all programming.

There's a reason Git is written primarily in C; it does a lot of heavy-lifting, and we want something that performs.

jackmaney3• 13 years ago

Now that GitHub has the ability to move files within a repo (https://github.com/blog/1436-moving-and-renaming-files-on-gi...) what will JS Git be able to do that can't be done by creating, editing, and moving files around in a GitHub repo?

67726e• 13 years ago

It allows you to work offline.

Additionally, not everyone uses GitHub.

shurcooL• 13 years ago

While I'm not interested in this at the moment (I choose to work with native tools), I can see the value for many other ppl so I've backed your project on Kickstarter and you on Gittip.

creationix15• 13 years ago

Thanks!

rektide4• 13 years ago

File API + node.js filesystem shim + node-git. It's been a weekend project somewhere on my todo list for a year, and it'd work just fine as a impl, and requires naught but working some glue into the cracks between these already made pieces.

That said, support away, would love to see Tim get this initiative kickstarted & see what solid work he kicks out.

MatthewPhillips6• 13 years ago

node-git just calls the git binary, which you won't have in the browser. This project is definitely needed.

rektide4• 13 years ago

Hi MP-

https://github.com/christkv/node-git is the node-git I had in mind: it's a native (js native) impl modeled strongly after the Grit library used heavily by a little company called Github, and I believe it's in pretty OK state although my first hand experience hasn't made it past the touring stages.

Sorry for the confusion! I knew Tim had a node-git as well, but I didn't know it relied on the binary or other native type stuff (such as a a library binding), so I didn't think the naming collision I let happen would cause any confusion.

Is there anything else I'm missing in my equation?

vjeux• 13 years ago

"Today there are fewer and fewer kids getting into programming because consumer devices are locked down and don't come with a way to program them."

I honestly doubt this is the case, I would expect more and more kids are getting into programming because it has become so widespread and easy.

jackmaney3• 13 years ago

Agreed. In fact, I'd like to see anything resembling evidence that a) "fewer and fewer kids [are] getting into programming" and b) if so, that it's caused by some devices being locked down.

creationix15• 13 years ago

I can only judge on what I've seen personally and what I've heard from industry leaders and both line up.

Wherever I'm living I go around to schools and groups looking for kids to teach programming to. They all have computers of some sort (mostly iPhones and some tablets), but don't know how in the world they can develop on them. The only way to write native apps for iOS is to buy an apple laptop, get a developer license and learn an advanced programming language. That's beyond the reach of most kids.

Now suppose that new websites popped up using my library and all the kid needed was a device with a modern browser and optionally a bluetooth keyboard? They could learn JavaScript and write webapps. There is huge demand for new programmers in our economy and most kids only see electronics as consumption devices.

Speaking of, my Chromebook just arrived via UPS, now for some fun!

rimantas• 13 years ago

That's a very weird statement. As if in 2005 it was easier program for your so open phone, or your not existant tablet.

TazeTSchnitzel• 13 years ago

The development environments have certainly improved. Now we don't have to deal with Java's MIDP, and we can instead deal with Android Java, a much nicer flavour.

chjj• 13 years ago

I agree with that statement in a way.

Linus Torvalds wrote a kernel because he bought a PC: hardware which allowed him to do anything he wanted.

Today, smartphones and mobile devices are much more popular among the youth. Could you imagine if Torvalds had been born in 1993? By this point, I'm sure he would have made something much less spectacular than Linux. Maybe he would make some pretty cool Facebook or smartphone games, and maybe a few cool web apps, but the freedom to tinker with hardware directly had been slowly slipping away over the past ~10 years. Since that avenue was becoming less available, there's a chance it wouldn't have grabbed his attention in the first place.

We still have laptops/desktops which aren't locked down. Hopefully that's what the young/future version of Torvalds would have gotten interested in. But how long until you buy a laptop only to find out that it doesn't grant you root access?

mattstreet• 13 years ago

To be fair, it's easier these days to afford both a smartphone AND a piece of much more hackable hardware. At least as far as prices go.

One of the main real reasons for Linus's success (other than say his talent) is that there wasn't a good alternative to Linux so even version 0.2 which could barely do anything by today's standards got interest and community involvement.

andrewflnr• 13 years ago

I'm not a Linus worshipper, but I don't think you're giving him enough credit. Even given a locked down platform, I don't think Young Torvalds would have been satisfied with a "pretty cool Facebook app". Granted, there aren't a lot of people who can just go write a kernel under any circumstances.

FuzzyDunlop• 13 years ago

There are still things like the Raspberry Pi, and all the things you can wire into it, which obviate the need to spend a few hundred quid on a full on desktop PC.

goldfeld2• 13 years ago

On absolute terms I'd think so, considering how many more kids have access to technology today. But if you compare the percentage of kids who get into technology's guts, out of all kids with access to it, I think that number has been decreasing. In other words, our market may be expanding but our conversion rate is in free fall.

nicholassmith3• 13 years ago

I've just read that a few times and I'm still none the wiser.

You want a working git implementation in the browser, okay, cool. Why? What for? I don't want to just disparage, I'm genuinely in favour of anything that gets good tools to more people, but I'm completely at a loss at the why from reading the Kickstarter page.

creationix15• 13 years ago

So I can develop offline using nothing more than a browser. I clone a repo from github to my browser when I'm online. Then I'm offline in a bus or on a plane, but I can still work because browsers have file storage now. When I'm done working, I can go online and push my changes back to github.

And all this will work anywhere that has a browser, ChromeBook, Surface RT, iPad, etc..

Obviously, it's mostly of use to web developers since JS is the only runtime, but I also have other later projects attacking that problem. But for now, JS only is a pretty big market.

nicholassmith3• 13 years ago

Hey, thanks for the reply!

Couple of things, repos can get pretty big, how are you planning on storing things? Using the HTML5 storage? Kicking it down the drive? If it's with HTML5 storage after that guy found a work around allowing sites to completely burn through a drive browser devs might start putting a cap onto the limit of storage that you can use.

I don't know, I'm a big fan of getting tools to everyone, but I'm still wondering if this sounds like a great idea, without actually being a great idea.

EDIT: that last line might be a little overly harsh looking back on it, it's not that I don't think it's a good idea (definitely is!) but I'm wondering if it's a great idea that's worthy of a Kickstarter.

drrotmos• 13 years ago

I'm a bit intrigued. As the main developer of a Git library[1] myself (in pure PHP, not depending on the git binary or esoteric extensions), I have no doubt that the actual parsing of the git objects and packfiles will be very much possible to achieve in a browser environment.

[1] https://github.com/e-butik/Gittern/

However, there are technical considerations. For example, pack files can get pretty large (indeed version 2 packfiles supports files > 4 GB), and I doubt browsers are optimized for random access to large files (so you wouldn't have to load the entire file in memory). Another potential problem would be pushing and pulling. The dumb HTTP transport is obviously possible to use (but rarely supported anymore), and the smart HTTP transport should be able to get working, but for implementing the git protocol, or git+ssh you'd need regular sockets, which AFAIK browsers don't provide.

I do however wish the author the best of luck with this project, and since I see that he's here, I will leave with links to a few resources I found useful:

* http://git-scm.com/book/en/Git-Internals

* http://opensource.apple.com/source/Git/Git-26/src/git-htmldo...

* http://www.opensource.apple.com/source/Git/Git-17/src/git-ht...

* http://git.rsbx.net/Documents/Git_Data_Formats.txt

creationix15• 13 years ago

Thanks for the links. You are correct that I probably won't be able to support all git repos because of memory constraints and unless I get special APIs from the browser I won't be able to do git protocol or git over ssh. I do feel pretty confident that I can implement dumb and smart HTTP though.

Even if this library can't be used for everything, it will allow a whole lot more use cases that are currently possible.

maksimlin• 13 years ago

I'd say smart http is more than enough, especially since Github for example is pushing it as the preferred protocol for most users: https://github.com/blog/1104-credential-caching-for-wrist-fr...

The main issue is that Github do not have CORS support for http access to repos (though they do support CORS for access to repos via their proprietary API) and I don't think the other big public hosters like Bitbucket or Google Code support CORS either.

On the subject of big pack files, sure you'd want to stream them to storage instead of trying to store in memory (for instance JGit has this as an option) but I agree Tim that to start with, even supporting small-medium size repos would be a huge win.

tmzt2• 13 years ago

Any idea what is going on with the progressive Git protocol, I know there was some discussion on the git devel lists but I'm having trouble finding it.

(Essentially it's a protocol based on fast-import to allow access a remote repository without cloning or receiving a pack.)

Also, there are proxies for converting TCP into websocket that should work with the Git protocol, and at least Chrome has some kind of native TCP socket from extensions.

goldfeld2• 13 years ago

I decided I like this guy a lot by the projects he has under his belt. Always nice to see a prolific open source contributor getting financial support out in the open like this. And this project seems like something I could use in the future.

zellyn• 13 years ago

Agreed. I'm in for $25. :-)

meryn• 13 years ago

Tim, do you know about https://github.com/danlucraft/git.js ?

tlrobinson• 13 years ago

I wonder how hard it would be to just compile the real git implementation with Emscripten? Isn't it mostly C?

JS Git: working git clones for offline development in the browser

Related Stories