How to Write a New Git Protocol

https://rovaughn.github.io/2015-2-9.html

by alecrn • 11 years ago

162 13 11 years ago

rovaughn.github.io

fortytw2• 11 years ago

Nice job! I had no idea anything like this was even possible.

cornstalks• 11 years ago

This would have been very nice to have a few months ago! At Aspera we wrote a protocol handler for Git[1] that let us send data over fasp (a reliable UDP protocol for fast bulk-data transfers). I learned a ton by reading Git's source code to figure out how to do it, but this blog post is much more pain-free.

Very good write up!

[1] https://www.youtube.com/watch?v=ECq7dCwixaY

alecrn2• 11 years ago

That's very interesting! The alternatives to TCP that can be written in UDP which are optimized for a certain application are especially interesting to me. Working at the UDP level can present a lot of interesting performance options, kinda like working in C instead of Python.

Something I've been curious about is if there are proxies that can convert protocols. So for instance, on your local machine, you could have a proxy that turns TCP connections on port 8000 into FASP connections to another machine. This would let you use an ordinary web browser over FASP.

You could even pipe the proxies, e.g., TCP -> FASP -> MinimaLT [1]. That way any program could have really fast data transfer over an encrypted tunnel.

[1] http://cr.yp.to/tcpip/minimalt-20130522.pdf

michaelmior2• 11 years ago

Thanks for sharing this. It's a good walkthrough and potentially quite useful for anyone wanting to implement a new protocol. It seems like the wrong tool for the job to me however.

This seems like it would be more easily implemented with smudge/clean filters[0]. In fact, this is exactly what git-crypt[1] does. The idea is that you run a filter on every file as it's checked out to do the decryption and every time a file is staged to do encryption. This requires nothing extra on the remote repository and you can you git commands as normal.

[0] http://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes [1] https://github.com/AGWA/git-crypt

JoshTriplett2• 11 years ago

Depends on the behavior you want. With smudge/clean filters, the files in the repository history are encrypted. That means you don't get the benefit of delta encoding, and any tools you have that look directly at the repository (such as gitk or anything based on libgit2) cannot operate on the cleartext data.

On the other hand, git-remote-gcrypt (https://github.com/joeyh/git-remote-gcrypt/) encrypts when pushing and decrypts when pulling, which leaves the local repository unencrypted but keeps it encrypted on the untrusted remote server.

michaelmior2• 11 years ago

That's a fair point :) Although I do like the fact that the smudge/blur approach means that you can selectively encrypt files as well as still share the entire repository publicly via any protocol.

frutiger• 11 years ago

Smudge/clean filters will reveal the metadata of the repository (directory structure, file names, etc.)

fiatjaf• 11 years ago

For people wanting to store their passwords, personal data and large files in a git-like fashion, I recommend Git-Annex: http://git-annex.branchable.com/

callesgg• 11 years ago

git over ssh should be as safe as ssh, right? Does it somehow go outside the ssh tunnel?

beagle3• 11 years ago

He is worried about data-at-rest after the ssh.

Specifically, he mentions wanting copies of his private keys backed up, but NOT having copies of his private keys in a github repository -- which would be the case with a standard push through ssh.

packetized• 11 years ago

This seems interesting to me from the perspective of being able to securely & incrementally back up a media collection to a cloud storage provider (S3, etc) without fear of it being scanned for copyrighted content, a la Dropbox. There are probably solutions out there to do this already; I'm just not aware of them.

Very interesting article, and well-written walkthrough.

alecrn2• 11 years ago

That's the reason I wrote this for myself, and why I made https://filegrave.com, as I thought it could be useful to others as well.

Git is already a great tool with a workflow for handling incremental updates that I happen to be used to working with, so there was no need to reinvent the wheel.

How to Write a New Git Protocol

Related Stories