Hashpipe – Pipe iff the hash matches

http://jbenet.github.io/hashpipe/

by _prometheus • 11 years ago

144 83 11 years ago

jbenet.github.io

jrlocke• 11 years ago

A polarizing name, if perhaps less so amongst programmers

_prometheus18• 11 years ago

yeah, i was going to call it `hashcat` -- maybe i should do that. or `pipehash`.

nibbler• 11 years ago

already have a hashcat purring on my computer...

https://hashcat.net/oclhashcat/

Crito• 11 years ago

I wager more programmers would object to hashcat than hashpipe; cat is for concatenating! http://en.wikipedia.org/wiki/Cat_%28Unix%29#Useless_use_of_c... ;)

cachvico• 11 years ago

Don't change it, the name is brilliant :-D

chralieboy• 11 years ago

This is a fantastic name. It says exactly what it does and has an excellent Weezer song to back it up

_prometheus18• 11 years ago

https://www.youtube.com/watch?v=qguED5Aouv4

UPDATED: whoops wrong link. I meant: https://www.youtube.com/watch?v=_9BGLtqqkVI

(the first one's better though...)

wrsh07• 11 years ago

The name perfectly described its behavior, and I didn't even think about other meanings. huh.

tyho• 11 years ago

Be warned that this reads everything until EOF into memory, doing something like:

    hashpipe QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 < large_file

will produce unexpected behaviour.

_prometheus18• 11 years ago

yeah, hashes are decided by every single bit including the last one, and not a single bit should be output until the hash matches. It currently buffers everything in memory, but might do this: https://github.com/jbenet/hashpipe/issues/1

(some settings dont have disk though).

hashpipe is intended for most executable use cases (usually under <50MB)

doomrobo2• 11 years ago

Loading everything into memory at once shouldn't be necessary to produce a hash of the entire input. All of the hash functions currently supported allow for incremental hashing. That means you can hash in blocks instead of all at once.

stormbrew• 11 years ago

You should be able to detect if the stream is seekable (by checking the result of 'lseek(fd,0,SEEK_CUR)') and only buffer if it's not.

Of course, if you're really paranoid, the file could get changed out from under you. But honestly you're probably screwed either way with an attacker who can do that.

gojomo6• 11 years ago

Another fun option would be to use a tree hash. The distributor of the content-to-be-verified would then envelope it in a format (to-be-defined) that includes proofs-up-to-root every N bytes. Then the verifier could stream, and know that everything it emits fits into the target hash, needing only N bytes of working space.

Caveats:

The source doing-the-enveloping will need two passes (and enough working space for the remainder-tree).

An attacker could still choose the moment-when-content-goes-bad; in the envisioned use of immediately-executing the verifier output, this might leave things in a problematic/resource-consumptive state. (Scripts could be hardened against such partial-execution failures.)

geoelectric• 11 years ago

So too much in the pipe and things get wonky? Still a perfect name!

kazinator2• 11 years ago

It doesn't have to read into memory; if that becomes a problem they can fix it. The hashing algorithm just needs a small, constant amount of state. While it is hashing, it can divert the data to a temporary file. Then it can just pass the file to its standard output.

A "hashexec" command could also be produced which passes the name of this temporary file to a subprogram:

   whatever | hashexec <hash> arbitrary --command with args

This "arbitrary" command's standard input is redirected from a temporary file created by hashexec (so no cat-like loop has to execute to feed the data).

Also, since this is for scripts, there could be an argument which limits the size. This could have default value, say one megabyte. Anyone pulling down scripts which are anywhere near one megabyte has to add explicit overrides for the size:

   whatever | hashpipe --max=4M <hash> | ...

The real problem isn't that hashpipe buffers everything, because the scripting language running the script will also do that; rather that because it buffers everything, it can be DDoS'ed with an infinite stream or whatever.

By incorporating a size limit, hashpipe could provide an additional protection measure to the next pipeline element: it protects against content which doesn't match the hash, and against content which is too large.

Idea: the size of the input could be encoded as a few digits of the hash. Then the argument is unnecessary. The hash itself tells you that the script is exactly 6059 bytes long; if you read 6060 bytes, the input is not the right one. Or the program could just stop reading at 6059 and check the hash at that point, and either pass on the 6059 bytes or error out.

_prometheus18• 11 years ago

All great ideas! mind adding some to https://github.com/jbenet/hashpipe/issues ? over time we can implement some

wyc• 11 years ago

Great, thank you for this software! Any idea if there are trusted 3rd party hashes for popular install scripts?

If a website wants you to:

curl "http://sketchyurl.com/script.sh" | hashpipe PRECOMPUTEDHASH | sh

it might be even worse, giving you only the facade of additional security.

_prometheus18• 11 years ago

That's a bigger problem, because we essentially need full PKI. My preferred solution is via http://ipfs.io -- but i may be a bit biased :)

I think we can get halfway there with a "signed hash" construction, but yeah-- PKI...

tlrobinson• 11 years ago

Keep in mind whatever script you run through this could download additional unverified code.

_prometheus18• 11 years ago

Great point!! -- \o read this everyone o/ --

Please please dont use hashpipe thinking you'll be super safe about everything. It only raises the bar a bit! It solves my biggest gripe with most `curl <url> | sh` things, which is that any MITM can own my machines without compromising the origin http servers.

(of course, if the HTTP server + page you got the checksum from is owned too-- good luck!).

jamiesonbecker2• 11 years ago

Good point - of course, this would mean that the additional code was vetted (or at least consciously included!) by someone who cared enough to use hashpipe in the first place.

DanielDent• 11 years ago

Not only could it, downloading additional code is often the entire point.

Additionally, it's common for the installer to include things like version numbers, which means the hash will change with each release.

Meteor suffers from the "|sh" install pattern. Creating a Docker packaging of it that felt safe required a lot of extra work as a result:

1) Transforming the installer file into a canonical form free of version numbers. This verifies if the assumptions made about the installer are still valid. It also enables a "latest" tag which installs whatever MDG has currently published.

2) The installer is patched so it checksums the tarball it downloads.

(To be clear, I'm aware of areas both upstream and downstream in the process where unverified code could sneak in easily. But at least I can feel good about the part I'm responsible for.)

quicksnap• 11 years ago

Since the main use case for this utility is verifying network shell scripts, it would be interesting to see a query param convention, so we could use a tool such as:

hashcurl http://load.this/script?hash= QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 | sh

gojomo6• 11 years ago

When the just-over-the-next-hilltop promised-land nirvana of content-centric networking arrives, the hash will be enough to locate & download the content – so you shouldn't even need an URL:

  $ hashcurl QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 | sh

Maybe it's even a special filesystem path, that contains (but does not list) everything-that's-nameable-and-findable:

  $ sh /everything/QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8

(BTW, personally not a fan of the opaque 'multihash' format, which obscures the algorithm-in-use to save a few characters.)

andrewflnr• 11 years ago

I'm not sure if it's really great or a really terrible, but I love the "/everything" idea.

stouset3• 11 years ago

It's not save a few characters. It's to allow for flexibility of encoding. "sha1" all but requires ASCII. Sometimes you're limited to hex, base64, etc.

_prometheus18• 11 years ago

this already works. install ipfs: http://ipfs.io/docs/install then:

    ipfs init
    ipfs daemon &
    sleep 20  # sorry this will go away
    ipfs mount
    sh /ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4sGgwQzjqtBwLX6

you can see it on the web at: http://gateway.ipfs.io/ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4...

serendipitous• 11 years ago

This sounds like DHT, which has existed for a while now. https://en.wikipedia.org/wiki/Distributed_hash_table

latchkey• 11 years ago

I love this idea, as it has bothered me for a while now that homebrew initial install just points to a script on github.

That said, now we just need someone to hack gobuilder.me and replace the binary distribution of hashpipe with something that always returns the input.

_prometheus18• 11 years ago

Great point!! working towards fixing that (the general problem of safe execution of binaries). signed releases will help.

I will release as self-describing signed package for this soon.

vortico• 11 years ago

Deployment one-liners with hashpipe will only work if hashpipe is installed, which would be equally difficult for users to install properly than the software itself. Then you'd need something like this:

Simply install using `curl hashpi.pe | bash`

deevus• 11 years ago

Hah. Nice one :)

RegEx• 11 years ago

This is a really, really interesting point!

In addition to your point, (and perhaps this point has already been made elsewhere in the thread), but the type of people that would install hashpipe would be the kind of people to avoid the practice entirely.

Perhaps there's some mechanism for convincing people to use the tool, and if that mechanism ends up being easier than convincing people to stop piping directly to bash, it sounds like it would be worth pursuing :)

_prometheus18• 11 years ago

yeah, this is just a stopgap for me. in the end i want:

  ipfs daemon --mount &
  sh /ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4sGgwQzjqtBwLX6

(this actually works, btw :)

Nullabillity• 11 years ago

Well, it'd be easier to get hashpipe into all the typical package managers than to get every piece of software ever into them (though that way would of course be preferable).

jamiesonbecker2• 11 years ago

Hopefully people will only ever pipe from a HTTPS (PLEASE!), but this is still a great idea and a big step forward!

moe• 11 years ago

I would say the author now morally owes the world an equally convenient PGP-signer/verifier.

_prometheus18• 11 years ago

you're right https://github.com/jbenet/hashpipe/issues/3

nawitus• 11 years ago

How does it protect from man-in-the-middle attacks? The man in the middle can simply replace the hash. If there's an additional communication channel to provide the hash you could simply provide the whole command with it.

_prometheus18• 11 years ago

not everything is hosted in the same place. most binaries use CDNs different from the websites. e.g. github + s3.

(see my other comments on this page-- hashpipe is only meant to raise the bar a little)

languagehacker• 11 years ago

Getting this in before the gratuitous negativity brigade starts hammering down:

This an implementation of a stupid joke looking for a problem. If someone actually needed this in their toolbelt they could use perl or even (amazingly) bash to handle this problem.

I would personally not name anything I worked on after a song from one of Weezer's shittiest albums, but that's just me.

_prometheus18• 11 years ago

many execution contexts don't have perl. i have some <10MB VMs in mind.

bfg• 11 years ago

If the code url and hash are provided at the same place, why wouldn't an attacker just MitM that and switch them both? What does this add?

Also anyone who is not serving code over https should fix that immediately.

pritambaral• 11 years ago

I think this one isn't too far from useability:

curl -o filename url && sha256sum -c <(echo "PRECOMPUTEDHASH filename") && sh filename

And works without requiring installation of third-party tools too!

comex• 11 years ago

This is non-portable, as OS X has no sha256sum out of the box. But it does have shasum from Perl, and Linux distros typically come with Perl, so 'shasum -ba256' should work...

I wonder if there is any concise way to do this without needing to save a file to disk. I can't think of one, as it requires splitting the input in two, which bash can do with >(command) but not sequentially or with the ability to communicate from the subshell to the outer one.

tbrownaw• 11 years ago

You need to buffer the entire input somewhere.

You don't get your hash without consuming the entire input.

You can't start feeding the input to the shell, until you verify the hash.

You don't want to download it twice, it case it comes back different the second time (or in case your network is slow).

At the point that you are verifying the hash, the entire file must exist on your system somewhere. Disk, memory, OCR-friendly printout, whatever.

Buffering to memory can only work if the input is "small", whatever that means. And pipelines aren't meant to do this, so you'll have to do something a bit odd (ie, confusing to anyone trying to understand your code) to make it work.

So, buffer to disk.

Or, use Perl. Perl solves everything.

e12e4• 11 years ago

Maybe we should just recommend that software is distributed as a (not detached) singed file. So:

   curl http://yolo.example.com/lulz.gpg| gpg -d - | sh

(Which of course doesn't work either, as one could replace lulz.gpg with "#/usr/bin/env sh;rm -rf /" signed by some key one trusts... which isn't so far fetched assuming some trusted entity publishes a utility for wiping a system automatically (think: shred /dev/sd*))...

Still, gpg -d, seems preferable to detached signatures (not to mention detached hashes) for this type of thing? The files can't be used without verifying, or running through gpg [ed: or other manual intervention, for shooting oneself in the foot] -- and there is a single, sane way to do that.

aablkn• 11 years ago

You might want to `set -o pipefail` on your bash because a failing process does not stop things from getting piped:

`echo OK | false | echo OK2` --> this command returns zero exit code even though false does return non-zero exit code. If you do `set -o pipefail` entire pipe will fail with non-zero exit code (of `false`).

_prometheus18• 11 years ago

great point. if you have a good idea of how hashpipe should do it better pls comment at: https://github.com/jbenet/hashpipe/issues

at least hashpipe stops the output, so this would not run the bad code. hashpipe would stop it:

  cat evil | hashpipe <some-other-hash> | sh

without pipefail, it would still exit 0, but at least the evil would be contained.

mpeg2• 11 years ago

I think this is a topic worth raising, I spend so much time auditing the "magic" in open source projects to try and find out how they work behind the scenes, and whether they are doing anything to protect me.

For example:

Pip (python package manager) tells you to run a "secure" script off the internet [0], where the only real protection is being behind SSL. But, surely I can just check the code of that script? Oh wait, it's embedding a zip file inside the python script, so I need to manually unzip it and look into it before I can start to consider it safe.

I like what CoreOS does, they don't even provide SSL download links for their isos, it's all plain text http, so they stress that you need to verify the image signature against their GPG signing key which is distributed via github.

[0] https://bootstrap.pypa.io/get-pip.py

_prometheus18• 11 years ago

I like what CoreOS does, they don't even provide SSL download links for their isos, it's all plain text http, so they stress that you need to verify the image signature against their GPG signing key which is distributed via github.

Sounds very much like TRTTD. I'll see about this with `sigpipe` (hashpipe with signatures). The fundamental problem is PKI is weak. using github isn't great either.

We need a chain of trust that's sane (CA system is not sane).

digi_owl• 11 years ago

What is "sane" when you are potentially up against attackers that may have clusters to brute force collision with?

hueving2• 11 years ago

so they stress that you need to verify the image signature against their GPG signing key which is distributed via github.

If someone has the ability to MiTM HTTPS connections, they could just as easily MiTM the victim's connection to github to return a different GPG signing key. Also, the protection requires active checking rather than coming 'for free' like it does by downloading via HTTPS, so 95%+ of the users won't bother and won't have any protection.

I think CoreOS took a big step backwards here. Just offer the download links via HTTPS in addition to the GPG key. That way people that don't bother with GPG have protection.

e12e4• 11 years ago

If you use CoreOS, you just need to get the Core OS key one time, and sign it with your own key, then let the keyservers mirror it for you. An adversary can still make sure you can't connect to any key servers -- but it's an order of magnitude easier to verify a single GPG key (even if that involves getting on a plane and talking to a CoreOS developer in person) -- than it is to verify every single CA cert shipped with your browser.

Just look at the latest CNNIC key removal: essentially Google and Firefox are saying that SSL was never secure, because we couldn't trust CNNIC, even though everyone shipped their keys.

Before that, no browser was secure, because a number of CAs were compromised, and or incompetent.

While GPG isn't some magic dust that makes trust easy -- it's crazy to claim that SSL is somehow more secure, or more importantly, easier to trust.

[edit: And that doesn't even touch on the number of lines of code involved, the directness of the path between what is signed, and who will use it (security of the web server, dns etc not an issue if you know you can trust the GPG signature...]

mpeg2• 11 years ago

SSL does nothing to guarantee the integrity of code you download and run off the internet.

I'm not worried about MITM attacks, SSL provides adequate protection against those in most cases; but if someone gains access to the coreos systems SSL would not help.

In this case, I have a GPG key saved for the coreos image signing, which hopefully is done offline and I got that through a third party which I have a certain amount of trust on (github), and I can always verify it on other channels.

cheald• 11 years ago

I am rather amused that the author provides precompiled binaries without hash checksums. Heh.

humanarity• 11 years ago

This is cool. What was your thinking about potentially hashing whole commands (and even their executables) in addition to their parameters? You could be the hash (with as you say PKI) gateway for all execution... :)

ForHackernews• 11 years ago

Maybe OSX developers should just stop asking people to pipe random scripts into their shell. This is a nice idea with a clever name, but it's kind of a false sense of security.

dcposch• 11 years ago

I don't see how this gives additional security. When you run

   curl https://project.com/script.sh | sh

...you're relying on three things:

1. That the people running the project are trustworthy

2. That the server hasn't been compromised

3. That the CA system will ensure you're talking to the correct server

(I can think of recent news stories where each of those were violated.)

If, instead, you go to `https://project.com`, read the instructions, and paste in the following command...

   curl https://project.com/script.sh | hashpipe <somehash> | sh

..then you're relying on those same three things! Someone who wants to serve a modified version of `script.sh` just has to serve modified instructions as well. You also have a new requirement: you have to get a trusted install of hashpipe first.

vincentclair• 11 years ago

You add an extra layer of security, if you receive the hash over a different channel, such as encrypted messaging.

That means, that even if the CA system is broken, you only execute the intended contents.

ZoFreX• 11 years ago

That means, that even if the CA system is broken, you only execute the intended contents.

No it doesn't - how are you installing hashpipe? ;)

xnyhps• 11 years ago

There is another assumption:

4. You download the script successfully.

It is trivially easy for a MitM to interrupt the download of the script between two TLS packets, without any CA or server compromises. When you do:

    curl https://project.com/script.sh | sh

Then sh happily executes instructions as they come in. It may be that the script starts by moving important directories aside or by creating large temporary files, so if the script is incomplete the user may end up with a broken system. Or maybe if you're really unlucky an attacker might manage to truncate "rm -Rf /..." to "rm -Rf /".

With hashpipe, you are at least guaranteed to have the complete script before you run it. I still don't like the practice, but it is better.

jakeogh• 11 years ago

Cool. The use case that comes to mind is knowing you are going to get the file you think it is if you are looking it up on some local or remote hash indexed storage scheme.

It would be nice if we had (it prob exists already) a standard block based scheme so files larger than the RAM could be handled without writing to disk. Making something like a .torrent for every file and using that as the per block checksum and then checking the hash of the concatenated checksums could do it. Right? I'm sure there is a real name for that but idk it.

Really this seems like something that the FS should be asked to do. ZFS seems to be close, but I haven't found out how to address a block by it's hash yet.

LukeHoersten• 11 years ago

I implemented multihash in Haskell: http://hackage.haskell.org/package/multihash

Hashpipe – Pipe iff the hash matches

Related Stories