_prometheus

awesome! :)

This is great! SOOO useful to be able to do mounts w/o fuse, w/o kernel extension, w/o root. This is a HUGE UX thing, specially for macs.

Fuse is a really great idea & project, but sadly today impls require a lot of install steps in some platforms, and some have painful bugs/UX. Amazing if we can mount VFSes from Go without those hurdles.

Datasaur looks awesome! Can't wait to try it out. Congrats on the launch!

Curious about data security and privacy? How do you guarantee privacy? Is there some cryptography or secure enclaves used? Some sets of documents (and email) are super high trust.

Guessing the on-prem version is probably safest route

Hello! o/ We've responded to a number of points here.

You can check out our roadmap

IPFS is unworkable... making it usable in the next few years

Absolutely not.

The OP brings up a lot of great, useful feedback for us, and we'll respond to it.

But the OP is also simply wrong in saying it's "not usable". There are millions of end users benefiting from IPFS, 100Ks of libp2p nodes, we see PBs/mo of traffic in the infrastructure that we run, and millions of daily requests to our gateway. Look to fully decentralized applications and systems like OpenBazaar, Textile, Dtube, and others.

Beyond that, we're well aware of the many shortcomings, and working on them. We're unfortunately spread thin across a lot of projects (IPFS, libp2p, filecoin, ipfs-cluster, etc), but each is seeing significant growth and improvement.

You can see the long-term IPFS roadmap here https://github.com/ipfs/roadmap

Decentralizing how we link, address, and move content is the basis of IPFS and Filecoin both. Both are solving different parts of moving our digital infrastructure to infrastructure that we control, that we govern, and that we are not held hostage by.

While we still have a long road to go w/ IPFS usability, it’s worth pointing to what works today -- there are millions of end users benefiting from IPFS, 100Ks of libp2p nodes, we see PBs/mo of traffic in the infrastructure that we run, and millions of daily requests to our gateway. Look to fully decentralized applications and systems like OpenBazaar, Textile, Dtube which use IPFS and paving the way. New technology -- especially new technology that seeks to build new platforms from the ground up -- is hard and extensive to build, and to polish. We’re working on it, and though nowhere close to where we want to be yet, there’s a lot of utility already provided.

Re Filecoin incentive structure -- this comes from recognizing that computers are not all the same -- there is huge utility to be had in dedicated infrastructure running 24/7, well maintained, high performance spread out around the world. P2P systems of the past have failed to match the reliability, uptime, and performance levels of centralized systems. Cryptocurrencies enable the creation of Open Services (like Bitcoin) that run public infrastructure w/ high uptime and reliability guarantees. With Filecoin, we aim to bring that to file storage and distribution. More here: https://www.youtube.com/watch?v=6h2WNxEV8q4

Exactly.

P2P systems historically have dealt with the problem altruistically, or with limited tit-for-tat, which both work in many cases, but have so far failed to work for large scale long-term resilient systems.

This is where Bitcoin managed to do something remarkable: achieve high uptimes typical of the best centralized systems, through a very clever, but still open and permissionless, economic incentive structure. Markets have been shown to be extremely useful to create a robust Open Services. More on these ideas here: https://www.youtube.com/watch?v=IfLIoOr4p0A -- we think this kind of thing is going to lead to extensive, global, public utilities run w/ internet-native money.

But it will take a while-- this stuff is extremely difficult to build right now-- it feels similar in nature to very early Web, or pre-unix systems. Lots of hand-rolled primitives, many with the capacity to cause serious failure (not very old cryptography, and complex security questions). Perhaps better programming languages will help us build these systems dramatically faster/easier. For now though, you can see the entire blockchain space wrestling with these problems.

This is excellent feedback! Thanks!

We even created a plugin (https://github.com/almonit/almonit-plugin - unreleased officially yet!) for websites that combine IPFS with ENS (a decentralized DNS).

This is great! We’ve been wanting something like this for a while-- and there are a bunch of utility in bringing ENS and IPFS together.

A lot of the problems you mention are known, and being worked on. Let me describe a bit more in each.

Technically, IPFS work well for us, but we made sure to have a server seeding our website at all times (where we suffer similar problems like the author of the post describes).

The IPFS content model requires somebody with interest in the data to keep serving it. So for now, yes you need to keep some ipfs nodes with the content around. For example, we serve all our content (all our websites are distributed w/ ipfs) using ipfs-cluster, which connects to the gateways. We’re working on ipfs-cluster and filecoin as the ways to solve this. Ipfs-cluster for when you want to run your own infra (or a community gets together), and filecoin for when you want to hire someone else to serve it for you.

Even then, we still had to make sure that the website is available in all main gateways, since most people don't run their own IPFS daemon. The strange part is that sometimes content is available in one gateway, but not in others. Or sometimes it's available in one gateway, but we can't get it on our local IPFS node.

This is a big problem that we’re working on right now. Many of our recent releases aim to fix / reduce this problem. The nature of this comes from content-routing scaling. We’ve detected this getting worse as the network grew orders of magnitude, and we’re working on fixing this right now.

I understand that IPFS is not a blockchain, so I can't expect all the nodes to have the same content. But I do expect the main gateways to communicate more directly with each other.

Yes, definitely. We agree-- on it.

Conceptually, IPSF websites are a bit like sending your website to someone via an unreliable slow mail; i.e., it's not that attractive. You can make it somehow more dynamic, using what they call IPNS (it allows you to update your content). But the result is so slow, that even the most devoted monk would lose his patience eventually.

IPNS key names are just not working well -- and getting worse with scaling. We’re working on fixing it, but lower prio than other more important problems. For now, we’ve been directing people to use DNS -- check out https://docs.ipfs.io/guides/concepts/dnslink/ -- these should be fast for you. But yeah, figure you know about this and may just be avoiding DNS in favor of more decentralized tools (ENS, IPNS key names).

Also check out https://dev.peerpad.net/ -- this is an experimental tool w/ dynamic content (give it 10-20s -- unfort takes a while to “get online” -- this is the dev pad). Once peers are connected you get a pad that’s distributable across ipfs nodes.

A workaround is using a decentralized name system, like ENS. This works very well, but the result are still static websites. No comments or anything really interesting happening.

+1 for ENS. And, you could use ENS to point to something like the content hash of peerpad above (see the latest content via `ipfs dns dev.peerpad.net` or `dig TXT _dnslink.dev.peerpad.net` == /ipfs/QmWbsqqqG9YpNYDt5afp6HY8TrKMtCtdGUtUfgkS9fRYeH -- and this would be a _static_ html5 bundle that gives you a _dynamic_ local app. The hard part is backing up your content beyond your own browser-- pinning it to an ipfs-cluster or other tools, etc.

Anyway, the picture is clearly not there and usable yet, but the implications are big: you can get fully distributable p2p apps w/ fully dynamic content. (you could even encrypt the app bundles themselves, but this needs an extension to decrypt browser side) -- this feels like this: http://www.accademia.org/wp-content/uploads/2014/01/slaves-b... -- the form is emerging, but much work to be done.

Those websites are censorship-resistance and very robust, you don't have to worry about ddos attacks. But then again, how many people worry about such things?

A lot.

* https://en.wikipedia.org/wiki/Block_of_Wikipedia_in_Turkey and https://ipfs.io/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34is...

* https://en.wikipedia.org/wiki/Censorship_of_Wikipedia#China

* https://firstmonday.org/ojs/index.php/fm/article/view/9402/7...

That said, I still like the concept of IPFS. We are exploring a few options to add dynamic behavior to that now, where the dream is to mimic existing services in a decentralized way. Surely, they won't work as well, but the pro would be that it will be controlled by the users, and that they will be able to survive financially with no ads.

It takes a long while for all of this kind of tech to come together. Keep at it-- you make significant progress YoY, and it adds up. What we can do now in 2019 is vastly superior to what we could do in 2015. Big innovation leaps take significant time to develop -- but the good news is _a lot_ changes decade over decade:

* https://en.wikipedia.org/wiki/History_of_the_Internet

* https://en.wikipedia.org/wiki/History_of_hypertext

* https://en.wikipedia.org/wiki/History_of_personal_computers

* https://en.wikipedia.org/wiki/History_of_operating_systems

* https://en.wikipedia.org/wiki/Cryptocurrency#History

* https://en.wikipedia.org/wiki/History_of_self-driving_cars

Hey HN, this is jbenet -- an author of Filecoin.

We think it's great that people ask hard questions, and get involved. It's great to see others studying our work and we really appreciate the open discourse. There are a few things from this article I’d like to address. (Despite the length of this post...) these are quick comments, and not a proper in-depth response.

- (a) The article gets some things right and some things wrong -- there is good summarizing of several of our projects, and discussion of many difficult aspects in these projects. The article discusses many technological aspects in good depth, and highlights difficulties in building these systems, aligning incentives, and the trials of past projects. The article also has significant inaccuracies. For example, the sale figure -- which appears in the title and impacts the analysis -- is incorrect. We raised $205M -- officially here: https://protocol.ai/blog/filecoin-sale-completed/

- (b) The authors chose a provocative title. As some commenters have already pointed out, the conclusion is “[we] believe that it is not one.” Despite Betteridge’s law ( https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines ), many people who only read the headline will come to the opposite conclusion, and now we (not they) will have the burden of correcting those misunderstandings. Provocative titles, though they may drive imagination and clicks, can do a huge disservice to everyone in the space, and contribute to misinformation. Most people will only read the title, maybe the abstract, and use that to form and drive opinions. We choose titles of our research with diligence and care, and hope others do the same.

- (c) The article has a great technical overview of the Filecoin stack, and how it fits with IPFS and libp2p. This is a large structure with many pieces, and it is rare to see articles grasping how all the pieces fit together so well, and then explaining it cogently. In particular, it’s great to see this article diving deep and discussing advantages and disadvantages of low level technical structures (multihash, ipld, libp2p, and more). We modularized everything in the hope to generally improve peer-to-peer systems, and improve reusability. We hope these components will be useful to the author’s Tribler project (a network similar in goals to Filecoin), and we hope that we can also learn from and leverage solutions they have made.

- (d) many of the objectionable things described in this article are common in ICOs in general. Put another way, consider those claims also in terms of other significant token sales, such as Ethereum, Tezos, Polkadot, Blockstack, Cosmos, Golem. People were saying similar things about Ethereum when they did their sale in 2014. Perhaps worth doing a survey / analysis over all of them, comparing and contrasting the different things groups have done, how the ecosystem has improved, and suggest new directions.

- (e) It’s worth mentioning that the analysis gives a definition of a ponzi scheme, but their discussion does not map to that definition. Instead, the discussion centers on claims about future investor sentiment or speculation as the driver of value in the token, which is not the only way to establish value in token networks, and ignores the value of the services provided. That kind of analysis does not work for projects like Ethereum and other live and functioning crypto tokens. If the network is useful, and there is a way to generate or introduce value, by providing new or better services, and if the network can capture that value in the token itself (important step), then the tokens can hold value, based on the utility of the network as a service and not just or primarily speculation. Networks like Ethereum, Bitcoin, Zcash, and Filecoin aim to provide useful services, and much of the value stored in their tokens will be thanks to the utility of the networks.

Perhaps it’s worth pointing out that most crypto token projects are compared to ponzi schemes at some point :(

  - http://www.google.com/search?q=is+bitcoin+a+ponzi+scheme
  - http://www.google.com/search?q=is+ethereum+a+ponzi+scheme
  - http://www.google.com/search?q=is+zcash+a+ponzi+scheme
  - http://www.google.com/search?q=is+tezos+a+ponzi+scheme

- (g) We do share the legitimate concern that ICOs need stronger accountability, and some are structured in a way that leads to abuse. The community as a whole needs to raise the bar on accountability and ethical behavior. We have taken significant steps in this direction, not just in our sale but to improve the ecosystem -- the SAFT project, which was a gargantuan undertaking that many other networks are now using, is one example. Many other networks are introducing and improving structures. We believe token networks present a very important new way to form capital, with promising advantages to users, investors, and creators, but the space is still in its infancy, and significant changes are still ahead. Token sales have improved dramatically in the last three years, and we hope they continue to improve to find the right balance and protection of the interests of all parties involved with the network.

Thanks, Juan

It's coming shortly (days), it's a WIP that needs significant polish, but the core result is there.

Thanks Craig and YC. Ton of fun.

That's precisely one of the innovations in the new Filecoin paper :) -- we call it Proof-of-Replication. Papers coming soon.

There are people in turkey replicating this snapshot right now. people browsing locally get it there. Even if all connections to outside the country get shut down it can be distributed locally.

the table is so small you don't actually need to decode varints. any serious application needing speed would build a lookup table for all the common values -- in fact we will. size does matter. speed does matter.

sounds interesting. we're definitely interested in figuring out ways for these protocols and formats to be as useful to others as they can be

Yeah you're right, we've been meaning to add MD5. Filed issue: https://github.com/multiformats/multihash/issues/71

yep, exactly :)

See https://news.ycombinator.com/item?id=13740369 -- does that make sense? the idea is to treat those parameters as "part of the hash digest value", and write wrappers for siphash and functions like it, s.t.:

  mh_siphash_digest(input, length, rounds) {
    digest := siphash_digest(input, length, rounds)
    mhdigest := concat(rounds, digest)
    return mhdigest
  }

  mh_siphash_verify(input, mhdigest) {
    mh := multihash.parse(mhdigest)
    rounds := mh.digest[0]    # the rounds
    expected := mh.digest[1:] # rest
    actual := siphash_digest(input, mh.length, rounds)
    return expected == actual
  }

@hobofan Yeah, exactly. We've explored this by making the value of such hashes (when generated and checked with multihash libs) literally be the `<hash-rounds><hash-result>` pair. This avoids complicating the table or multihash model/implementations. We can just redefine the hash function verify to use the first bits as the round, etc. We haven't done this yet, but we may.

Yeah this is a good point, which we discussed somewhere (not finding the issue atm). The resolution was to treat those as different hash function codes, because the normal usage of truncating hashes by chopping off bits is extremely common. We had direct use cases from past experience trying to help old systems that did things like take a sha2-512, trunc to 256bits and use instead of sha2-256 (for the speedup on some archs). So we saw the need for _literally_ a different size of the exact same function.

When the functions encourage it, we support the addition of the specific different constructions (if named): https://github.com/multiformats/multihash/blob/master/hashta... -- we did a silly thing with Blake2 where we imported all the valid numbers. (this is suboptimal in table space, but super explicit).

Are there other functions you think we should add for the sha2-256/512 set?

Multihash's design begs implementations to validate these things by allowing the sender to arbitrarily truncate them

If the sender is manipulating the hash you get (i.e. changing the length prefix counts), you're already in huge trouble. They could change the code and the value too. The threat model here is that the hash you have cannot be altered by the attacker. If the attacker manages to truncate a stream to get you to think it's a shorter hash, the attack fails as you have the length to tell you what you should be expecting. (again, the crux here is that if the attacker can change the length bits they can probably also change the function and own you anyway)

Also-- as noted in https://github.com/multiformats/multihash/issues/70 -- we should make implementations allow clients to lock the hash function and length combinations they want to use, so that attackers cannot manipulate those parameters.

Truncations happen in many sizes. The combinations here are not helpful: N functions * M commonly chosen sizes. Simple assumptions: 20 functions * 10 sizes = 200 codes. already not good.

You can print a human-readable version of multihash as:

sha2-256-20B-<hex>

Using the same values of the table. But again, it is preferable that it travels in-band with the values, as otherwise it will likely get chopped up and placed into out of band context.

Codes reserved for app-specific things should satisfy your concerns. coming soon: https://github.com/multiformats/multicodec/issues/39

One thing you should consider about using full names (like "sha2-256") is that you still have to keep a table. What does "sha2-256" even mean? you know from context. you know that maps to a particular algorithm and particular set of implementations from context.

Writing a table and distributing is cheaper than writing a new implementation of a new hash function and distributing. The "distributing code" part still has to happen.

I think what you would like is if some body (NIST + all others who make hash functions) was maintaining the registry. That's fine, we can give this table onto IANA (though that's harder to update than a PR on Github).

Attackers can exploit that by cutting off a stream. it's best to be explicit.

That's right. It's really important to make sure there is restrictions on what hashes to use if your system is receiving hashes and only checking them for self-consistency.

Particularly relevant is "Crypto Extensibility" (formats and protocols to be able to extend a protocol), vs "Crypto Agility" (the use of Crypto Extensibility to use simultaneously a large variety of algorithms, with the key feature that one can be downgraded to an old/possibly broken hash.

AGL describes it well here: https://www.imperialviolet.org/2016/05/16/agility.html

---

I've filed https://github.com/multiformats/multihash/issues/70 to track this.

Hey everyone. The SHAttered break made us release a new website for Multiformats: http://multiformats.io which includes a page explaining Multihash -- http://multiformats.io/multihash --

The page goes through a bunch of examples of how a multihash would work in practice, why it is useful to do this, and goes through some FAQs.

I'm writing a post about all this that I'll push out soon, but wanted to drop the site here now.

Please, as you make new systems or start to upgrade your systems out of SHA1 (or MD5!!!), please use Multihash so that your upgrades will be much, much easier in the future. This makes rolling up a matter of recompiling some tools, and by far not facing the horrible breakages that occur when tools and scripts assume certain things (like the hash digest of git will always be 160bits... now git faces breaking a lot of things and may have to stick with 160bits of a truncated SHA3 or BLAKE2...)

Oh also-- definitely check out BLAKE2, it's a fantastic hash function from Zooko, JP Aumasson, Samuel Neves, and Christian Winnerlein -- https://blake2.net -- use it for all your hashing needs! So much faster than SHA2, and has the same "unbrokenness" that SHA3 enjoys. (And, I believe deep cryptanalysis has gone into BLAKE, Keccak is not particularly safer)

The Multiformats site also has a Multiaddr description, but that's far from complete and doesn't have the examples. The other multiformats arent well explained on the website yet. Sorry, coming soon :) we wanted to get this out ASAP -- PR's accepted at https://github.com/multiformats/website (oh and yes, TLS certs coming too)

Hey Everyone, thanks for the great discussion and excellent questions. Was surprised to see this rise up to FP, given we have not updated the site in ages. :] Thanks for the great discussion here! Unfortunately i can't give much in form of answers as time is super punishing right now :(. Good news is we're hard at work, and we'll have a big announcement within 2 mo. ;) See you then! --- Oh also, big shoutout to Sia, Storj, & Swarm. They're all doing solid work on this. One thing that we've been working on is how to increase collaboration, interop, and shared upside across our systems. Stay tuned!

(I'm jbenet, of IPFS, Filecoin, & Protocol Labs)

Take a look at https://github.com/ipld/cid and the fs: handler and browser support mentioned in other comments.

@Ericson2314, yeah thnaks for bringing this up. As was mentioned elsewhere:

* CID is finished and live in go-ipfs@master and js-ipfs@master. We haven't announced it widely because go-ipfs@0.4.5 is still to land. (ooof)

* IPLD spec needs work, but work continues.

I wanted to add that:

* please contribute to CID to get it where you need it to be.

* i am personally very interested in defining IPLD data structures and their operations in a good language. This will probably be transpiled to the IPFS impl language, or compiled down to WebAssembly and run in a small WA VM (the web of datastructures)

Good to see other people are inventing AppFS ( http://appfs.rkeene.org/ ) :-)

I'll take the construction "other people are inventing <the thing i invented some time ago>" to mean that you think you came up with this first, or at least prior to Nix or IPFS. And thus ":-)" to be a bit sarcastic and unhappy, instead of genuinely happy.

Similar times:

* http://appfs.rkeene.org/web/timeline?c=78c60b0c9e7da1c9&unhi...

* https://gist.github.com/jbenet/8f000606f2009495c56177f6ca2c1...

* https://github.com/ipfs/ipfs/commit/8004db75262fcd29399d6c7f...

* https://github.com/jbenet/random-ideas/issues/19

* i think the Nix people probably came up with this way before either of us

* and i am willing to bet everything that at least 100 other people (maybe thousands) have came up with this exact same idea over a decade before any of us...

In fact, most of the best "original ideas" in the IPFS body of work were probably first discovered decades prior.

I repeatedly see people succumbing to sadness over multiple discovery. It shouldn't be sad, it should be a happy event, as it confirms our own thoughts and presents an opportunity for collaborations. :)

Further thoughts here: https://gist.github.com/jbenet/8f000606f2009495c56177f6ca2c1...

You can make oblivious storage (and even ORAM) on top of IPFS, without too much work actually. A critical point here is that making something oblivious by default is a nonstarter for 99% of users on the internet, because major consumer and corporate applications would never use something that (a) adds that much latency to requests, or (b) may have pushed "bad bits" to their computers. This is a design constraint BECAUSE of adoption.

The point is to establish IPFS as a base layer, and build the oblivious storage platform on top.

pfraze: this is great! such a nice surprise to see the recorded demo today in HN. <3 keep up the good work!