Paper review: IPFS – Content addressed, versioned, P2P file system

https://muratbuffalo.blogspot.com/2018/02/paper-review-ipfs-content-addressed.html

by mad44 • 8 years ago

227 108 8 years ago

muratbuffalo.blogspot.com

amelius3• 8 years ago

I don't understand his argument surrounding scalability. Isn't IPFS naturally scalable?

zekrioca3• 8 years ago

His point was about scalability within centralized (logical) services, which people often confuse or understand them as a single point of failure or not scalable. But that's not true, as he pointed out there.

mmjaa3• 8 years ago

I want to have IPFS services built up to the point that I can rely upon it so I can .. take the 100 or so of my peer group, and give them access to Content I Create.

I think I can have a machine up and running in my life that serves the principle nodes, and if a few of my other peers are also capable, we can build a scalable net among us that would serve the rest.

This is a very different scenario to mass-scale datacenter levels of delivery, and at a much more localised scale.

Among my peers, it'll scale just fine.

(So its mostly about the UI and presentation... I don't mind sending my friends a funky/cool URL. If that URL solves the IPNS->IPFS issue .. and if we also can get some pub/sub channels happening, to make the URL-swapping even more fluid .. wrapped up as a standard distro item .. )

burkemw3• 8 years ago

I understand part of the scalability concern being about the physical network capabilities ("Today's networking ecosystem evolved for the client-server model, what kind of problems could this create for switching to peer-to-peer model...") as opposed to the scalability of the IPFS protocol and node implementations.

woah• 8 years ago

Brings up a lot of good points, but his question at the end “Does IPFS mean I may be storing some illegal content originated by other users?” is so off-base, it casts doubt on the whole rest of the post. I only have a passing familiarity with IPFS, but I know that you only store files that you have explicitly chosen to store, or “pinned”. How did the author miss this?

klint• 8 years ago

It's pretty easy to miss IMHO. The project could probably benefit from having this spelled out on the homepage.

aw3c2• 8 years ago

I thought content was shared even if not pinned until it is garbage collected?

Jtsummers5• 8 years ago

If you've downloaded it, yes. If you run IPFS on your laptop and you haven't viewed some content, you won't be sharing out that content.

cypherpunks01• 8 years ago

Content that is viewed is stored temporarily, yes, not random unvisited content.

ahnick• 8 years ago

As I understand it though, this wouldn't be the case in the future if one becomes a Filecoin storage provider in which case one could be holding some illegal content. IIRC though some sort of blacklisting protocol will be provided in order to respond to DMCA, NSLs, etc, so I think Filecoin storage providers are covered in that regard.

zekrioca3• 8 years ago

It does not "casts doubt on the whole rest of the post". It is a legitimate question afterall, and the same one faced nowadays with so many files being removed from websites. He questions how could one do it in a decentralized system since copyright issues will show up as well.

Jtsummers5• 8 years ago

It's a legitimate question, especially for those familiar with other similar distributed systems. But it's easily answered and the article leaves it unanswered.

chc4• 8 years ago

If you read the white paper, it pretty clearly says that your node will cache files that peers request in order to build reputation on the network. Maybe IPFS doesn't actually do that currently, but it could and the whitepaper says it should, so it is a very valid question to ask.

edit: double checked the whitepaper, and it says "In the case that a node has nothing that its peers want (or nothing at all), it seeks the pieces its peers want, with lower priority than what the node wants itself. This incentivizes nodes to cache and disseminate rare pieces, even if they are not interested in them directly."

edraferi2• 8 years ago

I believe this is correct.

I'm not sure how IPFS addresses this, but one method is to have an oracle that reports DMCA takedowns. Nodes can achieve takedown compliance by subscribing to the blacklist.

Obviously there are lots of technical challenges. Just a few:

- Who gets to add entries to the blacklist? - How do infringement victims know that's the right person to contact? - How do you avoid creating a central point of failure? (at minimum nodes need ability to choose their blacklist provider) - Since it's all content-addressed, how do you handle near duplicates? e.g. same movie encoded in different formats (basically have to punt on this one unless you implement some kind of contentID, which is a whole other problem)

Anyway, you need to have some figleaf policy so people have a clear way to opt-in to compliance.

I would be completely skeptical of this approach except that it appears to work for adblockers. Extensions like uBlock Origin let users choose from a several third-party blacklists, with sensible defaults. Of course, there are fewer ad servers than pirate movie copies...

thinkcontext• 8 years ago

I thought software distribution would be a no brainer. There are plenty of projects that still have lists of mirror sites, using IPFS would be much better from a bandwidth and security perspective.

sneak• 8 years ago

How many thousands of times per day are copies of software downloaded over transoceanic links when many hundreds of copies of that same data exist already in the user’s own county or even building?

These efficiency gains are real; major distributions should adopt this ASAP.

heavenlyblue• 8 years ago

But by downloading the distro I am not applying to be a distributor of that distro.

hoschicz5• 8 years ago

Does IPFS actually take the peer's actual geographic location when picking one? I thought they just search for peers in the local LAN and if they don't find one, they just go for the one that's closest to them according to their PeerId (Kademlia DHT)

hoschicz5• 8 years ago

There already is Bittorrent.

The distros are getting distributed over Bittorrent and since more people have it and know how to use it than IPFS, there is no point in distributing it over IPFS.

Jtsummers5• 8 years ago

As a host, though, there's little reason not to offer both. It offers the same benefits as bittorrent but a slightly different network, and also canonical domain names using IPNS which would be a nice addition for hosting their content.

fiatjaf2• 8 years ago

Not the same use-case.

Today you have mirrors, tomorrow you could have multiple IPFS gateways, which would just take a hash and return a file.

Besides that, IPFS could be used to fill the BitTorrent niche -- so it would replace both mirrors and BitTorrent.

It's easier to use than BitTorrent and less frightening too.

detaro• 8 years ago

IPFS has some benefits if I understand it correctly, e.g. I think it can share duplicate data between multiple trees, while Bittorrent can't share blocks between multiple torrents. Still understandable that distros with working infrastructure won't jump on every new thing.

jacksmith21006• 8 years ago

What is not?

toomuchtodo• 8 years ago

Globally distributing the Internet Archive.

bra-ket• 8 years ago

Filecoin

ldiracdelta• 8 years ago

All IPFS needs is less install friction. The local IPFS server packaged as a chrome extension is what is keeping it back:

https://github.com/ipfs-shipyard/ipfs-companion/issues/248

diggan7• 8 years ago

IPFS is really trivial to install in most OSes and distributions. If you don't want to run it as a daemon via terminal/systemd/whatever, you can either run the browser extension together with ipfs-desktop and it's GUIs all the way down.

https://github.com/ipfs-shipyard/ipfs-desktop/

hoschicz5• 8 years ago

TLDR: What's keeping it back is Web standards. Extensions cannot initiate TCP connections and there is no IPFS network made of (possibly hybrid -- both protocols) nodes that are able to use WebSockets instead of TCP.

The js-ipfs daemon can only fully run on Node.js where it has access to real UDP/TCP sockets. (I don't see this as a clear proclamation in their readme, so don't take this for granted)

diggan7• 8 years ago

The js-ipfs daemon can only fully run on Node.js

Not sure what "fully" refers to here, but js-ipfs in the browser can fetch/add content like a normal IPFS node via either websockets or webrtc. Although, TCP connections would make a lot of things, a lot more efficient and nicer.

book_mentioned2• 8 years ago

Interesting to consider WebTorrent vs. BitTorrent in this context; BitTorrent has grown much closer to mainstream but WebTorrent doesn't seem to be growing.

Klasiaster• 8 years ago

Paired with an anonymity layer and TOR hidden services for direct communication it is really solving a lot of the problems present todays Web, where DPIs can mess with the unencrypted HTTP contents, DNS responses faked, web servers and CAs hacked and centeralized services decide on content availability.

HugThem• 8 years ago

The web already is decentralized. Most HTTP requests are fulfilled by a computer that is part of a CDN, not by the publishers computer.

IPFS is like Wikipedia, Uber, AirBnB. There have been encyclopedias, taxis and hotels before. But now it's easier to participate.

My computer has this article cached right now. So if my neighbor want to read it, he could get it from me. The infrastructure is just not there yet. IPFS will add it.

nodja• 8 years ago

I disagree, the internet is very much centralized.

In your example, this very post on HN points to an https url. An url that for now, means this article, but years from now might land in a 404 page, or a dns not found. Sure I can use the internet archive or find your post here on HN and ask you for it. But those are alternative services, they're not part of http.

IPFS ensures that a url means one thing and one thing only, if changes, the url changes. No censorship, no ninja edits, as long as someone in world is willing to host that specific url, the ipfs protocol will serve it.

the-dude• 8 years ago

But will there be a server which serves it?

skybrian• 8 years ago

Immutable data has its place, but you wouldn't want to use that for Hacker News because reloading the page wouldn't show new comments.

Also, when someone edits a comment here, usually that's either harmless or beneficial.

alex_duf2• 8 years ago

This may be an overblown response, but I can't help to give my opinion about centralisation. I think the web is decentralised, but has never been that centralised. What I'm about to say is about the Internet in a broader meaning, but applies to the web just as well.

A few actors own most of the servers (aws, google, ovh, azure), a few actors own most of the personal data (google, facebook, amazon) and a few actors own the pipes to transfer the data between users and the above service (depends on the country, but on average 3 to 4 provider per country, often the same across borders). The fact CDNs are spread geographically has no impact on the balance of centralisation, as what matter - in my opinion - isn't about geographical distribution, but about power or ownership distribution.

It's also very vertical due to the way we consume it (think asymmetrical bandwidth). We mostly depend on someone else's computer, or someone else's service, and has nothing to do with the utopic beginnings of the web when people were self-hosting their content on their machines.

An example would be streaming: we only download stuff from Netflix or Youtube, when technologies like Bittorrent or webtorrent could be leveraged to limit the reliance on one actor. Technologically these actors aren't really single points of failure as they've learned to be reliable, but it's a matter of who has control over the Internet.

JumpCrisscross• 8 years ago

A few actors own most of the servers

OP is measuring on a graph where each node is a server. You are measuring on a graph where each node is a person. "De-centralisation" can usefully apply to both. The former measures resilience against technical risks. The latter against sociopolitical risks.

mmjaa3• 8 years ago

IPFS should be enabled and treated as a first-class service by my distro. I install said distro, it comes with a folder - and a basic service - upon which I can replace Dropbox, Facebook, and Instagram - for my target peer group, i.e. any of my friends and family who will also run the very same distro, for this purpose.

Like, plug and play OS support, and its a go.

braderhart2• 8 years ago

I agree. I am working on an Arch distro (may roll my own for containers) that includes secure decentralized storage and configuration by default, and a federated marketplace for sharing your own local and server resources.

diggan7• 8 years ago

That sounds really interesting, do you have a link to a work in progress or something like that?

nerdponx• 8 years ago

Isn't that kind of thing already handled by package managers?

ericfrederich• 8 years ago

Isn't this open for everyone to see though?

I think you want something like KeyBase for encrypted storage which can be shared among groups, right?

mmjaa3• 8 years ago

Isn't this open for everyone to see though?

Once the scale problem starts to be solved, the question becomes discoverability. And the answer to that is, there is an infinite namespace of things - one might suggest it up to the distro maintainers to adopt policies - behind which one can hide ones content. Like, if the distro has a button that says 'publish your stuff' and a button that says 'keep this stuff secret' but inform your closest friends when there are updates so they can get approved copies, well .. thats a distro thing.

QasimK3• 8 years ago

I was thinking of something similar for applications. Just earlier today I needed a large dataset for some machine learning with Jupyter notebook in a git repo. I don't want to store the dataset in the git repo due to the size, and I don't really want to deal with "downloading the file if it isn't already downloaded into some location".

What would be awesome is something like this little bit of hypothetical Python code:

  import ipfs
  file = ipfs.file(FILE_HASH, keep=True, share=True)  # THIS LINE
  with open(file) as fin:
      # Do stuff like a normal file.

No need to distinguish between local files and cloud files! You can choose whether to keep the file permanently or temporarily, and whether to share it back to other IPFS nodes. You would probably need a system service to do this seamlessly as you suggested.

Sure, it might take a some time to download it at first - but you have to deal with that either way. On subsequent runs, it all just works. (I assume for popular enough files you don't have to worry about the availability of the file, especially if people tend to share back by default).

The big benefit being it is simpler to share datasets and actually use datasets - you can do both at the same time with one little file content identifier (FILE_HASH above).

Jtsummers5• 8 years ago

IPFS includes a way to mount it as a filesystem. So if you know the IPNS URL for some data, or the hash, you can access it like accessing any other file. If it's already downloaded then there shouldn't be any overhead versus just opening a file, if it hasn't it'll download it. You oughtn't need any extra library or logic for it.

voltagex_• 8 years ago

Windows has been experimenting with variations of remote-files-as-local via OneDrive for some time now. I really think it needs to be baked in to the filesystem or at least the file browser to be successful, and be absolutely transparent for the application that's requesting the file.

restuijs• 8 years ago

Why not something like dat instead? Genuinely curious.

stefpr• 8 years ago

Indexation of content on IPFS thru Atmos (cryptocurrency). Watch this one.... might really be the "killer app" you are all looking for. link : Novusphere.io

hoschicz5• 8 years ago

I've been following the development of IPFS for the last few years and I'm getting the impression that it's fizzling out. Maybe Protocol Labs are working on Filecoin now (that's what's bringing them money), but they never really finished the pubsub implementation and js-ipfs still can't run in the browser because there is no IPFS network that communicates using WebSockets and not TCP/UDP.

xwvvvvwx2• 8 years ago

Isn't filecoin built on top of IPFS?

hoschicz5• 8 years ago

Yes, it is, but Filecoin is not live (yet).

kodablah2• 8 years ago

they never really finished the pubsub implementation and js-ipfs still can't run in the browser because there is no IPFS network that communicates using WebSockets and not TCP/UDP

I don't believe the pub/sub is critical functionality (progress at [0], but it's really just chatty). Progress of that additional feature should not be a measuring stick for the project as a whole. For js-ipfs, I see work being done frequently, e.g. the DHT work[1] and weekly syncs[2]. I agree work is slow, but I am not getting the impression it is fizzing out.

Having said that, I think there are fundamental problems with the ideas behind it. I want MaidSafe's ideas and IPFS's completion/MVP schedule. IPFS doesn't bring enough anonymity or forced, equitable, encrypted block sharing IMO (I am aware of efforts on both fronts of course).

0 - https://github.com/ipfs/notes/issues/64

1 - https://github.com/ipfs/js-ipfs/pull/856

2 - https://github.com/ipfs/js-ipfs/issues/1179

fiatjaf2• 8 years ago

Wait, what?

js-ipfs runs in the browser just fine. See https://filemap.xyz/.

kodablah2• 8 years ago

Only some features work in the browser. Notably DHT and relaying does not. See the repository's "Please read this" at https://github.com/ipfs/js-ipfs/#project-status

brodo• 8 years ago

I find that unlikely. They got 250 MUSD quite recently and they need IPFS working if they want filecoin to work.

pavlov4• 8 years ago

They already got 250M USD. Why do they need to get Filecoin to work?

The easiest and least stressful outcome is to drag it along forever while paying themselves giant executive salaries from that stash. As long as there’s some minimal progress, it’s not fraud.

jadedhacker• 8 years ago

It's a wonderful concept to decentralize the internet. However, I am a little skeptical of this concept. Won't IPFS be subject to the same problem as the global blockchain for BitCoin? That is, it can't process the kind of transaction throughput the global economy requires. Wouldn't that technical limitation be present for IPFS?

diggan7• 8 years ago

No, IPFS is not subject to the same problem as IPFS doesn't use or is a blockchain. There is no transactions.

There is data, and data is referred via their content hash. People can request that data via the content hash and then it's downloaded. There is no mining or similar things in IPFS itself.

marknadal• 8 years ago

IPFS is important because all devices need:

- Computation

- Storage

- Bandwidth

Ethereum tackles computation and Filecoin storage. Our system

IPFS and Ethereum are well recognized because they actually exist and work. Our system, which also exists, and is ranked #2 "Blockchain" by developers on GitHub ( https://github.com/topics/blockchain ), will be tackling the bandwidth incentive problem.

Juan and I met a while back (we all have been doing P2P stuff since before 2014) and I'm trying to convince him + us + ethereum team to partner together, so app developers have a single solution to build any dApp on top of, using real tech (not vaporware!) that already exists.

Financial transactions academically must solve the double-spending problem, which Bitcoin addresses, and you are right, requires a form of Strong Consistency (via the CAP Theorem) that bottlenecks global throughput.

However, storage (IPFS) and data sync/bandwidth (us) don't have those requirements, so they can be much more scalable.

hiccuphippo• 8 years ago

Is there any project like Wikipedia or Linux package repositories making use of IPFS already? Where can I find out who and how to help?

hex12648430• 8 years ago

There's a repo dedicated to the coordination of archival efforts.

https://github.com/ipfs/archives

IPFS is also used for the experimental gx package manager, which IPFS uses.

https://github.com/whyrusleeping/gx

dreit1• 8 years ago

lunyr is doing this I believe

diggan7• 8 years ago

First off, I'm a developer working on IPFS for Protocol Labs. Let me try to answer some inaccuraties and questions from this blogpost, to clear any confusion.

There is even the Beaker browser to help you surf IPFS

Beaker does not currently support IPFS, even though it used to in the beginning. We're hopeful Beaker might support IPFS once again in the future, but for now, Beaker just supports Dat.

How common are petabyte or even gigabyte files on the Internet? There is definitely an increase in size trend due to the popularity of the multimedia files. But when will this become a pressing issue? It is not a pressing issue right now because CDNs help a lot for reducing traffic for the Internet. Also bandwidth is relatively easy to add compared to latency improvements

Bypassing the fact that centralized vs decentralized are fundamentally different models with different reliability, argueing that CDNs help with reducing traffic is just wrong. CDNs just moves the traffic from "your server -> your client" to "your server -> someones CDN -> your client", it doesn't actually reduce traffic. However, using a decentralized protocol like IPFS would actually reduce traffic, as closests nodes can help serve content and maybe clients would not need to contact your own IPFS node at all, if the data already exists in a closer node (at your ISP or even neighbor)

For scalability, shard it, georeplicate it, and provide CDNs for reading. For fault-tolerance, slap Paxos on it, or use chain replication systems (where Paxos guards the chain configuration), or use the globe-spanning distributed datastores available today.

IPFS is actually a globe-spanning distributed datastore, so IPFS would be excellent choice if you're building a CDN. IPFS gives you the benefits you want for a CDN, scalable, content-addressed and used to passing around a lot of data.

Case in point, Dropbox is logically-centralized but is very highly available and fault-tolerant, while serving to millions of users. Facebook is able to serve billions of users.

For now. In the future, who knows? The argument for using IPFS is that we don't know if a central server will be around in the future, but to make sure, let's add content-addressing to data so we can serve this data from anywhere. Then it doesn't matter if Dropbox is around in the future, I can just pull down the data from anywhere and verify it's correct locally.

If you want to make the natural disaster tolerance argument to motivate the use of IPFS, good luck trying to use IPFS over landlines when power and ISPs are down, and good luck trying to form a multihop wireless ad hoc network over laptops using IPFS. Our only hope in a big natural disaster is cell towers and satellite communication.

Here the author is comparing a physical network with a software stack. Of course IPFS is not a antenna, so it'll be hard to use it as such. But, IPFS works excellent over radio and ad-hoc networks, because it's content-addressed. The current web? Not so well, as we have bunch of assumptions that the endpoints we're using are all serving us "good" content and that you have a server-client model. Ad-hoc networks are bandwidth-constrained, and everything we can do to reduce that bandwidth is worth doing. IPFS helps a lot with that since everything is content-addressed.

Does IPFS mean I may be storing some illegal content originated by other users?

No, IPFS only shares what you already accessed (until GC) or when you explicitly "pin" something (basically tell IPFS you want to help share this content with others)

How does IPFS deal with the volatility? Just closing laptops at night may cause unavailability under an unfortunate sequence of events. What is the appropriate number of replicas for a data to avoid this fate? Would we have to over-replicate to be conservative and provide availability?

This questions lead me to the belief that the author hasn't really understood IPFS before writing this post. Yes, shutting down your server will make content stored on that server unavailable. IPFS does not automatically distributed data. Just as your local HTTP server does not automatically distribute data to other servers.

But can the Byzantine nodes somehow collude to cause data loss in the system, making the originator think the data is replicated, but then deleting this data? What other things can go wrong?

Again, other nodes can't control your data. If you add data to your IPFS node, it's there until you remove it. No other nodes can control your data. Also, data is fetched based on hashes and when the content is downloaded, it's verified again to make sure it's correct. No way of screwing around with that.

Happy to answer any more questions people have about IPFS.

cocktailpeanuts6• 8 years ago

Beaker does not currently support IPFS, even though it used to in the beginning. We're hopeful Beaker might support IPFS once again in the future, but for now, Beaker just supports Dat.

Since you're a developer at protocol labs, I will assume you remember why beaker dropped IPFS in favor of DAT. I think IPFS is cool but the biggest hurdle for adoption is the naming scheme. More specifically what happens when a file content changes. You have IPNS to handle this but as far as I know it's virtually unusable because it's extremely slow. And I got the impression that you guys have no real solution for this, and maybe IPNS is not even one of your top priorities because you're focused on other important things.

But as a user this matters a lot because without a name resolution system, we're left with static files only, and while this may be a good alternative to S3, this means that's about it and people can't build apps on top of it. I wonder what your plans are and if you have any solution to building an actual functional IPNS, or if you're focused on building an S3 alternative at the moment.

diggan7• 8 years ago

Indeed, I do remember why Beaker dropped IPFS support. Basically, it comes down to the URL structure and how we structure them in the IPFS ecosystem, which is not super easy to integrate with browsers today. There is some logs about the discussion over at Github: https://github.com/beakerbrowser/beaker/issues/2

There is a open feature request for (re)adding IPFS support to Beaker as well, if someone feels a bit experimental with adding IPFS support to a Electron application: https://github.com/beakerbrowser/beaker/issues/480 Reach out to me if you feel interested in helping out and I can point you in the right direction. Email is in my profile.

Re: IPNS is currently slow: Yes, it is. Apologies for that, it definitely has to get faster to be really useful. The network has grown a lot and we are working on improving the performance of IPNS, both for adding the records and reading them. Currently the adding of records is slower than reading, as reading now uses pubsub to push updates and making it a lot faster than before.

We do have more solutions in mind for IPNS performance and still doing research on the best way of implementing them.

If you haven't seen IPFS pubsub before, I urge you to take a look, as it gives you the ability of doing communication with a large number of nodes cheaply.

So while IPFS started out as a distributed file system with naming on top, we now have more functionality for handling dynamic data and working on making it even easier to build applications with just IPFS.

book_mentioned2• 8 years ago

Why does IPFS get all the HN attention?

In my ignorance I'm not really aware of the landscape for these types of technologies; does anyone have time to share a brief summary or link to further info?

woodandsteel4• 8 years ago

The short answer is that a great many techy's are working on decentralization because the present centralized model has a lot of serious problems, like Facebook et al making their money selling ads that may have malware, and also selling your data to who-knows-who.

There are many projects to decentralize the web. IPFS gets so much attention because it is such a broad platform that other technologies (like Ethereum) can operate on top of, and because in some important ways it is farther along than anybody else.

ttul• 8 years ago

I think the author is overly critical of the limitations of a distributed file system. With a decent incentive system (FileCoin for instance) there is no reason that distributed file storage won’t scale really well.

I for one have a stack of SSDs waiting for FileCoin to arrive. I am keen to leverage the 10Gbps fiber in my office to start ghetto hosting for some $$.

woodandsteel4• 8 years ago

IPFS may be a way of sticking it to the man. But the invisible hand of the free market forces also help here; when one big corporation starts playing foul and upsets the users, new companies and startups quickly move in to disrupt the space and fill in the void.

No, once someone like Facebook gets a monopoly, it becomes impossible for a startup to get anywhere. It is hard for me to believe the author doesn't know this.

He also doesn't address the problem of web companies going out of business, like Geocities, so you lose all your data forever. Likewise the if-you're-not-the-customer-you-are-the-product problem that is behind so many ills today. As far as defeating governmental censorship goes, IPFS is being combined with Tor.

In general, he seems to be looking at things from the perspective of someone who works for a company based on the present centralization model, rather than the perspective of those who are suffering under it and could solve a lot of problems with IPFS.

brownbat• 8 years ago

Those are great questions.

Especially provocative point about how mobile devices would most likely be leeches in a p2p world. The world wants easy to use mobiles. A system like this may need to constrain leeches. There will be some tension serving both goals.

Paper review: IPFS – Content addressed, versioned, P2P file system

Related Stories