Design and evaluation of IPFS: a storage layer for the decentralized web

https://dl.acm.org/doi/abs/10.1145/3544216.3544232

by dennis-tra • 4 years ago

193 68 4 years ago

dl.acm.org

noasaservice4• 4 years ago

Lets remind everyone:

1. IPFS attaches ALL network interfaces (internal and external) to your identity.

2. Tor is still "experimental" done by 3rd parties. https://flyingzumwalt.gitbooks.io/decentralized-web-primer/c...

3. Due to 1 and 2, any hosted content is EASILY trackable to a user's computer, even behind NATs. A machine cryptokey also helps cement that (but can be changed). This allows easy DDoS'ing of any and all endpoints hosting content you don't like.

4. It is trivial to ask the dHT for *who* has a certain content key, and get all (or the top 50?) computers hosting that content. (this matters with regards to "sensitive" content)

5. Running a node is still high cpu, ram, and network chattiness - so using a VPS to keep IPFS off your local network is still tenuous to run.

forgotmypw17• 4 years ago

For not "sensitive" content, but just as an alternative to other hosting methods, would you say that these shortcomings fall under "do things that don't scale"?

blamestross• 4 years ago

Yes on all counts.

4) Just keep watching the dht entry. It isn't "Top" it is "most recent". If you really are in a rush to know, just join the dht with that entry as your node's hash (or one close to it) and they will even advertise themselves to you.

chriswarbo4• 4 years ago

These points also apply to HTTP, except (5). In fact, resource usage is my biggest problem with IPFS at the moment; and the reason I turned off my node :(

noasaservice4• 4 years ago

Yeah, you really can't get away from the chattiness of IPFS because that's just how distributed hash tables work.

But the CPU and RAM costs... well, those are frankly exorbitant to what amounts to a glorified Apache web server bolted on with a Bittorrent tracker.

hecturchi3• 4 years ago

It is relatively new so not many people know about it, but now libp2p (and Kubu/go-ipfs) can be configured with a Resource Manager that effectively can keep a lot of the resource usage at bay: https://github.com/ipfs/kubo/blob/master/docs/config.md#swar...

pclmulqdq3• 4 years ago

A lot of Web 3 stuff is about federating infrastructure in ways that still allow centralization of control. IPFS is just another one of those protocols. One other thing that turned me off about IPFS/Filecoin was the huge amount of compute power needed to run a node.

hosh2• 4 years ago

IPFS was around before people started talking about “web3”.

I see it as part of increasing resiliency and data locality. A key characteristic is that the cost for data storage is shifted to the people who care about preserving that data.

So a good example is distributing Linux distro packages.

Apparently, cpu/mem can be tuned.

Hamuko• 4 years ago

I've been under the impression that IPFS has way lighter computational requirements than Filecoin, which is basically impossible to run without a server.

hecturchi3• 4 years ago

I think the point is not to force decentralization, but to allow it. When the protocol is open, there are more chances to break the monopolies through innovation, quality of service etc. and avoid lock-in.

In this case, IPFS breaks the monopoly of trust (only obtaining content from a single source because that source is the reputable source for it). Content addressed data makes the source irrelevant. Many possibilities open from there (i.e. breaking the monopoly of storage).

rhodorhoades• 4 years ago

Interesting. Can you explain how that’s different than the normal internet/ hosting solutions currently available?

moomoo11• 4 years ago

You get way more VC washing I mean funding.

chriswarbo4• 4 years ago

The reasons I like IPFS:

- The address of a file has no relation to the machine hosting it (it's just a hash of the contents). Unlike HTTP and DNS, which rely on machine-specific IP addresses

- I can host files on multiple unreliable machines (e.g. a powerful desktop, a mostly-always-on raspberry pi and a couple of frequently off/disconnected laptops). As long as at least one machine's online, the files are still available. Doing this for HTTP requires some sort of load-balancer (a single point of failure), plus mechanisms for discovery, authentication, etc.

- Since IPFS uses a content hash, all copies of a file will get served from the same address; hence anyone with a copy of a file can bring its address back to life. With HTTP, people can host their own copies of files that get taken down or disappear; but their copy's address will be different from the original, and hence all hyperlinks to the original will remain broken. (Note that it's not perfect, since IPFS can perform hashing in a few ways, e.g. to better support streaming; and that results in different addresses)

This is really useful for stuff like git repos, software packages, etc.

zcw1002• 4 years ago

1. IPFS attaches ALL network interfaces (internal and external) to your identity.

I don't know what you mean by attaching a network interface to your identity or even just identity. IPFS identifies a node by PeerID which is mostly a public key, that's it.

2. Tor is still "experimental" done by 3rd parties. https://flyingzumwalt.gitbooks.io/decentralized-web-primer/c...

Tor has nothing to do with IPFS. There have been some people that have worked on some integrations to use them together.

3. Due to 1 and 2, any hosted content is EASILY trackable to a user's computer,even behind NATs. A machine cryptokey also helps cement that (but can be changed). This allows easy DDoS'ing of any and all endpoints hosting content you don't like.

Yes, it is easily trackable. There is nothing about the design or goals of IPFS to be anonymous. In some ways the entire point is to be found and announce that you have content available. I don't know why people associate decentralized with anonymous. Not sure how you think it's cemented. Just toss the key and make a new one. How does this make DDoS'ing endpoints any easier than anything else hosted on the internet let alone DDoS'ing all of it. In face being distributed and content addressed can mitigate DDoS if it's malicious or just a hug of death.

4. It is trivial to ask the dHT for who has a certain content key, and get all (or the top 50?) computers hosting that content. (this matters with regards to "sensitive" content)

I hope so, that's the way IPFS works. Ask for who has what you're looking for and retrieve it. I have no idea what you mean by "sensitive" content. If it's sensitive you can encrypt it.

5. Running a node is still high cpu, ram, and network chattiness - so using a VPS to keep IPFS off your local network is still tenuous to run.

There are configurations to mitigate this but just making a blanket statement that it's high cpu, ram, and network is just FUD.

dannyobrien2• 4 years ago

(Disclosure: I work for the Filecoin Foundation/Filecoin Foundation for the Decentralized Web).

I do actually agree that the privacy and anonymity aspects of IPFS are not well- conveyed. I think people get hooked on the "censorship-resistant" nature of decentralized systems, without understanding that even if you have multiple sources, for instance in a content-addressable network like IPFS, aggressive censorship systems have other strategies to dissuade dissemination or punish readers. You always have to be thinking a few steps ahead. Services like Tor and, I hope, the IPFS network both try to convey what threat models they are useful for, and which they are not, but it's really hard to stop overenthusiastic re-statements that give them super-powers they do not, in fact, possess.

That said, there's a bunch of careful thinking right now going on about how IPFS's privacy story could be improved: https://blog.ipfs.tech/ipfs-ping-2022-recap/ has a couple of sessions on this, and is a great summary of some other recent developments in the space.

One of those improvements is in the point about nodes being high CPU, RAM, etc. (I actually find this to be more of a challenge when running the full IPFS Go node locally on my desktop, rather than on a VPS; it requires some tweaking.)

The strategy right now is to encourage more implementations of IPFS to cover more use-cases; the original go-ipfs had to do everything, including maintaining some legacy decisions. Nowadays, there's a lot of effort on alternative IPFS implementations that can be slimmer, or optimised for particular scenarios, e.g. on an embedded device, serving a high-load web gateway, or providing millions of files. Protocol Labs recently renamed their canonical go-ipfs to kubo (https://github.com/ipfs/kubo ) to make it more of a peer with other implementations.

Of course, I love all these new generation implementations EQUALLY, but if you pushed me, I've enjoyed playing around with https://github.com/n0-computer/iroh , a modular rust implementation building off the increasingly robust rust libp2p etc libraries. There's some more to pick from here: https://docs.ipfs.tech/basics/ipfs-implementations/

colinsane• 4 years ago

> 5. Running a node is still high cpu, ram, and network chattiness - so using a VPS to keep IPFS off your local network is still tenuous to run.
There are configurations to mitigate this but just making a blanket statement that it's high cpu, ram, and network is just FUD.

it’s not FUD. this is the out-of-the-box experience for many users, and has been basically since its inception. i know enough about IPFS to import, retrieve, and pin content, convert between different CID formats, use the IPNS and dnslink stuff and host a gateway, configure the set of resources that gateway will serve, and so on. what i still don’t know how to do after being an IPFS user since 2015 is how to make it not noticeably bog down the typical low-end/SBC home servers lots of people in this space use for all their other p2p stuff.

it’s not FUD: perf/resource use is a real problem that is legitimately hampering adoption. try running just basic IPFS workflows on the latest gen of raspberry pi before labeling perf complaints as FUD. if you’re close to the dev team and somehow don’t understand that this isn’t FUD then setup a conference call with me and i’ll demo perf of a typical home setup for you.

px433• 4 years ago

Yeah, doing crime on IPFS is a really bad idea. I talked with Juan Benet about this in maybe 2016 or so, and he said it's a feature, which is fair enough.

Privacy is very much not the goal of IPFS, and never has been. Not only is it obvious who is hosting any given file, but if you listen actively you can usually see who is downloading any given file as well.

7v3x3n3sem9vv• 4 years ago

sounds like a potential privacy disaster of Google Facebook and others can just listen in on erroneously assumed privacy. I suppose the only reason Google/Facebook aren't pushing for it is it allows anyone to collect this data vs just the big players.

hecturchi3• 4 years ago

1) To be fair, private IP ranges should no longer be published to the public DHT (there is a LAN DHT for them).

And of course what interfaces are announced is and has been configurable for as long as I remember ("NoAnnounce" setting in the config).

eminence32• 4 years ago

3 of the authors work for Protocol Labs (the company that develops IPFS), which is likely why the paper is able to analyze data from the ipfs.io gateways

purim• 4 years ago

IPFS is just too slow for it to be usable on a mass scale. It's a neat idea but unfortunately, p2p file storage is tough, you absolutely need a central model to scale up. Offering coins to cash out at casinos where the other side of the order book are people with unlimited supply of it. Doesn't work.

VonGuard• 4 years ago

I am told this is very specifically a default setting in the app that makes the minimum connections something like 100 or so. If you crank that down to 5 to 20 it goes much faster. YMMV.

matrix_overload• 4 years ago

Yeah, but Filecoin raised $250M two years ago. They have enough capital to keep writing these articles until the thermodynamic death of the Universe.

simonebrunozzi• 4 years ago

In 2017, not in 2020.

yamtaddle• 4 years ago

The fundamental problem with any of these p2p-hosting solutions is that most devices folks use these days are battery-powered. Any solution that requires a persistent connection or intermittent network wake-ups to achieve even semi-decent performance won't make it, and if you overcome that hurdle you still won't be able to count on most of the devices downloading content to also serve content. No-one's gonna sacrifice battery life to make the distributed Web dream happen.

They seem a lot more useful for internal infra of hosting providers, though for IPFS in particular I expect the performance isn't consistent enough to be a suitable solution for most of those use cases.

tinalumfoil• 4 years ago

Having used ipfs casually the big draw for me has been the gateways. Running a full ipfs node on client devices is mostly not practical, but I can choose a trusted (or not so trusted since I can check the hash) gateway and that can handle the heavy lifting for me. And even though the ipfs network is slow, the distributed nature lends itself to very heavy caching, so the gateway doesn't have to be any slower than if it was serving me its own static files.

Plus the gateways provide compatibility to www.

Given, things are far from ideal now. But looking over the water it's good to see Mastodon taking off and I think that's largely because you have the option of just choosing a single trusted provider ("instances") from which you can access the rest of the network. The trusted provider does the heavy lifting for you.

zozbot234• 4 years ago

IPFS is built to work reasonably well with ephemeral connections. Sure, not everyone is interested in serving content but for those who are it seems to be a reasonable choice.

hinkley• 4 years ago

When I worked for a National Supercomputing Center, I discovered that they had a distributed filesystem called Andrew Filesystem (AFS) which they used for archival purposes. With every new generation of distributed filesystem I always wondered why the older ones failed.

It wasn't until maybe 10 years ago that I finally got my answer: It turns out that Amdahl's Law kills AFS. There's a total throughput wall that becomes very painfully visible once you move to gigabit networking, and any one client can pretty much saturate the network.

pjmlp• 4 years ago

That is also what CERN was using in early 2000 before the Grid efforts.

rakoo• 4 years ago

Bittorrent has clearly shown that p2p can perfectly scale with very little centralization, that is now becoming unneeded. There is an issue with IPFS, not with p2p.

My uneducated guess: content in IPFS is split in too small blocks, making the data-to-control ratio way too low.

jhunter1016• 4 years ago

Disclaimer, I am head of product for Pinata

I think of IPFS as being an open data platform first. You can connect to it and disconnect as frequently as you like. The underlying p2p capabilities don't have to necessarily be fast. It just has to do the job it was designed to do—get content in a permissionless fashion.

Speed, convenience, reliability, and more are not the problems protocol need to solve for. Providers can solve for these problems without infringing on your ability to "take your ball and go home." Take Pinata for example. We provide dedicated IPFS gateways that provide essentially the same experience you would expect from traditional cloud providers. But if you ever want to leave Pinata or back your data up or just inspect your data, you don't need Pinata's permission. IPFS media is public and open. Convenience is a layer on top of that.

IPFS also doesn't need or have tokens. Filecoin is a separate entity. IPFS is especially powerful because it is not linked to a specific blockchain or currency.

dmitriid6• 4 years ago

Speed, convenience, reliability, and more are not the problems protocol need to solve for. Providers can solve for these problems

They can't if the protocol doesn't allow it.

Otherwise it will be regular centralized storage providers with IPFS bolted on top for one or two geeks who care about it.

xani_2• 4 years ago

You don't need central central, but the more you distribute the more you lose on performance.

It could be interesting idea on smaller scale, say you start a "virtual hosting provider", where each of 10, 100, 1000 people connects into a mesh and store eachother's data so in event of failure it just keeps working.

dartharva• 4 years ago

I only know IPFS from it being one of the enablers in downloading from Library Genesis, and man am I glad that it works so good for them..

uncomputation• 4 years ago

Perhaps but Library Genesis worked long before it started using IPFS as well.

unmole• 4 years ago

Is IPFS usable without the Cloudflare bridge?

px433• 4 years ago

Yeah, there are something like four major public IPFS gateways, one run by Cloudflare, one run by Protocol Labs, or you can just run your own node, which is something that many thousands of people are doing.

sixtyfourbits• 4 years ago

It actually doesn't work so well, unfortunately.

Picture this: 3 million books, at least one CID each (in practice it's often multiple, since the libgen collection uses a chunk size of 256kb). Section 3.1 of the paper talks about content publication - for each CID, a provider record is published on up to 20 different peers. Because the CIDs are derived from a high-quality hash function, they are evenly distributed. So this means that a node with a sufficient number of items ends up connecting to every single node on the network. For 3 million CIDs * 20 publication records, this means sending out 60 million publication records, every 12 hours, i.e. an average of 1388 publication records per second (assuming one CID per file, which is conservative). This is just to announce to the network "hi... just wanted to let you know I still have the same content I did yesterday". And every full replica of libgen is doing this.

Another major flaw derives from the way bitswap works as discussed in Section 3.2, which states "before entering the DHT lookup, the requesting peer asks all peers it is already connected to for the desired CID". I'm not sure if that actually means all the machines the node has any type of connection to, or only those connections over which it is using bitswap. Even so, asking every peer you're connected to, even if it's only the subset for which you have a bitswap connection established, is inefficient.

Compare this to bittorrent:

- First there is a much coarser level of granularity; in the libgen case the torrents contain 1,000 books each, so there's much fewer announcements to the bittorrent DHT and trackers. Of course the tradeoff with this is that you can't look up the identifier of an individual book. Instead if you know the magnet link (which can be constructed from the torrent hash) and the name of the file within that torrent.

- Secondly, a node hosting a large number of torrents (and a large number of active connections) will only send out want lists to the peers that it knows are also hosting that torrent. Peers also exchange have lists, and I think one or both can be represented as a bitfield for efficiency (rather than a list of CIDs/hashes). With bitswap you can end up asking every connected peer, just in case they have it.

On a practical note, hosting the torrents is quite practical with adequate hardware, but even with fairly powerful machines IPFS (at least in it's current main implementation, go-ipfs aka Kubo) really struggles and can bring a machine to its knees, even when hosting only a portion of the full collection. In terms of scalability, bittorrent and IPFS are in completely different leagues. Scalability is the main reason the archive of papers from sci-hub (over 80 million) isn't available via IPFS yet, because it's just not going to be able to handle that at all in its current state.

Having said all this, I should state my knowledge of the protocol and go-ipfs is incomplete, as I've only used it but not done any development work on it or dived deeply into the code. I'm happy to be corrected if I've misunderstood anything mentioned above. Also, bittorrent has more than 20 years of implementation experience and i'm sure with further work IPFS can be made to scale better. I don't have the answers as to how to achieve the granularity you get with IPFS vs bittorrent (which is a major point of difference, and something that sets IPFS apart in a significant way). But it's something that definitely has to be fixed to be truly capable of achieving it's stated goals.

b_fiive• 4 years ago

_Disclaimer: I'm the founder of a project behind a new IPFS implementation_

I have a fair amount of experience with the kubo (go-IPFS) codebase, and can confirm the broad strokes of what you've posted here, including the part where bittorrent is straight-up better at scaling, both in terms of protocol design choices & having robust implementations.

The chattiness of the protocol is a very real problem. It used to be _much_ worse. Further order-of-magnitude drops will require rethinking numerous aspects of the protocol. The implied-start topology of the network needs more thought. What remains to be understood is if those pile of changes can bring IPFS into the same league as bittorrent in terms of network efficiency, while also having the "single-swarm property" that provides fine-grained content routing.

A bunch of us are committed to building this. Hopefuly a HN post in a few years time will point to this one as a reference for just how far we've come

xani_2• 4 years ago

Bittorent's main advantage (aside from not hosting every single chunk a file as separate entity) is that with tracker it effectively creates its own small network of hosts sharing a given torrent, and even in case of DHT you kinda just use it to find other peers for the one big torrent and that's about it.

Even in massive torrents with thousands of peers you just pick a bunch to talk with and that's it.

And publishing one was same thing essentially. Just that each peer's peer network will mesh enough to propagate data quickly.

Also, bittorrent has more than 20 years of implementation experience and i'm sure with further work IPFS can be made to scale better.

It honestly looks like fundamental design issue. Bittorrent was blazing fast from the beginning and it only got better.

truth_seeker• 4 years ago

The content retrieval process across all regions takes 2.90 s, 4.34 s, and 4.74 s in the 50th, 90th, and 95th percentiles

Good improvement over the years but still a long way to go feel it even soft real time. Not sure these servers are using on the fly gzip compression before sending over network but they should consider adding compression feature at file or block level natively in "ipfs add" command.

There was an interesting paper "Hadoop on IPFS" (around year 2016-17). I hope these continuous improvement will play good role in making big data and analytics decentralised before it hits v1.0

https://s3-ap-southeast-2.amazonaws.com/scott-brisbane-thesi...

IanCal• 4 years ago

I'd like to use it for analytics. One issue I hit was when I was testing things out for that use I added a file to several pinning services and the result wasn't available via my node or the ipfs main gateway for hours.

morsch• 4 years ago

I keep confusing IPFS with the (much older) I2P. Just in case someone else makes that mistake.

wespiser_2018• 4 years ago

I'd like to image a world with globally addressable devices, maybe that's through NAT hole punching or just IPv6 where we can share everything, but the way things are now, I really struggle to determine what's the legitimate use case for this technology if we ignore the interest in decentralization for it's own sake and just consider engineering tradeoffs like cost for performance and end user experience/ergonomics.

As a storage layer, there are major challenges to adoption IPFS that have persisted almost a decade in to the project. At this level of partition comes at an incredible cost to availability, and from everything I read, the best practices for hosting user generating content still involve paying a service to "pin" your content to ensure it doesn't get dropped, so you still pay someone to host your data!

So what I'd like to know, is why would I want to use IPFS to host anything, when better, more performant and cost effective alternatives exist, and IPFS doesn't guarantee a file is actually hosted? Like, are there words you can say to your boss to argue for IPFS as a rational choice in systems architecture? What is the use case here?

Design and evaluation of IPFS: a storage layer for the decentralized web

Related Stories