Blueprint of a distributed social network on IPFS, and its problems

https://beyermatthias.de/blog/2018/02/25/blueprint-of-a-distributed-social-network-on-ipfs---and-its-problems-2/

by diggan • 8 years ago

183 92 8 years ago

beyermatthias.de

discussedbefore• 8 years ago

One decentralized project is willing to list alternative options, many past the 'create a prototype' stage:

https://www.scuttlebutt.nz/faq/misc/related.html

ZeroMe – Decentralized Microblogging on ZeroNet https://news.ycombinator.com/item?id=15464156

An off-grid social network [Scuttlebutt] https://news.ycombinator.com/item?id=14050049

A recent discussion of building on IPFS, where a developer was honest about drawbacks:

OpenBazaar 2.0, powered by IPFS https://news.ycombinator.com/item?id=16702684

haven't solved the decentralized reputation problem and reviews can't entirely be trusted

I am interested in this space because first mover advantage is still kind of up in the air until something works well enough to build a network advantage.

thisacctforreal• 8 years ago

Any thoughts on Mastodon[1][2]?

I think "federated" is the term for its model, moreso than "decentralized" (?).

[1] https://joinmastodon.org [2] https://instances.social/list/old

sriku2• 8 years ago

That's the right word.

(General note)

Federated is a bit different from decentralised. For example, email is federated, but not decentralised. In a federated network, if "your service providing server" is down (like gmail.com), then you don't have service, though others on other parts of the network would continue to have service. With a decent sized network, there is no "your service providing server" and other machines can take over when one fails. Decentralisation always goes along with redundancy whereas federation doesn't require redundancy.

heynk• 8 years ago

Awesome post! I think a ton of people are coalescing on this idea - I’ve seen multiple people post thoughts around this very recently. I’m about to launch a distributed social network, built on ethereum and ipfs. Ethereum helps solve some of these distributed discovery issues. It includes an open source, centralized ‘gateway server’, which makes it a little easier for onboarding, plus it has built in support for ActivityPub out of the box, which makes it compatible with mastodon. I’m posting it here on Monday morning! Sorry for the self plug. I love your ideas around deleting content - that’s a tough problem that I’m only really supporting on the client side right now.

Alex39172• 8 years ago

Why not just pay a few bucks a month for a social network, and get all the benefits of centralization without the evils of ad-supported corporations? Most of us here use and trust GitHub, and imho that's what we need more of.

While I'm a huge supporter of blockchain projects and their allies, in the case of social networks I don't see how this is going to turn out any differently than FOAF.

throwaway127831• 8 years ago

Because not all of my friends and family are willing to pay a subscription fee :-/

evolve2k• 8 years ago

Would you pay to be a power user who can invite and manage say a dozen family & friends?

In this model it might be possible to assist uptake by having a class of power user who can manage their group over onto an ad free network.

Example: I have a private github account with repos I share with various friends who are not paying github users but are nonetheless now engaged in the github platform.

anonymous5133• 8 years ago

Doesn't matter. Our data is only secure when it is encrypted. The user should have absolute control over their data. If the user deletes their data then it should be permanently deleted.

ekianjo2• 8 years ago

how would you trust a centralized network NOT to use your data even if they make you pay for the service?

Alex39172• 8 years ago

That’s what the ToS and privacy policy are for.

olivernyc• 8 years ago

I made something similar to what's described in the article: https://static.network (https://github.com/olivernyc/static)

It's an IPFS based social network, running entirely in the browser. Posts are JSON objects of content and metadata, with links to the previous post and the author's profile. Currently updates are propagated over a single pubsub channel, so it only works for real time communication. I plan to replace this with IPNS once it is implemented in js-ipfs.

Long term, I want to implement a web of trust to facilitate search, discovery and spam control. Here's some good reading about how this might be done without central authorities: https://www.scuttlebutt.nz/stories/using-trust-in-open-netwo...

sneak• 8 years ago

There is a distributed (blockchain based) social network today: steemit.com. Several of the frontends (like busy.org) use IPFS for content distribution, as the Steem blockchain only holds text and json.

tuxxy• 8 years ago

This is great!

The privacy problem is especially interesting because I work on a distributed key management system called NuCypher. We use proxy re-encryption to do this and have a distributed network of nodes that perform the actual re-encryption on split keys.

If anyone is reading this article and wants to work on this problem, we have a split-key threshold PRE scheme called Umbral that we used to decentralize the key management part. The link is below:

https://github.com/nucypher/pyUmbral

fake-name3• 8 years ago

If you're thinking about building a distributed social network, you HAVE to have answers to the question: How do you moderate it? (Or how do the users moderate what they consume, if you will)

Basically, think of the most annoying person you can imagine, give them infinite free time, and a uncontrollable urge to do EVERYTHING they can think of to make your life miserable on said social network (doxxing, abusive messaging in every way possible, creating dozens or hundreds of accounts, etc...). If you do not have facilities to deal with this, your network is garbage.

With the focus on "distributed" and "resilient" and "unblockable", you're effectively creating the perfect environment for every abusive stalker, internet creep, and troll.

Lord of the flies is not a functional governance mechanism. Moderation and control have to be designed into the network from the outset.

There's no reason I can think of that some sort of "distributed control" isn't possible, but you'd better have it planned out first. Bolting on tools for handling abusive users after-the-fact will likely be much harder.

hodl• 8 years ago

Users mod their feeds as per RSS you just unsubscribe from idiots.

Its a bit like the internet itself. Lots of shoddy sites, but no one needs to visit them. Search spam is the exception but let a Google handle that.

fake-name3• 8 years ago

That would be fine, but a bunch of RSS feeds is in no way a social network.

A group of people who all subscribe to each other could be a minimalist network, but how would things like conversations work?

harlanji2• 8 years ago

I'm going old school, promoting RSS and offering an email address + call-in number with vmail and no comments on the page. I post video to CDN, do live streams with ~5sec latency (could be reduced I think). I post editorial style, sometimes have live chat for live streams via twitch. To self host I could just use an xmpp server with strophejs on the page, live streams might get a few thousand concurrent. Delivery works via HLS + CDN so scalability is in theory not an issue, although I've not tested multi-region viewership yet.

z3t4• 8 years ago

Each user can run their own software, either self hosted on a computer they own, or as a service. Kinda like a chat app, but with an open protocol. Then you can control who to follow and who can send you messages. And who can see your messages. Post group messages etc.

Gibbon1• 8 years ago

Moderation and control have to be designed into the network from the outset.

I wonder about the ability to set up privacy canaries to detect information leaks. Because with stalkers and the like what you are looking for is to make sure that they don't have side channel access to you. So it's not just being able to block them, but block people they associate with. And importantly have ways to detect when they've gotten shared info they aren't supposed to.

xorcist• 8 years ago

So, in your view, email and the web are examples of garbage networks?

Decentralized networks (such as the web) work differently from centralized network (such as cable TV and Facebook) and have different strategies for coping with misuse. You can not moderate a permissionless network. That's literally part of the meaning.

In a decentralized network, I follow you by subscribing to your data. Conversations usually follows that data naturally, just like a mailing list is centralized but email is not.

espadrine• 8 years ago

So, in your view, email and the web are examples of garbage networks?

Mostly so, but it is worth the trouble.

About 90% of emails I receive are spam. So, email is 90% garbage. The 10% left is the sinews of my life.

Fortunately, the industry found ways to cope, by contributing to mostly centralized DNSBL services such as spamhaus.

The same goes for the Web: centralized services like Google or (is there something else? ) filter out the irrelevant, dangerous, and fake (with varying degrees of success).

Moderation? Inevitable. Distributed moderation? Not so much.

The parent post is right: if you don't make it built-in, it will not be how you like.

dingaling• 8 years ago

Good examples. Plus Usenet too.

All of the above work by permitting the end-user to moderate to the extent he / she wishes, or not at all.

I had kill-lists for Usenet, I can DNS-blacklist websites I never want to see, I can grey-list or silently drop troublesome e-mailers. No-one makes the choice as to degree or method except me and yet it works.

ema• 8 years ago

Sometimes I do want to see the messages of people I did not subscribe to. For example when they reply to my message. In this case I need to have some filtering (read: moderation) system in place that ensures I do not get overwhelmed with spam, or gore[1] for that matter.

[1] of course the beauty of a distributed system is that those who want to see gore can not be denied this right.

fake-name3• 8 years ago

No, but they're not social networks.

kromem• 8 years ago

This is really more of a client issue than a server issue.

In a distributed scheme, people I "block" or even content I filter out should only happen in regards to me fetching content from the distributed network.

In particular, if the content pathways are P2P (which they should be, even if other measures in place such as relays or caching), inappropriate content ends up being reduced by a result of less peers propagating.

Once the right solution appears for the network, ideally there will be multiple clients working through the UX of using the network.

Personally the best I've seen in this space is Secure Scuttlebutt. I just wish they'd drop the write only aspect for the log, and switch pubs over to acting as relays.

acdha2• 8 years ago

Think a little more deviously: you block someone but your personal data, slanderous fakes, etc. is still out there. They can’t message you directly but can send it to everyone you interact with. They can setup new accounts faster than you can block them and those could either swamp a target or try to impersonate them so it’s hard for the target to have any unpoisoned interactions.

A client side solution can’t stop any of that. Anyone thinking about social software needs to look at what motivated assholes like 4chan have done and think about how they’d stop the next weev, GamerGate, etc. If you don’t have a good answer all you’re doing is building weapons for them.

douglaswlance2• 8 years ago

I believe we should look to the human brain to discover how it filters out noise; then try to replicate in our human-scale networks.

naasking• 8 years ago

then try to replicate in our human-scale networks.

Human scale networks are small. Our brains can only handle a few hundred people in our social circle, maybe up to 1,000.

amelius5• 8 years ago

Basically, think of the most annoying person you can imagine

I'd change that into companies or even governments that actively try to subvert your system.

teleclimber3• 8 years ago

You need to defend against both. Heck, if your social network ever becomes popular you'll need to deal with:

- fake instances / relays meant to vacuum personal data for sale and profit - identity thieves, both the broad-and-shallow, and the target-a-specific person kind - governments who got a taste of being a surveillance state thanks to FB and now can't kick the habit - massive numbers of people who decide to mess with someone at once (like when Twitter or 4ch put a bullseye on some person, whether it's deserved or not) - bots. bots bots bots. - maybe your client is super tight and solid, but what if company X decides to fork your client and lures people into using it? Now they have access to each user's stuff, and all their friends? Nice.

Obviously you can't realistically defend agains all of that. But it's worth thinking how best to mitigate before launching.

amelius5• 8 years ago

Moderation and control have to be designed into the network from the outset.

Perhaps even in a way that allow multiple regimes. For example, one user/group/admin could prefer one style of moderation, while another would prefer another style.

I think a new social network should leave as much "open" as possible, so that not just the software is federated but the ideas on which it is based are also federated in a sense.

harlanji2• 8 years ago

In the context of DTube IPFS is simply storage for video files, where the metadata + interactions are on the SteemIt block chain. It's cool that the author has generalized the concept onto pure IPFS. It may work better than my experience, which used IPFS to store very large contiguous mp4 video files (eg. 15min 720p max).

IPFS is nifty but I'm not convinced of its ability to support an intergalactic workload quite yet. I've looked at it vs. CDN for a video delivery platform and have a much more robust system built on primitives DNS -> CDN -> HTTP -> HLS segmented MP4 + static HTML site w/ SPA chunks in just a few weeks. I explored similar ideas with using encoded public keys, hashing, and side-channel crypto key distribution to support some interesting workloads. This is cool because it's a no-voodoo solution, just HTML and RSS and JS and chunked video files, and the CDN handles the details invisibly. With IPFS on DTube I've only been able to author a little data before running into queuing blocks where I was unable to continue adding content.

I have some videos on my YouTube channel talking about IPFS and giving demos in the past 1-2mo, 'iSpooge Daily'.

jancsika3• 8 years ago

How do I find Waldo?

On Facebook, LinkedIn, MySpace, Twitter, Google, Github, GnuCentral, etc., I type in Waldo's name and instantly discover Waldo's profile. On Facebook it even shows me Waldo's profile before I consider discovering him.

Also, how does the system make it likely that when I search for Waldo I get Waldo and not his evil twin Odlaw?

Also, how do you incentivize adoption so that the community doesn't end up rationalizing "Error 404: Waldo not found" as the proud badge of an artisanal social network?

RSSify2• 8 years ago

How do you find people's email addresses?

alexandercrohde5• 8 years ago

Exactly. I think there are two types of social network users

1. People who want to consensually maintain two-way contact with people they talk to in normal life.

2. People who want to anonymously get life details of classmates they have a crush on, ex's, etc

Building a network to support #1 while maintaining true anonymity and privacy is much more doable. Not being in group #2, I don't worry about it.

jancsika3• 8 years ago

I'm confused-- I thought all these HN posts about federation and distribution were appearing because Facebook looked like it could be crumbling.

Facebook got its start for one reason and one reason only-- you could use it to find Waldo.

Are you saying that global discoverability is not a valuable feature of a usable social network?

Eric_WVGG2• 8 years ago

Badly. Most non-techies I talk to would rather “find you on Facebook.”

This was largely behind the relative failure of RSS, in my opinion, even more than the Google Reader debacle. Various friends and family still ask me how I keep informed, I show them Reeder for iPad and they immediately want to get on board; then I explain how you go around to the news sources you like and gather feeds or use a bookmarklet and their eyes glaze over.

I hope this brainstorming on a distributed/federated social network goes somewhere, but if the creators don’t even bother considering why Facebook worked in the first place, their efforts are doomed.

rolleiflex2• 8 years ago

It seems there's a lot of people working on this area, just looking at this thread. I did something similar to this in 2013 and I got to around 20k users: https://www.theverge.com/2013/11/27/5150758/aether-aims-to-b...

Two things that need to be solved:

a) The structure of moderation. No moderation does not work. Been there, done that, it gets very ugly very fast. But what can be done is to make moderation more visible, and allow people to choose who they want to be moderated by.

b) Scaling. 1-100 users are fun, 100-1000 are exciting, 1000-10000 are a little nerve racking, and after 15k users, you notice the reality of how nodes and networks behave has diverged from your 'reasonable' assumptions. Around 20000 or so, I stopped maintaining it. Scaling distributed networks is much different than scaling regular networks where you can add more hardware and servers and sharding, it requires new behaviours from each node as the system grows. The software that works at the level of a million users in a decentralised system will be much different than the one that works with a billion users.

Funny enough, I've recently quit my day job and I'm focusing on the second version of it, so the memories from 2013 are pretty fresh.

wjnc• 8 years ago

Robert Nozicks' "Anarchy, State and Utopia" (1974) describes how a system of multiple concurring "governments" could work. Quite a stretch from social networks but first thing that came to my mind when reading your comment. Moderation is a form of social network government, right? One could subscribe to a government / moderation scheme and the governments form meta-governments to decide which content from competing governments are fit for their users. Moderation in two (or N) levels.

rolleiflex2• 8 years ago

Interesting! I'm actually doing something pretty close to what he is proposing with v2. Meta-'governments' layered on top of each other.

tscs372• 8 years ago

That's somewhat what federation solves. You pick out an instance which represents a particular way of moderating. You can still listen in on (most) of the other network.

If you don't like the moderation/government of your instance, you pick another.

ianopolous• 8 years ago

People might be interested in the social network on IPFS a few friends and I are working on in our spare time, Peergos [1] [2]. It also happens to be the first social network on IPFS (We started before IPFS itself existed).

Everything is encrypted, including metadata and we try not to expose your follow list to the network. It's inherently multi-device, independent of DNS and even has streaming E2E encrypted videos playable in the browser (streaming is disabled by default at the moment because it only works in Chrome) and even hides file metadata like size, name and directory structure. It's still pre-alpha though.

[1] https://peergos.github.io/book

[2] https://github.com/Peergos/Peergos

hateful• 8 years ago

I've been keeping an eye on peer-to-peer/federated social networks for a while. And though I left Facebook a couple of months ago, none of these can replace it until they do one simple thing: Allow one user to post to another users "feed". This is what Google+ was missing when it came out. This is what Twitter is missing. This is the reason why Facebook is so popular, IMO.

I see something on the Internets which I think my wife would like, but I also want her/our friends to see it and I post it on her wall/timeline. I don't want to post it on my feed, that won't help, and I don't want to private message it to her, that also won't help.

Any of these social networks add this feature - then I will champion them to the end of the Internet.

And perhaps add support for events. But that can be an adjacent but compatible application.

da02• 8 years ago

It sounds like a "profile"/feed also acts like a message board? (I never had a fb account, so pardon the stupid question.)

Asdfbla• 8 years ago

While I like the distributed approach technically, it solves none of the social problems that social networks bring.

Filter bubbles, propaganda and competition for users' attention with clickbait and other cheap stories might even become worse, since a centralized platform could at least tweak their recommendations algorithms to break the bubbles (which Facebook doesn't seem to do, but it could).

Basically, I'm pessimistic because a distributed network only solves the (comparatively) easy technical challenge without addressing the problems of social networks.

stareatgoats3• 8 years ago

I believe the main problems of "the social network" (FB) can't be solved in this way, it focuses almost entirely on the privacy and "freedom from surveillance" aspect, to the detriment of other vital aspects (like non-anonymity).

What we seem to need is a solution that does what Facebook does right, which corresponds to the networking needs of billions of people, combined with a transparent and ethical content policy approach. Plus: that doesn't leak data at an individual level (except after a court order), to name some of the requirements that seem most important.

On the other hand, a non-corruptible utility-type service like this might be too much to hope for as well ... ?

orbifold2• 8 years ago

In principle every problem solved by facebook in a centralised fashion has already been solved by phone companies in a decentralised fashion. You can buy one or several numbers, which are quasi anonymous and distribute them to anyone you care to share them with. Noone can easily find you unless you give them that number, it easily possible to separate between work and private numbers and you get a new number if you change your job. Calling by phone is nearly universal as is access to SMS, it already connects all people across the globe without any need to sign up to a centralised service. The only advantage services like WhatsApp and Facebook have is that it is really cheap to communicate over them, whereas the only way of operating a traditional phone service is to charge money.

I think what will eventually happen is that the traditional telco providers will gradually improve their service quality, prices and infrastructure and eliminate the need for parasitic communication platforms like WhatsApp and Telegram by introducing standardised and provider agnostic solutions. For example why not extend the protocol that is used to route phone calls to a protocol that allows to route arbitrary packet or streaming data transfers.

stareatgoats3• 8 years ago

"Noone can easily find you unless you give them that number"

Yeah well, finding anyone is not a flaw; it's a one of Facebooks killer features, without which it would not have become the de facto global registry of single individuals.

"The only advantage services like WhatsApp and Facebook have that it is really cheap"

I believe this is a gross misunderstanding of the utility of social networks compared to older systems like telephony. But granted, the brave new world comes with a number of problems as well.

throwaway736369• 8 years ago

I think what will eventually happen is that the traditional telco providers will gradually improve their service quality, prices and infrastructure

That's going to take a long time. Maybe 50 years? Working in the telecom business has taught me that federation has pretty ugly failure modes. The result of the structure of the telecom business has been the SIP "standard" and the horrible abuses of SS7 before that. The horrible reality of telecom is that most work is integration work. New features or similar is a secondary priority. Did you know that telecom providers actually test every single new phone on their network and add workarounds? It's the Tower of Babylon, except made from technical debt and divergence from expected behavior.

I've been thinking about this a lot, and I've arrived at an explanation: Each telecom provider is large enough to do their own development, switching costs are high, and vast majority of traffic is internally in a country. This situation incentivices creating new features and services that only need to work internally in a country or even just in a single network on a low budget using the path of least resistance (read:abuse of standards, bad solutions, technical debt). Most advanced functionality break down at country borders. For advanced functionality it's common to only support a small set of clients.

I'd suggest not waiting for Telco providers. They're going to become dumb pipes because of their incentives. Centralization works extremely well. Any decentralized network needs a lot of thought put into how it will avoid the traps of the telco industry.

jeswin• 8 years ago

Some of the points made in the "why ipfs" are inaccurate wrt scuttlebutt. Secure Scuttlebutt is a signed immutable log/database of a person's posts, replicated with peers. It is also distributed, instead of federated. (Add:) And because SSB is a database, you could build pretty much any app on top of it.

The SSB network is very active, take a look - https://www.scuttlebutt.nz/

amelius5• 8 years ago

Interesting that we're now posting blueprints instead of code. I think that's a good development!

Now if only we could have the requirements posted and discussed first :)

amelius5• 8 years ago

Why aren't governments issuing grants to universities to research the development of a new social network?

zzzcpan• 8 years ago

Isn't social network a purely commercial idea born to manipulate people, get them obsessed over it and exploit them? I mean it's not an actual problem to solve that can be worth any research.

jsilence3• 8 years ago

no sure whether you're cynical

syllable_studio• 8 years ago

I just started dabbling with IPFS. It has inspired me to revisit a project prototype that I abandoned for a year.

A year ago, I started making this prototype: https://www.dearnation.org/ "Dear Nation is a living archive of the letters we write to our nation's leaders." I've been torn about pursuing this concept. I think I'm particularly scared of how messy it is to moderate and foster democracy online. I'm hoping that by working with tools like IPFS it will inspire me to give more free time to ideas like this.

teleclimber3• 8 years ago

I'm kind of baffled that this article did not start with a statement of the problem, and doesn't even consider other technologies. Instead it goes straight to IPFS as the technological answer without bothering to define the kind of social network they are trying to build.

A "social network" can mean very different things to different people. Even among FB users you find a very wide range of primary use cases.

So if you're going to propose a new social network at least start by defining the desired characteristics of your network, and even more importantly in this day and age, talk about the kinds of threats you are going to defend against and those you won't.

Then you can explore possible technologies to see what fits best. And if you did that you might come across the DAT protocol[0] and find it's better suited than IPFS.

Characteristics I'd like to see discussed before launching into a solution:

- Is this meant for users to connect to wide audiences (like Twitter, Reddit), or is it meant for me to keep up with my known friends/acquaintances privately?

- what features are you going for? Groups? Events? Chat? Just basic timeline?

- What's the visibility model? If a friend replies to your post, does that mean friends of that friend see your post? [there are hundreds of questions similar to this one. laying out some general principles before starting on the solution seems like a good idea]

- is end-to-end encryption supported? How do you limit the distribution of private information when many of these decentralized protocols assume that knowing an unguessable url is enough to get the contents at the url.

Anyways sorry if this comes off as grumpy.

[0] https://datproject.org/

pjkundert• 8 years ago

The problem is, moderation is actually a poor solution to the real problem - equal visibility of ignorant/vile and insightful comments.

https://pol.is/home presents a more viable option. But people self-grouping (eg. vile people only seeing other vile people) could also create echo-chambers.

For example, as a libertarian I’d love to see comments by anyone with “skin in the game” - regardless of political bent. How could we achieve that?

Blueprint of a distributed social network on IPFS, and its problems

Related Stories