The unrealized potential of federation

https://drewdevault.com/2020/09/20/The-potential-of-federation.html

by als0 • 6 years ago

194 197 6 years ago

zaksingh• 6 years ago

The biggest challenge I see with federated systems is the discovery process for finding a node you want to be associated with. This is a hard choice - will this server still be running in 6 months? How is it funded? Is it being actively maintained?

These questions are too much for laypeople, and I can't imagine a scenario in which your friendly local system admin is your service provider for a critical service in your life like Facebook. Could you imagine email working that way?

I see federated systems naturally ending in centralization in the form of a couple big megacorps that the average person trusts to not shut down tomorrow. Convenience and ease of use will win out. It happened with email, it arguably happened with xmpp back in the day. Change my view?

ddevault3• 6 years ago

There's no evidence of this happening with, say, Mastodon. The culture is different, and isn't changing. If we know what we want and what we don't want from the start, we can keep centralization from gaining a foothold.

kixiQu3• 6 years ago

You can migrate a Mastodon account and the old server will forward on your follows.

warkdarrior• 6 years ago

"This [federation] system also makes it hard for marketing and spam to get a foothold"
"You’re certainly familiar with another federation which is not based on ActivityPub: email."

Self-contradicting article?

ddevault3• 6 years ago

Email is a bit different from the new federations we're seeing today: there's less weaker guarantees of identity, and there's less emphasis on choosing what other instances you federate with. In the example of Mastodon, the cost of setting up a new server is also too high to make spam effective, since you need a persistent, reachable account in order to interact with other instances on behalf of it. And the social factor counts, too, newer federations like this tend to be opposed to corporate influence in general and this makes it hard for it to get the initial foothold.

Email has a lot of spam, but a federated system which keeps this in mind from the outset needn't suffer the same fate.

EDIT: Incorporating the replies to my comment, I also want to mention that the lack of large, centralized players like gmail also prevents spam. If there are a lot of intances, and they're all small, then cutting one off for bad behavior is a lot less difficult. Instance admins can be held accountable for the behavior of their users.

Jasper_2• 6 years ago

That would imply that centralized solutions would have even less spam, since they have much stronger guarantees of identity. And yet sites like Twitter has plenty of spam. Even Hacker News gets spam! And this is not a popular place in the grand scheme of things.

How is distributed identity solved in the long term? Nobody would run a server to communicate spam; that's rarely how email spam is sent these days. Most of the spam I get is from accounts hosted on GMail and Yahoo!. So the weakest link is a single server that's community-respected, but doesn't have adequate spam protection. Federation makes that much harder, because the whole point is that every server is run separately.

Mastodon gets no spam today because it's smaller than Hacker News. There's relatively nobody on there.

jethro_tell2• 6 years ago

Email suffers from one of it's early defining operational principals which was to be backwards compatible forever. In this way, operators still have to decide if they should accept federation with weakly identified entities or close the door and federate only with high reputation sources. To add to that, some forms of it's adoption are difficult to work around with normal methods. Mailing lists are tough, since they spoof the senders address and break dkim. And once again, the decision between allowing the old way, and just blocking anyone who doesn't have proper dkim.

We see the complaints about this every time someone posts about an email server. It's hard to get the big guys to play ball. On the other hand, every time email comes up, people complain that there's too much spam.

riffic4• 6 years ago

note that you do not need a full-blown Mastodon server to participate in the AP ecosystem. There are very little barriers to entry here - one is being able to process dynamic scripts and the other is a registered domain name.

I think the lowest cost solution to this would be a WordPress site on a shared host somewhere, with this plugin installed:

https://wordpress.org/plugins/activitypub/

upofadown• 6 years ago

Email is one to one mostly. So reputation should really also be based on individual entities. The normal way to make that possible is to have email clients sign messages by default with the senders identity of choice.

The technology exists in the current system. We just need to activate it. This isn't a technical problem.

ilammy• 6 years ago

You can fight spam by refusing to federate with servers which are often spamming. That acts as a deterrent for servers to host spam since they'll lose participation. However, this only works if there are enough of other servers to federate with so that you can actually choose your peers.

The status quo with email is not like that, unfortunately. Either you federate with GMail and accept the possible spam, or you disrupt significant part of communications of your users by refusing to accept email from Google's mail servers, with Google not losing basically anything (and having no intrinsic reason to fight spam). So it becomes a loss-meh situation instead of win-loss.

fragmede2• 6 years ago

Almost - intrinsic to offering modern webmail, (and not simply selling SMTP as a service) is fighting spam. Empowered users aren't going to put up with receiving spam and can and will go elsewhere.

nemothekid• 6 years ago

You can fight spam by refusing to federate with servers which are often spamming

Awesome, so users will naturally gravitate to the servers that spent the most $$$ money fighting spam. You end up with Gmail all over again.

I don't know why you are using Gmail as the spammer in this situation. Arguably the 2nd biggest reason Gmail got so big is at the time their spam detection was lightyears ahead of everyone else. Then as they got larger they killed off smaller servers that didn't do the magic song and dance, and those users couldn't talk to most people as everyone was on Gmail. There's nothing stopping this from happening again on some new federation platform.

This is my biggest gripe with federated services. They all start from the point where "the Internet is centralized" and offer some proto-Internet to undo it. The Internet started federalized and the easiest solution to many of the problems that came with it was centralization. It's always unclear how you prevent this from happening again.

Semiapies• 6 years ago

Heh.

Honestly, I'd like to see more yammering about federation by people aware of the early federated systems of the internet and some perspective on their uses and failings.

Or, better, less yammering and more realizing. Preferably more so than small networks where Nazi Twitter is the biggest node.

littlestymaar• 6 years ago

Email is particularly broken regarding spam for at least two reasons:

- it's a distributed system and not really a federated one: you can set up your own SMTP agent and send mails to anyone without prior registration to a federation.

- anyone can impersonate anyone: if I want to send an email pretending I'm bill.gates@apple.com there's no built-in mechanism to prevents that.

Theses aren't fundamental features of federations, and came from a design which just overlooked how many bad actors email would attract when it became mainstream.

remram2• 6 years ago

You can set up your own Mastodon or Matrix server and send messages, I am not sure which federation you're thinking of that requires "registration".

The way the impersonation problem is dealt with in ActivityPub is very similar to DKIM in the SMTP world: the sender uses a cryptographic signature (HTTP signature) to prove that he is allowed to send messages from that actor. Of course this doesn't prevent someone from sending spam from their domain, in either case.

bantunes• 6 years ago

it's a distributed system and not really a federated one: you can set up your own SMTP agent and send mails to anyone without prior registration to a federation.

One could say publishing an MX record on your domain is that registration.

jasonhansel2• 6 years ago

Spam is a problem on centralized social networks also--and they have advertisements, which are really just well-organized spam.

To the extent that this problem is email-specific, the ultimate root cause is a failure to deploy DMARC. Any new federated system will require a DMARC/SPF-like mechanism for preventing request forgery.

toast02• 6 years ago

DMARC can effectively eliminate email spoofing. Unfortunately, that doesn't prevent spam, phishing, or malware spreading through email. In part because email addresses are largely hidden. In part because not a whole lot of people would notice if their scam mail come from webmaster@somerandomsite.example.net instead of from their bank.

throwanem2• 6 years ago

Or a federated system that was designed before anyone imagined spam could be a problem, and thus can only deal with it in a backwards-compatible way by piling on ex post facto hacks. There's no reason to assume the designs of today will be informed by the same naïveté.

erpellan• 6 years ago

I was slightly surprised to get all the way through the article without seeing a single mention of NNTP.

hinkley• 6 years ago

As a solution or as something that was tried before?

Spam as a term transitioned from, “flooding a communication channel with the same message” to “ads sent indiscriminately” due to a green card advertisement on NNTP.

What we need along with federation is federated moderation. There are some characterizations of content where we can agree to a label but not whether the label is bad. You might trust me to label gore accurately, I might trust you to label incitement to doxing. We might never agree on what constitutes baiting. Or the crossover point where a flame war should be filtered. So you might ban an individual who I think is “keepin it real” or vice versa. But ten of us shouldn’t have to do the exact same moderation in order to maintain a federation. N log n would be great, 3N would probably be tolerable.

wmf7• 6 years ago

In retrospect NNTP and IRC, where every message is replicated to every server, are not a good style of federation.

toast02• 6 years ago

Does IRC replicate every message to all servers? It's been many years since I was active on IRC, but I thought it was only sending messages along the network tree where needed. So a message to a user would be sent by the sender's server to the recipient's server, or, if not directly connected, to a server one hop closer to that server. And for messages to channels, those only go along the tree if there is a user in the channel from that side of the tree.

With today's IRC user count and today's server capabilities, you might have an easier time running on a single server, but there's maybe some latency benefits for users with geographically diverse servers (maybe with support for a server to server mesh instead of a tree, to avoid multi-hop message paths)

asim2• 6 years ago

Strangely I posed a question to HN just a little earlier about the future of sofware - https://news.ycombinator.com/item?id=24547534 - and you've answered that question as I was expecting. I have been formulating similar theories and ideas but it's unclear how feasible it is without a collective willing to take it the distance.

For me it starts with the basis of a communication protocol for services https://github.com/asim/mucp. And then moves onto something on a broader scale https://gist.github.com/asim/ad6e157294d39c2040a5d51ab3d3216.... I don't have all the answers yet, but I feel something is coming. An answer to our future needs and its not p2p as you say. P2P might be useful for trade but not services or long term ownership and privacy.

troquerre• 6 years ago

Traditionally two of the issues with federation have been sharing namespaces and sharing logins. Once you need a single source of truth, the federation breaks and you need to rely on a central authority. There are some new decentralized protocols (Handshake's decentralized naming system is one) that provide a solution to these problems so I'm bullish on federation in general — I think a federated system could break into the mainstream given the recent geopolitical issues affecting centralized services.

asim2• 6 years ago

Is it easier to just use federated identity that creates pointers to the origin account? e.g account exists on server A and a new signup to server B creates a link as acc://alice@server-a/blabla

ve552• 6 years ago

Federated and decentralized solutions have significant UX issues compared to leading products in the area. They're not only harder to use and slower, but are much worse at converting and keeping users to begin with, since they're not ran like businesses are, which take in more funding and try hard to optimize all of these user metrics as KPI. It will be a long time before this can change I think, users still do not (and won't) care if what they're using is federated or not. I say this as someone who loves the idea of federated platforms, but we cannot truly succeed until we have a way to convince large amounts of (normal) users to give them a try.

ddevault3• 6 years ago

On the other hand, the fact that no one is thinking about "optimizing KPIs" is a major reason why people like the up and coming federated systems.

jcims• 6 years ago

Eh, KPIs aren't inherently bad...they just trend that way because of the typical audience for them.

tylerchilds2• 6 years ago

I agree about your UX notes, but I disagree with the success criteria. I've wholly quit social media apart from the mastodon instance I'm on. There are enough quality people I'm connected with now that it's a success for me.

Am I missing out on content from Facebook/Twitter/Instagram because I'm not connected indirectly to everyone in their gardens? Sure, but I'd argue it's low quality content that was only a drain on my mental health.

I guess I'm trying to say, I think it's about the individual success and not the % of market share.

ve552• 6 years ago

I've enjoyed them myself too, and I still use things like IRC on a daily basis. The two impluses I see to get federation to a larger market share are A) to 'save' people from the platforms they're currently stuck on, but also B) get rid of, or at least diminish, their network effects.

You could argue maybe A) won't work or isn't possible or needed, but I feel B) myself a lot. For example I meet someone and want to talk to them, but it's likely they might suggest we chat via Skype or Discord or something like that, and if I don't want to use that service I'm often out of luck.

That, personally, is what bothers me the most - that network effects seem inherently anti-competitive because they force me to use a platform I wouldn't otherwise want to use, only because they purposefully make their software unable to communicate with competing platforms.

For all of the open protocols and specifications we have from HTTP to TLS to HTML and literally one thousand others, we just re-use them over and over to create closed systems that can't talk to each other, which feels like going backwards. Sure it's great all of our computers speak TCP, UDP, TLS, HTTP, and so on. But they don't all speak iMessage, Zoom, Skype, Discord, Reddit, Twitter, Mastdodon, Matrix, Slack, etc etc. I still feel constantly locked-in to all of these platforms just when I want to communicate to someone that happens to be on it, and the area is certainly not progressing because the fight to gain this huge market share of users is never-ending.

Ideally, that is what I would love to end via federation. If I could actually use some type of FOSS and open protocol, and actually talk to any other person I wanted to with it, and I wasn't forced to use a certain platform, service, company, and give them all of my data in the process. I don't see any way for this to be achieved, but I can always dream.

grishka2• 6 years ago

KPIs that are being optimized result in increasingly user-hostile products, like Facebook and Twitter.

There certainly are UX problems that the current federated social media projects have. They're not technically unsolvable, it's just that current federated social media projects don't want to solve them for some reason. That's what I'm trying to address with my project (that's not yet ready to be demonstrated, sorry).

The primary problem that the current federated social media projects are facing is that there's no global user search. At all. You sign up for a Mastodon account on the instance of your liking, and now what? Sure you can follow Eugen and like the cat pictures he reposts. Cats are cute. But how do you find people you already know without asking every one of them whether they have a Mastodon account? You don't. And that's how you not build a useful social media service. Every single social media service that has had any semblance of success offered an easy way to bootstrap your social graph — be it a detailed global search feature and rich profiles to go with it or the ability to import your friend/contact list from somewhere else. That alone should be telling.

There are technical solutions to this, like making a DHT out of instance servers that could be used to link the user IDs on centralized platforms to ActivityPub URLs. Even if said centralized platforms refuse API access to instance admins, there's still machine-readable GDPR data export to take advantage of. It can be done. It's just going to take some time.

qlk1123• 6 years ago

The problems are big. What are the plans for your not-yet-ready project? Are they purely technical like what you stated in the final paragraph?

CKN23-ARIN• 6 years ago

People often forget that "the Internet" itself (i.e. IP, BGP) operates on a federated basis.

dpc_pw3• 6 years ago

Federated systems just don't work in practice because of inherent problems with them that don't have good technical&social solutions yet. Everything on the Internet (including Internet itself) degenerated from federation into monopolistic centralized services for a reason, and somehow people keep ignoring the evidence, and just keep trying harder.

These problems are so fundamental and universal that you pretty much need "a new form of Internet" that solves them, so then apps running in it can compete with centralized services.

So far the best stab at the problem is Urbit, but mostly because almost nobody else is even trying. Handshake and Holochains are another interesting projects in this space, but more of a pieces of potentially bigger picture.

IanDrake• 6 years ago

You would more likely need a new form of people.

Coke is number one for a reason. Sure there is Pepsi, and yes there are small soda makers that either get acquired or close, but not too many people make their own soda.

For the same reason, there won't be a federated social network...

Unless, people don't even realize they're using one, but that gets back to your point, we'd need a new form of internet.

zrail• 6 years ago

The current shape of the internet itself (BGP peering among autonomous systems, a decent number of large tier 1 ISPs, etc) is, on its face, a relatively successful federation. The applications running atop that federation are, as you say, centralized, but much of that is because residential ISPs forbid customers forbid servers and even if they didn’t (they haven’t always) residential customers are prone to accidentally running botnets of various types.

Mc912• 6 years ago

Everything on the Internet (including Internet itself) degenerated from federation into monopolistic centralized services for a reason, and somehow people keep ignoring the evidence, and just keep trying harder.

Sure there's a reason federated networks like Usenet died - they were killed off by a combination of the old/new AT&T/Verizon duopoly and various government organizations ( https://www.cnet.com/news/n-y-attorney-general-forces-isps-t... ). Yes, it happened for a reason - the old Bell last-mile monopoly and active government measures to quash it.

rootusrootus3• 6 years ago

Usenet died from a combination of web forums being a better experience for most people, and a saturation with warez binary traffic becoming so expensive that ISPs did not want to pay for it.

throwanem2• 6 years ago

If these problems are so obviously ineluctable, why talk around them instead of detailing or even naming any?

smichel173• 6 years ago

I am not sure this is self-evident; it could be that there is no end state, but rather a cycle of centralized and federated systems, and we simply haven't seen enough of the cycle to identify it, yet. As a parallel, consider systems of government in the last 2500 years or so.

walterbell2• 6 years ago

Could you share examples from the governance cycle?

F3nd0• 6 years ago

So far the best stab at the problem is Urbit, but mostly because almost nobody else is even trying.

What about GNUnet? They’re trying to build pretty much what you described, from the ground up. https://gnunet.org/en/

dpc_pw3• 6 years ago

I'm aware of GNUnet, and been following it loosely for too long to remember. It's been around for like 2 decades?

Superficially it looks to me like bunch of theoreticians overcomplicated everything, and it will never be practical. Probably great project to advance your academic carrier though. But maybe I'm wrong.

reificator• 6 years ago

Federated systems just don't work in practice because of inherent problems with them that don't have good technical&social solutions yet. Everything on the Internet (including Internet itself) degenerated from federation into monopolistic centralized services for a reason, and somehow people keep ignoring the evidence, and just keep trying harder.

Email would beg to differ. Google and Microsoft may have a lot of users, but interoperability is astoundingly high.

Yes, there is a high barrier to sending emails from a new provider, and yes, Google/Microsoft have a large chunk of the market. Still, there's little barrier to using ProtonMail or Fastmail or whatever free host you want, and everything interoperates shockingly well. You don't typically have to think about "what if this person has gmail?" or "I don't know if I can send to that host from outlook".

clusterfish2• 6 years ago

Email was already too big to fail by the time Facebook and Google got big enough to be able to kill it.

Looking at stuff created in the last 10-15 years, are there any federation successes? I can't think of any. (Blockchain excluded for the reason stated in the article)

thayne3• 6 years ago

Well most people use one of a few big email providers, not small individual mail servers run by tech-savvy friends or family.

I don't think that is necessarily a bad thing, but I think for federated services to catch on, you need big players that provide reliable service and a recognizable brand. The average person is not going to want to search through thousands of small federated servers to try and find one that works for them. And running a single server might be simple and inexpensive, but making it reliable and secure is not.

kstrauser• 6 years ago

You mean, other than Mastodon, which has a few million users and growing.

bsagdiyev• 6 years ago

My grandma will never use Mastodon, but she uses Facebook. Tech isn’t the only audience you need.

ApolloFortyNine• 6 years ago

Mastodon is proof at least that federation doesn't fix the issue of censorship. There's a common ban list in Mastodon that if you don't include, you will not be able to join the "main" Mastodon cluster.

And if your not part of the main group, what's even the point of federation? It's practically "servers" at that point.

shmerl3• 6 years ago

E-mail doesn't work too? Federation works, if some walled garden obsessive lock-in freaks (like Compuserve and AOL of old or today's analogies of Facebook and the like) don't manage to mess things up.

TylerE• 6 years ago

Email sucks.

Spam is ramptant. Running a server is a nightmare.

Standards are a tar pit.

It "works" only for fairly loose definitions of "works".

samatman3• 6 years ago

Urbit is also not a federated system, but rather, a true distributed system.

Each urbit instance is a peer. The stars and galaxies provide discovery and initial routing, which is a pretty neat solution, they're basically fancy DNS servers.

What I think is exciting about urbit, is that each identity can host services. So you can launch a chat room, and it's yours, other peers can show up and say things in it, but this is very different from federation, where there is a sharp distinction between administrators and users.

Anyone who comments in your chat room can have a chat of their own, and the cryptographic identity of you and them is identical and consistent, wherever in the fleet you may encounter each other.

Stability and performance have improved, but there's a long way to go before a smooth one-click user experience is a reality.

But the basic design is solid, and it has a chance precisely because it isn't federated, but distributed. Secure Scuttlebutt is another truly distributed system, of very different design; it's locked in some bad architecture but I found it quite pleasant to interact with.

Don't expect anything useful from Holochain, ever. Just... I don't want to get into inside baseball, but, nah. Never gonna happen.

dpc_pw3• 6 years ago

Yeah, for me a federated system taken to its logical conclusion turns into p2p system.

SSB is neat, but ineed could use rearchitecting. I could imagine ssb-like system + handshake to work quite OK for certain things.

Thanks for info on Holochain. I've been only lurking in there and reading docs, so my understanding is rather shallow and theoretical.

trabant00• 6 years ago

Federated systems just don't work in practice because of inherent problems with them

The entire form of current global human society is federated so you can't simply state it does not work. For the internet so far we have a few empires that battle it out, but I don't think they are manageable long term and this has started to become apparent.

mariusor3• 6 years ago

because of inherent problems with them that don't have good technical&social solutions yet

Would you care to mention them?

jasonhansel2• 6 years ago

For those who dislike federated services, keep in mind: email is a federated system, and it remains arguably one of the largest social networks in the world.

Email's federated nature has kept it alive for decades, and will ensure that it long outlives more proprietary forms of social networking.

Federated systems already work in practice and at scale.

edit: Some other examples with widespread adoption:

- The phone network (PSTN)

- The internet (via BGP)

- The postal service (federated between countries via the UPU)

- Podcasts (and RSS more generally)

Am I missing any?

rootusrootus3• 6 years ago

arguably one of the largest social networks in the world

At the same time, I know several people who gave up their email addresses and only use Facebook now. Maybe the next iteration of social media behemoth will be even more effective at that.

walterbell2• 6 years ago

How do they reset their Facebook password?

rtkwe2• 6 years ago

Unfortunately 2 of those I almost entirely get spam and business/commercial interactions through. On my cell phone I've gotten one call in the last 6 months that wasn't spam and it was an automated notification from the Sheriff about a permit I'd applied for. I haven't gotten a single useful call from someone not on the whitelist of my contact page in so long it's tempting to just autosend all unrecognized numbers to voicemail.

Email in particular I don't think really fits the mold of social network. I think principally because there's not really a public post concept it's all essentially point to point communications. There are some abstractions like mailing lists but they're still an express list of people who will get it. I guess that might not be so different from a 100% private profile but the lack of any public functionality excludes it from the social media category.

thekyle• 6 years ago

What about SMS? I don't know where you live, but here in America everyone I know uses it.

afiori• 6 years ago

Personally email is a bad comparison in my opinion. The way it is federated makes easy to fossilize the protocol and the "social" nature is at odds with the federation due to the spam problem.

And in general I consider it a terrible method of communication.

sellyme2• 6 years ago

[email] remains arguably one of the largest social networks in the world.

What percentage of email traffic do you think is social in nature (and not just notifications from an actual, separate social network)? I'm guessing <0.1%.

Vinnl• 6 years ago

- Podcasts (and RSS more generally)

Podcasts exactly exhibit what I fear for with federated systems. As podcasters move to "Spotify exclusive", Spotify still calling those podcasts, and those podcasts not being available through your general podcast subscription app, could lead to most podcasts being de facto centralised again.

The same could happen to the Fediverse: some player adds some feature that makes large swaths of people migrate there, those features are not interoperable, and then that service turns off federation altogether...

Mc912• 6 years ago

I followed somewhat the history of the various file sharing protocols, from Napster and OpenNap to Gnutella to Bittorrent.

None of them really stayed totally p2p. With Gnutella, nodes would slow network connections and weird NAT conditions would pop in and pop out. So eventually things changed, and some Gnutella nodes which were online for some time period and had a fast network connection would self-elect as super-nodes (or ultrapeers, or various names). In Gnutella, theoretically ever node was a server and a client, but eventually, Gnutella server/clients would hook into to these super-nodes.

If you want the maximum amount of content, both centralization and decentralization have their drawbacks. You tend to find elements of both (with Gnutella and Bittorrent too).

dzink• 6 years ago

Start this labyrinth from the end point. Why are giants attracting 95%+ of users?

1. Among other things, because of a dead-simple user experience that lets even grandpa post content after a few clicks. Federation, if it adds any level of complexity past the grandpa UX, would defeat the goal.

2. Next, identity. If grandpa can post, so can the bots, how do you ensure accounts don’t go to fake entities? What if an account has to pass a quality threshold before it can publish in public, but can be used privately any time. HN has similar model, but here you limit public posting until 10 random other users have seen and upvoted the new user’s post. Add to that possible additional authentication for payment of a pro feature, or a human check mark, and you’ve put a hurdle on spam.

3. To keep culture from running amuck provide positive and negative feedback. Public content has a span problem, not so much private behavior. Content ratings and discussion licenses can come into play as well, where any group can limit posts to PG content, or discussion that gets moderated automatically or manually if it passes certain thresholds.

Happy to suggest more if needed. I run https://www.dreamlist.com

donmcronald• 6 years ago

Next, identity. If grandpa can post, so can the bots, how do you ensure accounts don’t go to fake entities?

Invite only. If I'm running a federated service for me and my family, the moderation is going to be astronomically better than anything big providers are doing because a couple minutes of moderation by me every week is astronomically more on a per user basis than what someone like Google or Facebook is doing.

chc• 6 years ago

That's a nonstarter for a service that's for anyone but the geekiest of geeks, though. If you have to be close to someone with both the tech know-how and interest to host an instance of a federated service, the service is effectively off-limits to approximately everyone. Non-technical users have to be able to use the service and have a good experience doing so.

nicoburns• 6 years ago

How does that work with interacting with other nodes in the federation? Now you have the whole huge userbase to deal with again, and only you to moderate.

bawolff3• 6 years ago

If you're not talking to wide internet, what's the point of federating?

If you are, while i think email has proven that moderation is much much easier at scale if the attackers are spammers (astroturfing is a bit of a different story though)

synctext• 6 years ago

Next, identity. If grandpa can post, so can the bots

Yes, protection against a million fake accounts is the key scientific problem to solve (Sybil attack).

Our university lab has worked on this problem for 15+ years, its hard. Early indications are that ActivityPub and federation are not going to scale. Bot attacks are too ferocious for amateur admins. Big instances attract more people, however a single instance fails to scale.

Nobody knows how to build trustworthy self-organising systems with "self-governance". Somehow you need to deduce the reputation of others from their signed interaction history. Their privacy needs to be protected, but you can still calculate a reputation score. After 20 years of scientific progress around "distributed reputation systems" with 100+ scientific articles they have one thing in commons: they don't protect from the Sybil attack.

Server federation is not going to help if there is no trust fabric to build upon. The effortless onboarding plus mobile-first requirement of today means you even need to have smartphones talking to eachother directly, P2P-style. <plug> thankfully students in our lab now have carrier-grade NAT traversal operational between Android devices. Google: "Technology Stack for Decentralized Mobile Services" (quotes required).

tmaly• 6 years ago

I totally agree, the interface has to be simple.

First movers always take the majority of the market share if there are no compelling reasons to use a similar service.

MentallyRetired• 6 years ago

I also agree. "Federated" is not a killer feature. Simplicity is, and that needs to be a priority.

drran• 6 years ago

I totally disagree. There must a high bar to entry, for useful discussions, instead of talks about tits and shits. Look at HN, for example.

bawolff3• 6 years ago

I think also a certain amount of non freedom helps.

One of the big things facebook did is you couldn't customize your page. No autoplaying midis with seizure inducing gifs. Also helps with brand cohesion.

Federation gives people freedom. Quite frankly users cannot be trusted to maintain a good experience if given freedom.

andyroid• 6 years ago

So instead of we’ve got rampant racism, conspiracy theories, fake news and attempts to sway elections in that direction all over the world. If that’s the good experience, bring back the midis!

marksbrown• 6 years ago

Most of us here presumably recall the early iterations of Twitter and Facebook. Both succumbed to dark patterns a very long time ago. Your argument seems to be a slow death is preferable to a fast one.

throwaway333392• 6 years ago

Among other things, because of a dead-simple user experience that lets even grandpa post content after a few clicks. Federation, if it adds any level of complexity past the grandpa UX, would defeat the goal.

You could have stopped there, that's pretty much the only reason. But that has little to do with federation though. And yeah the ugliness and bad UX problems of free software - software that's freely unusable - have been known for literal decades.

HeadsUpHigh3• 6 years ago

1. Among other things, because of a dead-simple user experience that lets even grandpa post content after a few clicks. Federation, if it adds any level of complexity past the grandpa UX, would defeat the goal.

That's the point though, design for the lowest common denominator has been the modus operandi for the giants because it gets them the most profit. There is no proof that this is the way to create the most "useful" social network. Often I'm tired of the "organic spam" that users with easy access create non-stop in a lot of the communities I'm in. A technical barrier to entry isn't necessarily a bad thing in this case.

dalbasal2• 6 years ago

Being networks, a fundamental definition of "usefulness" is directly (or exponentially, rather) related to the size of the network. It's not the only aspect of value, but it is exponential, ad hard to beat.

One of the reasons amoral commercial efforts like fb won is because this modus operandi happens to harmonize standard commercial MO with what a network is.

There's definitely value in alternative MOs. As you say, a technical barrier to entry can have its uses. Wikipedia is a nice example. I don't think simplifying the editor UX to an intuitive one would necessarily be useful. But wikipedia needs a throttle on editing.

I'm not sure this relates to federation though. Niche communities don't need to be federated.

Jetrel2• 6 years ago

I do sympathize with the argument (I've made it myself, in the past), but ...

... the problem with that is there are a lot of damned smart people who are good at what they're good at precisely because they view "wrangling tech" as a complete waste of their time.

Artists and musicians are a really good example, and are the key reason I use Twitter (pretty much the entire indie game industry lives on that platform), but this principle also extends to doctors, lawyers, scientists, teachers, historians, practically everyone in all the sciences, trades, and humanities ... except computer programmers. I've had aspirations towards being a polymath, and one of the bitter things I've learned about the programming-autodidact death march is how much of what someone learns isn't important fundamental knowledge, but is just byzantine details that will evaporate half a decade later when the software in question goes obsolete. It's excruciating to waste time learning something like that (which will become worthless) when I could be learning core programming fundamentals -- or could be learning something in a different field like drawing that will stay valid for the rest of my life.

I was a desktop developer looking-in-from-the-outside at web development, and I timed my entry into the field specifically to avoid the hot mess that was the API differences between netscape (yes, Netscape Communicator 4.5) and IE 6 at the time. As a programmer with enough of a nose for bad code smell, I took one look at that and thought to myself "this is completely insane." I could have knuckled down, busted the books and gotten all those api minutiae down just pat. Just in time for jQuery (or future browser versions) to show up and render them irrelevant forever.

It wasn't insane because it was a mess - it was insane because it was definitely going to go away, and soon. And once it did go away all the time I'd spent learning and mastering it would have been utterly wasted, sure - but wasted at the expense of not doing something else that's more important. That's time I could have spent learning the piano. Or a foreign language.

Now - not all programming is like that - eventually you've gotta bite the bullet and decide something's "good enough to get the job done". Many tech stacks actually end up in a nice sweet spot where they're coherent enough to learn quickly, and stable enough (in the sense of an API) to get out of the way and let you approach long-term projects without the ground shifting under your feet. But if you just blindly vacuum up proficiencies as they come off the treadmill you'll quickly discover you're doing nothing else with your life.

This is the relation most humans have with tech. Especially the smart ones who are really into something else.

It's a means to an end - and can work wonderful magic. But it demands a price.

dalbasal2• 6 years ago

I think the "feed" aspect needs to be considered also.

The power of Facebook, Twitter and such is largely their feed. The feed is an answer to point #1: grandpa simplicity. The feed is the thing that needs to be protected from points #2 & 3: identity and culture.

The feed is that will motivate participants, content creators, etc. IE, many will create content for the feed. Reddit/Digg/HN upvoting were a simple way of doing the feed that proved this point and paved the way for Facebook & Twitter. A lot of mass adoption services created after went extreme. Feed only. The Tinder UX is a feed of datable people, for example. Spotify's importance is being solidified by being a feed. Importance in spotify's context is largely "being the place where music gets popular."

One way or another, the feed issue must be addressed. New content UIs will either have a successful feed or transcend the feed some way.

I'm not sure how this relates to federation. A feed seems a centralizing element by nature. Successful federated systems of the past (the the WWW) tended to leave the feed ("discovery," in a pre-feed context) to someone else... which turned out to be google, fb, etc.

However the labyrinth is is designed, what matters is how the feed works.

jondubois3• 6 years ago

> Why are giants attracting 95%+ of users?

Because they get free money from the Federal reserve through cheap loans and government contracts and money attracts people. They can use the money to saturate the media with their brand until everyone ends up using them. Then they can use the money to crush all potential competitors.

shawn-furyan• 6 years ago

This system also makes it hard for marketing and spam to get a foothold
You’re certainly familiar with another federation [...]: email

eternalban• 6 years ago

The only practical issue standing in the way of decentralized services is a lack of business vision. No one has managed or bothered to propose viable business models for decentralized services.

Novel approaches to the business side of decentralized services will naturally allow for a re-think of the UX issue. (Business model -> default config for "grandpa")

An alternative solution approach requires faith in one's government. Can you trust it? If you can, then a national identity services (yes, the dreaded digital id) can provide the infrastructural back-bone to build federated systems.

At the end of the day, location and identity is -all- you need to "reach out and touch someone".

Icyphox2• 6 years ago

No one has managed or bothered to propose viable business models for decentralized services

I think that is for the better.

danbolt2• 6 years ago

Earlier in the month, the website OnlyFans set a price cap on what creators could charge for content in response to a celebrity selling photos that weren't as advertised. This affected the incomes of a lot of sex workers whose content contributed to their livelihoods. [1]

I'm not knowledgeable about sex work at all, but a lot of the discussion I read noted that it seemed troubling how OnlyFans had so much authority over how the content creators did business. It's not uncommon to hear how sex workers or people in the adult industry are at the whims of opaque moderation on social media too. [2]

Sex work can be an exploitative industry, but I think federation could bring a lot of power and control back to the content creators involved online.

Disclaimer: Again, I'm not an expert on the topic of sex work or knowledgeable about it at all. It'd be a lot better to get feedback of the people involved or know the industry better.

[1] https://www.bbc.com/news/business-53979625

[2] https://jezebel.com/instagram-s-solicitation-policies-are-ex...

gpm5• 6 years ago

In the end, the problem sex workers have on the internet is that they are a huge fraud risk, with that risk coming from both the provider and the client. The mainstream payment providers won't touch it as a result. The less mainstream ones are expensive and still not easy to do business with.

Without easy setup of payment providers, federation is difficult. Any new instance in the federation needs to go through the same pain.

Meanwhile, what does federation actually solve? The only fans model (and most sex work) seems to be a social graph that is a single producer talking to a bunch of clients. The producer needs a server, clients need to connect to that server, where does federation come in?

Disclaimer: I'm not at all knowledgeable either - literally just know what I've read on hn on this subject.

ampdepolymerase• 6 years ago

The bottleneck is the Mastercard/Visa duopoly combined with the modern banking system. At the end of the day no amount of Patreon vs Shopify style business models for adult entertainment work fixes the fundamental payment risk problem. Onlyfans helped amortize the risk somewhat by spreading across multiple payment providers and controlling the UX tightly but even so they still had to change the pricing policy. Mindgeek/CCBill are the Paypal/Stripe of adult entertainment payment processing and it is not an easy field to "disrupt". You can put the best actuaries and machine learning engineers in a room and it would be doubtful if they could come up with a better risk model without fundamentally altering the method of transaction. (Crypto is often touted as a supposed fix for this sort of things but I am not knowledgeable enough about it to comment, somebody else please feel free to chime in.)

534626• 6 years ago

You're actually spending time and energy feeling bad for women who probably make more money selling nude (or often, just partially clothed) photos of themselves than you make selling your games.

danbolt2• 6 years ago

So?

laksdjfkasljdf• 6 years ago

Linux distros/brew/npm/maven/etc are federated software distribution.

wmf7• 6 years ago

Not really; most of them have centralized namespaces and servers. PPAs, Flatpak, and Deno are federated though.

ibn_khaldun• 6 years ago

Mastodon, Misskey and Pleroma are glorified clones of Twitter, and I'm sure there are many more like that. Pixelfed is a glorified clone of Instagram. A few others that I've looked at just don't seem appealing, at least not to me.

People need to stop harping over the technical appeal of the Fediverse, lest they only are interested in courting the technical crowd, which is actually fine with me. If that's what they want more of then they should not be concerned with who's a part of it. I think these federated platforms should not be interested in trying to appeal to anyone at all, because it's resulting in imitations of the exact platforms that we want to get away from. Under the hood they may unique, but all most people are going to care about is the front end.

Personally, my preference for a new social networking service would be very, very plain. Chat-orientated, share documents, comment on shared documents. No hearts or arrows or "Rebeeps, repoots, and boobops". Is there a way to federate message boards? I just want to talk, read and talk about what I've read. Is there a way to federate Hacker News?

ernesth• 6 years ago

Is there a way to federate Hacker News?

There's Lemmy https://github.com/LemmyNet/lemmy

mariusor3• 6 years ago

There are others too.

sk20203• 6 years ago

I remember talking to the author of Lemmy on Matrix before he made the github repo. Perhaps this has changed, but he didn't seem to have any actual plan for federation. The early work in the repo appeared to be a Vue or Angular frontend (I forget which) with a Reddit-like style with server-to-server communication left as a "TODO". The author seemed to me to be mostly preoccupied with writing essays about why communism is a good idea, so I assumed the author is "an ideas guy" and didn't expect any actual work to get done on the Reddit-clone.

Then one day, for no obvious reason, the repo received hundreds of stars within a few hours. It was a trending Rust repo and was recognized on TWiR, but there was no actual implementation of federation at all. Just a JS frontend, and I think a tiny bit of stub code in Rust.

I'm not saying there are upvote rings or botnets amongst any particular group, but I haven't paid attention to GitHub stars since then.

sk20203• 6 years ago

federate Hacker News Mailing lists can be somewhat like that.

I would think that most any pub-sub system with a minimalist protocol that supports text and maybe some file types would be adequate to build on. However, I think marketing and social factors have more to do with success and failure of these platforms than anything else. There are certainly a lot of capable programmers who have tried or who are trying to make a social media platform without pervasive user surveillance.

shmerl3• 6 years ago

Diaspora* and Matrix are using federation.

Thorentis2• 6 years ago

My issue with federation is that so far, everything still requires some centralisation. Take Matrix for instance. The most popular homeserver by a long, long shot is the one that the Matrix foundation runs themselves. This is same for Mastadon, PeerTube, etc. If you let this experiment play out long enough, you end up with a de facto centralised system.

I think P2P ends up being closer to the goal of distributed control because it doesn't tend towards centralisation over time. My issue with P2P though is that it still requires trackers for boostraping in most cases. DHT only gets you so far.

I think there will be some interesting advances once IPv6 becomes more widely deployed. Hopefully this time around multicast is widely deployed on the internet. Imagine being able to subscribe to a multicast group and instantly receive DHT nodes over the internet without a tracker? Not sure how this would work in practice, but I have heard of some research being done here. I think the future looks very good for P2P applications, whereas federated applications are still on shaky ground.

wmf7• 6 years ago

Ultimately I think developers need to take some responsibility for causing and fixing de facto centralization by not steering new users to "default" servers and refusing signups once a server has excess market share.

Federated communication or social media works differently than P2P file sharing. In file sharing you're locating random anonymous peers but in social media that's the last thing that you want. When communicating with a specific person you need their identity anyway and you can use some system like DNS or a blockchain to find the server backing that identity.

Multicast on the public Internet is not going to happen due to state management cost, and it's not helpful for federation anyway due to the spam problem.

Icyphox2• 6 years ago

My issue with federation is that so far, everything still requires some centralisation. Take Matrix for instance. The most popular homeserver by a long, long shot is the one that the Matrix foundation runs themselves. This is same for Mastadon, PeerTube, etc. If you let this experiment play out long enough, you end up with a de facto centralised system.

Yeah this was Gagron's fault. Running a flagship instance was bound to result in this. I make it a point to educate people about not joining mastodon.social and finding another instance that better suits their interests.

samatman3• 6 years ago

Mastodon and Matrix seem quite different in practice, in this regard.

Who cares what the most popular Slack or Discord is? I'm sure there is one, I bet it's a lot like hanging out on ##hottub back in the day. Matrix is fantastic for organizations.

Mastodon is... I mean it's a Twitter clone at end of day. People want reach, no on should be too surprised that poasters tend to cluster in the biggest instance.

I think chats for companies/organizations/projects are a good fit for federation, and with the adoption of Matrix by the French and now the German government, Slack should be looking over its shoulder.

Thorentis2• 6 years ago

What I'm saying is that despite Matrix and Mastodon saying they are federated services, in practice they (and things like them) tend towards centralisation anyway. The main selling point of Matrix and Mastodon is that they aren't meant to be centralised. Slack and Discord are much better products (at the moment) if you want a centralised solution. I have not yet seen a service/protocol/product that claims to be federated/decentralised, and actually is in practice. There's always a central server that heavily dominates or some kind of central system required for the whole thing to operate.

lokedhs3• 6 years ago

Matrix does require some de facto centralisation for the identity server. But Mastodon really doesn't. I can set up a server right now and start federating immediately without involving any central system.

phailhaus• 6 years ago

I'm not convinced federation is immune to corporatization. The article claims that "it optimizes for a self-governing system of human beings talking to human beings, and not for corporations to push their products." Replace "corporations" with "people", how does federation stop people from using the platform to sell products? Especially when in the same breath, the author admits that operators may be pushed to "solict donations" from their users. Or maybe, ads?

Again, you end up with a system in which money is correlated with user experience, and the operators who most efficiently monetize their userbase will be the most successful.

jandrewrogers• 6 years ago

I work on the design of novel federated systems for adverse environments, like the Internet, where you have to consider the full range of threats to system behavior, functionality, and economy. There is a pretty long list of theoretical trade offs and hard design problems that do not have practical solutions today, which makes it an interesting research area.

Some of the more interesting open questions for systems that want to strongly enforce the sovereignty of logical nodes:

How do you design inter-node protocol strategies that exhibit optimal, robust Nash equilibria in non-trivial real-world environments i.e. uses resources efficiently in all plausible configurations absent centralized control? This is fiendishly difficult because relevant properties of nodes scale in complex, interacting ways and game theory is not sufficiently developed to solve some of these cases in practical systems. In any sufficiently complex and real-time federated environment, you want these mechanics to be almost completely automagic. Currently, there is a bias toward restricting the number of logical nodes in federated systems because that means humans can plausibly deal with the brittleness, which implies centralization.

How do you prevent nodes from being attacked/exploited from both the outside and inside, including malware, DDOS, fraudulent data, exfiltration of private data, etc? This is a separate question from the intrinsic theoretical robustness of the protocol design in terms of resource and scaling efficiency. Also, how do you make a policy easy to describe and enforceable with strong guarantees? The state-of-the-art in high-assurance internetworking (which considers these kinds of problems) is quite far from where it needs to be address these issues convincingly.

How do you audit user behavior that spans multiple sovereign federation domains to limit abuse and misuse of the system while leaking minimal private state?

Some of these are just Hard Problems, unless you have a very simple/restricted system, that we will need to carefully design around to minimize human involvement; most federated system designs just ignore these issues. For others we have plausible approaches theoretically but no real technical implementations with sufficient generality or functionality. And then there are the fundamental trade offs and edge cases that we tend to ignore but which are really important at scale.

tl;dr: federated systems make distributed systems look easy. The reason they are not commonly used is that a correct design for a non-trivial system is deep into the frontier of computer science. We don’t have a complete theory for how to do it.

Santosh834• 6 years ago

On the other hand, only if we plunge in will we learn to swim. The solutions will come when there is enough at stake to solve them, but right now since almost no one is using decentralisation/federation at scale on the consumer side, no one has the motivation to tackle the problems you have mentioned. Waiting for a perfect solution to arise before jumping in will ensure it will never happen.

Santosh834• 6 years ago

Personally speaking I'd like to see a dead simple to self-host CMS that has built-in comments and RSS, making it trivially easy for people to spin up a simple website or blog or even a webpage, and connect it to a domain they have purchased (or even get one from within the CMS).

And by dead simple, I mean something grandpa can do. All they should need is enough skills to install an app, click through a welcome wizard, and start writing on its WYSIWYG editor.

swiley2• 6 years ago

You can just open a word processor and save the files to /var/www/ in html format. You don't even need an installation wizard (although on most mainstream OSes you do have to push a button in a GUI somewhere to enable the webserver.) If (for some reason) your computer doesn't have a public address you can install TOR. People with normal browsers will be able to access the pages (even from search engines) via gateways.

Santosh834• 6 years ago

I get what you're saying but grandpa is not going to be using Unix (Linux) so /var/www/ is out of question. I'm thinking more along the lines of Publii with a built-in Caddy, just to give an example.

If the computer doesn't have a publicly reachable IP then that's a much tougher obstacle. That can only be solved by voting with your wallet for ISPs that do give you a public IP with reachable ports. Net neutrality is critical for decentralisation and self-hosting.

foreigner• 6 years ago

Just occurred to me that passports (the paper ones that you use to cross borders) are an example of a federated system.

jondubois3• 6 years ago

Nice article. That said, to talk about the benefits of software federations without conceding that cryptocurrency is the ultimate foundation for such an ecosystem is delusional.

Money is the only incentive which works in our modern age. Any federated or decentralized system needs to start with money because money is the ultimate tool for control.

jondubois3• 6 years ago

Cryptocurrency is the digitally woke techbro’s ponzi scheme

The fiat monetary system is a ponzi scheme. Watch "Hidden Secrets of Money": https://youtu.be/iFDe5kUUyT0?t=62

I also recommend doing your own research after this. The evidence is clear and overwhelming.

Cryptocurrency is merely a response to this scam and its continued success helped to further prove that the financial system is a scam.

Even if you assumed that cryptocurrency is a scam, what kind of financial system enables a scam to thrive for 10+ years? In such a system, how can you tell the scams apart from the non-scams?

Clearly the so-called 'free markets' are not capable of distinguishing the two.

What's to say that most big tech corporations aren't scams? They get a lot of 'free money' from the government through massive contracts to build projects which only add more bureaucracy.

mariusor3• 6 years ago

This is an eloquent description of what my reasoning was when I started to work on an ActivityPub powered link aggregator[1]. I hope Drew will allow me to re-use some of his words to improve it.

[1] https://littr.me/about

The unrealized potential of federation

Related Stories