Locking the Web Open: A Call for a Distributed Web

http://brewster.kahle.org/2015/08/11/locking-the-web-open-a-call-for-a-distributed-web-2/

by hachiya • 11 years ago

221 97 11 years ago

brewster.kahle.org

wwwtyro• 11 years ago

I'm all for this, and consider it to be inevitable in the long run. In the short term, however, it seems like the major hurdle will be getting one of these projects into the mainstream: for the most part, the web already does what most people want it to do, and those people aren't going to be bothered to install a new web browser so that they can do things they're already doing. Especially if it lacks the features, performance, or ease-of-use of their current browser.

So, how do we address this? Is there a "killer app" for the distributed web that will motivate people to move to it? Can we use existing web tech like Web-RTC to bootstrap the system? Maybe a workable avenue is mobile, where people are pretty comfortable installing new applications - what if we built the next social network into an app based on the distributed web?

I don't know the answer, but I'd love to hear any ideas/brainstorming you clever people have to offer.

lsjroberts• 11 years ago

You may be interested in https://ind.ie/about/vision/

molsongolden• 11 years ago

https://indiewebcamp.com/ is a bit more interesting and open.

gagege• 11 years ago

I love the idea of doing a distributed social network via a mobile app. Being a native app, it could be kind of seamless as well, for the user.

mtgx• 11 years ago

Going mainstream would probably require something like Firefox or at least Ubuntu integrating IPFS.

simon_vetter• 11 years ago

Implementing other protocol handlers into existing web browsers and operating systems would be a good start IMO.

Taking ipfs/ipns [1] as an example, having handlers inside web browsers would allow people to link from http[s]:// to ipfs:// and vice versa in a seamless way, lowering the barrier to migrate.

From there on, there's nothing preventing you from distributing your application code (html/css/javascript/whathaveyou) over ipfs, and make use of WebRTC for user-to-user interactions.

Obviously http[s] is going to stick around for a while as it has its use cases (basically anything that deals with a centralized service, from online banking to search engines to apis), but having a secondary, peer to peer means of distributing content and applications would be a major plus.

[1] http://ipfs.io (they have a working implementation in Go with an HTTP gateway as well as a FUSE filesystem)

petrosh• 11 years ago

WebRTC seems the weakest point to me: WebRTC still needs servers for clients to exchange metadata to coordinate communication http://www.html5rocks.com/en/tutorials/webrtc/infrastructure...

basicplus2• 11 years ago

If every one had a wireless node on their house you might get an open web.. Just leaving the problem of connecting to the backbone

slxh• 11 years ago

too bad Google blocks adhoc WiFi on Android... that might have helped too.

EGreg2• 11 years ago

I think it's obvious that the current web is decentralized, but is heavily server-based. At the same time, there is something about propagating applications across these servers... Russia can ban Reddit but they can't ban Wordpress. For the moment, that is what we are working on at http://platform.qbix.com (and have been for the past 4 years). Making it easy to have a distributed social network the same way bitcoin makes money distributed.

Now, how would you take it further and make the web entirely peer to peer, so you wouldn't have to trust servers with your security and politics? You can have additional schemes like http and https, for various methods of delivery and storage.

I wrote this FIVE years ago but nothing seems to have been done about it since then: https://news.ycombinator.com/item?id=2023475

That would be an easy first step, that would do a lot. It's 2015 and we can't even have XAuth (http://techcrunch.com/2010/04/18/spearheaded-by-meebo-xauth-...) in the browser! (We would need a space for storing preferences where websites from any domain could read what was written.)

jeena• 11 years ago

Russia can ban Reddit but they can't ban Wordpress.

That's why I'm part of the IndieWeb https://jeena.net/indieweb or http://indiewebcamp.com/

The nicest thing about that all is that I don't need to wait until someone else writes a whole new WWW, my own website already is a small part of the whole big thing, I just make my HTML more machine readable and implement something like pingback (but easier, it is called webmentions). With this small building blocks I am, together with others, building a social network which we don't even need to call that.

EGreg2• 11 years ago

IndieWeb looks great. I am going to try to get involved with it.

Could you please get in touch with me by email? You can find it at http://qbix.com/about -- just click on "contact". I would like to find out more about this movement ... I'm beginning to participate more in the Offline First, Distributed Web, Mesh Networking and other such movements. Our company's spent 4 years building a platform that would decentralize social networking, because we see it as the catalyst to giving users control of their own data. Most people in the world are just using centralized services these days, and it's directly related to how difficult it is to make a seamless social layer for the web. So I think that we're solving a solution parallel to what bitcoin did with money. A good solution unleashes new possibilities, like the Web itself did, like Email did.

Anyway, reach out if you can! - Greg

vezzy-fnord3• 11 years ago

The author's proposals are strikingly similar to Xanadu, right down to suggesting embedded micropayments: https://en.wikipedia.org/wiki/Project_Xanadu#Original_17_rul...

Ted Nelson will rise. Sort of, not really.

dweinus2• 11 years ago

I want this for all the reasons they list, but it seems there are huge unanswered questions for anything beyond a permission-less static page. Imagine you are developing a modern web app in the locked open paradigm. Is all system data distributed, including private user data and passwords? The only solution I can come up with is homomorphic encryption, which is not performant enough and still probably leaves a huge timing/structure analysis attack area if anyone can download the database. If I make any mistakes on the database security, the entire DB is already pre-leaked to the world? The final dencryption/encryption happens in client javascript, which is a whole other hornets' nest. Besides that, the implication is that I write my entire system stack in client javascript that is exposed to everyone, including any proprietary algorithms or credentials? Even if that was ok, and the system can live in the user cloud, where does system processing that is independent of user activity (scheduled tasks, etc) happen? Again, I want all of these problems to be solved, but they are nontrivial.

colordrops• 11 years ago

A system like this that uses passwords would be poorly designed. You would use private keys that are never stored online.

kodablah2• 11 years ago

Point by point...(sorry for the long post)

"homomorphic encryption, which is not performant enough"

It is fast enough on a per viewer basis, and in a DHT downloading the database doesn't mean it was all encrypted w/ one key. Each user encrypts his data as needed, or common groups of users encrypt data for each other with each others keys.

"If I make any mistakes on the database security"

This is why encryption is the underpinning. Sure you can still leak your private key like you can leak an SSH key today.

"in client javascript"

Nobody would use a distributed network where this was the case. In many cases (i.e. MaidSafe) they are developing a browser plugin for client side to communicate with the backend.

"where does system processing that is independent of user activity (scheduled tasks, etc) happen?"

Many of these now-being-designed systems have a pay-for-computing concept. Granted several (not all, unless you want to be limited by a single-file-line blockchain forever) have to agree on the results. Give some computing for other computes and get some. As for "scheduled task" timing issues are inherently difficult for these systems and I don't expect the "system" to trigger a job but rather a user to trigger it. Introducing timing into these distributed networks can be hairy.

The real problem that needs to be tackled is a way for the common human to hold his private key in his memory or some other non-digitally-retrievable way.

dweinus2• 11 years ago

Thank you for the thoughtful responses! I am still getting my head around some of this, so I love hearing solutions I have not thought of.

"common groups of users encrypt data for each other with each others keys"

I agree, but I think this can quickly lead to massive multiplication of data without careful cryptographic gymnastics. It puts more pressure on the application devs to do it right or more pressure on the network in terms of data if you don't.

"Sure you can still leak your private key like you can leak an SSH key today."

If I leak an SSH key, I can revoke it and only data that attackers have already grabbed is out. In the described paradigm, everything is already out to everyone. It is all or nothing. That might not be a difference from a theoretical point of view, but in practice it is.

MaidSafe is very interesting, thank you! It seems like more of a shared cloud, which is halfway between present cloud computing and the completely distributed utopia described in the article. It solves pretty much all of these issues, with the cost of being a less-centralized network rather than a fully distributed network. Awesome work, I hope they succeed!

williamcotton• 11 years ago

It's really exciting to see the these various technologies coming together!

We're working on the micropayments for authors and rights-holders aspect of this:

https://github.com/blockai/openpublish https://github.com/blockai/bitstore-client

dikaiosune• 11 years ago

I would love to live in this future -- but where's the incentive for businesses? How do they make more money developing in this way? How do users get more value accessing sites developed in a purely decentralized fashion? How do we avoid JavaScript being the basis for all of this?

Interesting (almost exciting) vision, but I don't see why the majority of existing users would move. They just don't get much value out of privacy, versioning, reliability, etc. They get enough of those things out of Gmail, Facebook, et al for their purposes already.

vezzy-fnord3• 11 years ago

The interests of society do not revolve around making life as convenient as possible for business. It is the onus on business to adapt itself to new conditions.

A distributed web scheme should orient itself around giving its users freedom. All other concerns are secondary.

zaphar• 11 years ago

This may be philosophically correct. But in a non platonic realm you must somehow fund these sorts of things. If it's not possible to make money on them then it may not be feasible to implement for the masses. The keyword being masses there. There all kinds of distributed p2p web projects out there. None of them appeal to a majority though they are all niche products.

Most of these proposals forget the need fund marketing, promotion, and scale.

astazangasta2• 11 years ago

Who cares? The web preceded the Internet company. I remember, it was a great place full of interesting people and things, not shitty content mills crammed full of ads. A system like the one described means that ordinary people can build the web again, which is great. Businesses can continue as they are.

rgbrgb• 11 years ago

Ordinary people can and still do build the web. Why would decentralization allow for more of this? It's cheaper and easier than ever to spin up a heroku/do/aws/google/azure instance and put up a website. Things like squarespace and weebly even make it so you don't have to do any programming whatsoever.

In theory I want this decentralized web stuff to succeed but in practice the only killer apps I see are overthrowing governments and kiddy porn. I'd be happy to be proven wrong. From where I stand, decentralization seems like more of a social/product problem than a technical one. If you prove there's a product that end-users want that can't be built or accessed from the current web, people (end-users and developers) will switch.

charbz• 11 years ago

Data synchronization and Memory management is a major flaw in the concept of a distributed web as described by this article. Is the author suggesting taking all application data that exists on all web servers today, and hosting it on each device connected to the network (billions of devices) ?

saintx• 11 years ago

A good roadmap for a new distributed web should be broken down by OSI model layer, showing what protocols and technologies exist that need to be replaced, what levels of the OSI model they span, and identifies single points of failure lower in the stack that must be accommodated. Too few people understand how brittle the web is by its reliance on the "magical" underpinnings of the Internet continuing to "just work".

For example, let's say we want privacy, anonymity and high availability for something fundamental like name lookups. It's not enough to simply replace DNS with namecoin (L7), if there's a critical vulnerability in openssl on linux that could force a fork in the network, possibly leading to existing blocks getting orphaned (L6), if every single session that goes through AT&T gets captured, and the corresponding netflow stored in perpetuity for later analysis and deanonymization (L5), if this application's traffic could be used for reflection amplification attacks (L4) due to host address spoofing (L3). One might try to get around those issues by direct transmission of traffic between network endpoints (asynchronous peer-to-peer ad hoc wireless networks via smartphones or home radio beacons, for example), but then you not only need to deal with MAC address spoofing and VLAN circumvention, (L2) but with radio signal interference from all the noisy radios turned up to max broadcast volume, shouting over one another, trying to be heard (L1) and accomplishing little more than forcing TCP retransmissions higher up in the stack.

And really what's the point, when you can't even trust that the physical radios in your phone or modem aren't themselves vulnerable to their fundamentally insecure baseband processor and its proprietary OS? Turns out, what you were relying on to be "just a radio" has its own CPU and operating system with their own vulnerabilities.

Solving this from the top down with a "killer app" is impossible without addressing each layer of the protocol stack. Each layer in the network ecosystem is under constant attack. Every component is itself vulnerable to weaknesses in all the layers above and below it. Vulnerabilities in the top layers can be used to saturate and overwhelm the bottom layers (like when Wordpress sites are used to commit HTTP reflection and amplification attacks), and vulnerabilities in the lower layers can be used to subvert, expose, and undermine the workings of the layers above them. The stuff in the middle (switches) are under constant threat of misuse from weaknesses both above AND below.

It might be tempting for an app developer to read this blog post and think "Oh wow, what a novel idea! Why is nobody doing this?" But in reality, legions of security and network researchers, as well as system, network, and software engineers around the world toil daily to uncover and address the core vulnerabilities that hinder these sorts of efforts.

sebastianconcpt• 11 years ago

Well in this regard, I'm pretty mindblown by the possibilities of https://ethereum.org/

Titanous3• 11 years ago

I really wish that the Internet Archive would provide bulk access to the Wayback Machine dataset. It would allow for a lot of interesting experimentation and research.

nekopa8• 11 years ago

Is that even possible? I don't know the latest size of the IA, but it must be ridiculously huge by now, (1 billion pages a week added) bandwidth cost would be massive.

Maybe they could offer a mail-us-a-multi-petabyte-hdd service... Returned a few weeks later full of data :)

Titanous3• 11 years ago

It's totally possible, they already have the infrastructure in place and 14PB of data available for download. Unfortunately the Wayback Machine data is not currently exposed publicly.

justin66• 11 years ago

I really wish that the Internet Archive would provide bulk access to the Wayback Machine dataset.

Have you asked them? Did they refuse outright?

Titanous3• 11 years ago

I have not personally asked, but I think the Archive Team has. I don't know the reasons behind the policy.

omouse2• 11 years ago

That's been the goal of the Freenet project for a while, to build a distributed encrypted network protocol. It distributes storage and processing which is why full encryption is necessary; you don't want 10 people reading your email when it's distributed across their machines.

The challenge for Freenet has been speed and fun. To have something like Facebook you have to download a JAR plugin for Freenet that adds that capability. That's not fun. The speed is slow because of the encryption and constant syncing.

It might be better to look at the MediaGoblin and Pump.io (and StatusNet to some extent) for ideas on federated platforms. The challenge there again is fun; it isn't fun to set things up.

heywire• 11 years ago

Am I the only one who is scared to even try things like Freenet and Tor, for the risk that someone will somehow transmit illegal content across my connection? I'm not talking about content piracy, but the types of things that I don't even want to type for fear of having the terms associated with my username.

omouse2• 11 years ago

First...IANAL (I Am Not A Lawyer)

You aren't the only one, but with Freenet it's fully encrypted. Let's say you had a Freenet Silk Road application. You won't know it's a Silk Road web page that's being saved along with images of marijuana to your computer unless you go through an indexer/search site and even then you still won't know that those bits of data are stored specifically on your machine.

So in order for the cops to know your machine was used to store the drug listings, the cops would have to spy on your machine and crack the encryption of the Freenet protocol and essentially monitor it. This is why undercover work is important to the police. If no one reports you for the crime of buying drugs and no one discovers the drugs in transit, then the police don't know what's happening. The only way to catch mobsters was through some undercover work and hoping that someone in the criminal network would squeal. If one criminal says the other 10 criminals actually had a hand in committing a crime, the police have more to investigate and can build a case.

If you're not buying drugs or selling drugs and the data related to the drug listings is encrypted when stored on your machine and encrypted when served from your machine, you may be unknowingly helping a criminal to buy or sell drugs. But I'm not sure how that's discoverable by the police and I'm not sure how it would be turned into a criminal investigation. By the argument that you're allowing this, then all ISPs and cell providers are in big trouble because they also enable drug dealing.

There are some horror stories for Tor node operators though.

erkose• 11 years ago

One would turn to twister.net.co, which addresses the "next" features discussed in this article, rather than pump dot io.

nopcode• 11 years ago

Just replacing DNS with a decentralised alternative would be a big step, yet appeared to be an impossible one (zookos triangle).

Something this big requires everyone using the internet to switch to the new system or it will never work, and that will never happen. It's the dancing pig problem.

We can't go back on the decisions that have been made, only go forward.

LukeB423• 11 years ago

I've been thinking of how to decentralise the web as-is since 2011, the current development branch for this perspective on it is here: https://github.com/LukeB42/Uroko/tree/development

It's basically a collaborative caching proxy.

One process is a proxy that can also coordinate multiple users editing the same page and a subprocess acts as a DHT node.

You can use a raft-like log of hashes of pubkey,content and the previous hash to keep a history of edits in the network.

the hard part is this: How do you trust the validity of a singular node having a url you're requesting?

It entails a rating system, and then it becomes the byzantine generals problem where the overlay network should be able to tolerate up to a third of its malicious nodes saying they're all trustworthy.

Feedback/any help would be much appreciated.

notduncansmith2• 11 years ago

So the "log of hashes of pubkey,content and the previous hash" is conceptually similar to a blockchain, and I think reading into how that works (consensus, trust, etc) would give you some insight into the issues you're describing. You may also find the IPFS project of interest: http://ipfs.io/

LukeB423• 11 years ago

I've found IPFS very interesting and reccommended it to peers, but it lacks the collaborative editing aspect.

Trusting the initial public offering of a resource is still an interesting issue. IPFS is content-addressable by hash, addresses map to their content in a computable way.

The idea for the distributed hash table in Uroko is that the keys are existing URLs. Imagine thousands of peers all saying they have a new page on the domain "google.com" and you can see what makes this a fun problem to solve.

dubwubz• 11 years ago

It's too late for that. What was once the internet is now basically glorified cable TV. At this point, it's pretty much inevitable that it's going full Disney or bust.

Hopefully bust.

bobajeff• 11 years ago

I agree with one of the comments on that page that decentralizing the Internet is fundamental to decentralizing the Web.

So in my mind the problems that need to be solved are:

Information-Centric Networking > Unstructured Mesh Networking > Distributed Data Storage > P2P Information Retrieval

acd• 11 years ago

To enable p2p web Use ipfs or morph.is

Animats2• 11 years ago

Kahle's approach works only for static content. It's not hard to distribute static content; BitTorrent does it just fine. The Internet Archive stores static content. Kahle thinks in terms of static content, because that's what the Internet Archive does. But it's less of the Web today. Despite that, it's good to have a way to distribute static content. Academic publishing, after all, is almost all static content. That should be widely distributed. It's not like academic journals pay their authors.

There's the problem that distributing content means someone else pays for storing and serving it. This is part of what killed USENET, once the binary groups (mostly pirated stuff and porn) became huge. There's a scaling problem with replication.

Federated networks are interesting, and there are several federated social networks. A few even have a number of servers in two digits. You could have a federated Facebook replacement that costs each user under a dollar a month at current hosting prices. No ads. The concept is not getting any traction.

Kahle wants a system with "easy mechanisms for readers to pay writers." That's either micropayments or an app store, both of which are worse than the current Web.

gglitch• 11 years ago

Why are micropayments worse than the current web? People have differing opinions on the advertising-pays-for-content-so-don't-block-it issue, but are you referring to something technical?

Animats2• 11 years ago

No, just the general failure of the pay-to-read model. Other than the New York Times, the Wall Street Journal, and the Economist, few general publications with a paywall make money. They all have large, worldwide reporting staffs. Nobody is going to pay to read your blog.

Pando Daily is trying pay-to-read. It's too soon to tell how that will work out.

dredmorbius8• 11 years ago

Multiple reasons, though many boil down to Gresham's Law or similar: it's difficult to assess the quality of information, particularly when disaggregated. Most media advances have occurred through bundling rather than unbundling options: magazines, books (collected parchments, monthly serials), subscriptions. Even advertising-supported broadcast and Web models work by aggregating product, though in this case, eyeballs sold to advertisers.

See:

"Why Information Goods and Markets are a Poor Match" https://www.reddit.com/r/dredmorbius/comments/2vm2da/why_inf...

Nick Szabo: "The Mental Accounting Barrier to Micropayments" http://szabo.best.vwh.net/micropayments.html

Jacob Nielsen tries to make the opposite case. He's wrong. "The Case For Micropayments" http://www.nngroup.com/articles/the-case-for-micropayments/

I see a mix of some advertising, patronage, and a content syndication system similar to the existing performance payment model for music (broadcast, commercial establishment use) via ASCAP and the Harry Fox agency as most likely: https://www.reddit.com/r/dredmorbius/comments/1uotb3/a_modes...

See Phil Hunt's UK proposal: "A broadband tax for the UK?" http://cabalamat.wordpress.com/2009/01/27/a-broadband-tax-fo...

substack• 11 years ago

There are extensions to distributed protocols like bittorrent that are already deployed to address mutable, non-static content. The approaches I know of address content under the hash of the public key. One of these approaches is http://bittorrent.org/beps/bep_0044.html and ipfs supports this technique too.

If you have a single mutable pointer, you can build a feed of data that points at immutable content by its hash, which could replace the data model of twitter, facebook, or many other social networking web services. The benefits to decentralized distribution are huge: native offline functionality, trivially transferable identity, longevity and robustness against providers shutting down, direct commerce without middlemen.

Payments, or perhaps ISP-style peering arrangements may help with the spam/large binary problem. A big part of distributing the data model will also involve distributing the costs, but this is somewhere non-profits like the Internet Archive can play a very important role.

doublerebel• 11 years ago

There absolutely should be distributed short-form knowledge. Only metadata needs to be exposed in order to find small facts. I don't think there is any need for payment per-datum for such knowledge. This kind of static content is simple to trust as well.

Long-form knowledge, in contrast, still has massive value and many channels exist for distribution and consumption already.

If we base the web of trust on facts combined with people then we can walk the short web and exit at the long-form.

However nobody wants to create a web of trust based on a system that encourages selfies and ephemeral knowledge -- we've tried that. And this is where things get interesting.

schoen2• 11 years ago

I'm happy to see this article, and it reminds me of things that others have been talking about for some time (for example, the "Redecentralize" community).

I've participated in some file-sharing litigation which has made it very clear to me that decentralized P2P systems are not inherently more anonymous than other technologies. In fact, there's a cottage industry of P2P monitoring companies that participate as peers in the P2P networks and record detailed information about the IP addresses of peers that uploaded and downloaded particular files. There are often paradoxes where decentralization helps privacy and anonymity in some ways but harms it in others -- for example, if you run your own mail server instead of using Gmail, then you've prevented Google from knowing who communicates with whom, but allowed a network adversary to learn that information directly, where the network adversary might not know the messaging relationships if everyone on the network used Gmail.

I guess a related point is that information about who is doing what online exists somewhere by default, unless careful privacy engineering reduces the amount of information that's out there. Making the simplest kinds of architectural changes could just shift the location where the information exists, for example from Google or Yahoo or Amazon to dozens of random strangers, some of whom might be working for an adversary.

api2• 11 years ago

The only mechanism I'm aware of that truly allows anonymity over your own connection (or a connection that can be tied to you) is onion routing. On top of that, you must do it from a separate device or isolated VM to prevent hardware fingerprinting.

Anything less than that is like using snake oil crypto: it might make you feel good, but it's not really there.

phkahler• 11 years ago

> The only mechanism I'm aware of that truly allows anonymity...

We have a need for both solid anonymity and zero anonymity. I think the first step is to be able to authenticate whom you are communicating with, and to reach them without a central authority. After that, you can choose to strip identifying information, or build a web of trust, or anything else. I think privacy can be built on top of an authenticated net, but the reverse is probably not possible. Today we have neither.

schoen2• 11 years ago

Onion routing is an anonymity mechanism for low-latency communications; there could be other mechanisms that are as good or better for some settings of high-latency communications.

https://en.wikipedia.org/wiki/Mix_network

bsder• 11 years ago

Anything less than that is like using snake oil crypto: it might make you feel good, but it's not really there.

While technically true, it doesn't help the situation.

Against the NSA, yeah, you have to be perfect. However, most adversaries are not the NSA.

Encryption on the wire stops random eavesdropping on you while someone else is a target. Having your mail store on a colocated box instead of Gmail/Hotmail/Yahoo means that someone has to get a warrant and physically access your machine rather than filling in an automated request and having it turned over.

It's a modification on the old joke: "Sure, if the tiger is after me, I have to outrun the tiger. But if the tiger is simply hungry, I just have to outrun you."

dredmorbius8• 11 years ago

For email there are various mixmail systems I strongly suspect you're far more familiar with than me.

A recent talk (I don't recall the conference) on de-anonymising anonlymous online communications shows sharp limits to even this, though there is some workfactor required. Better than nothing.

gritzko• 11 years ago

Back in 2008 I was studying P2P networks. I've made a BitTorrent crawler by duct taping v8 and libevent (there was no node.js at the time). It took about 5 minutes to scan a fresh Dexter swarm of about 100K peers. Then, I had all the IP addresses and plenty of metadata (download progress, software used, etc)

Sleaker• 11 years ago

I was liking what the article was suggesting but then it shamelessly plugged BitTorrent inc. which is one of those 'big companies' you don't want touching anything related to privacy or freedom.

jokoon• 11 years ago

Cool, that's a nice way of promoting those technologies since many don't understand them.

I wish those things would land on the IETF board. I wonder what snowden think about those. I would surely make things much harder for the NSA to do massive surveillance.

Locking the Web Open: A Call for a Distributed Web

Related Stories