Peer-to-Peer Databases for the Decentralized Web

by fossislife • 5 years ago

261 79 5 years ago

w3clan• 5 years ago

I think https://gun.eco/docs/ does so much cooler job.

voicedYoda• 5 years ago

Why do i always feel things like this and ipfs cone back to blockchain? I think the blockchain is a very interesting tool in a limited capacity. Orbitdb and ipfs seem to capitalize on their relationship to it.

markhenderson4• 5 years ago

OrbitDB doesn't rely on a blockchain at all

encryptluks26• 5 years ago

I don't see how people consider this and IPFS decentralized. You still need peers to subscribe and pin, and services like Pinata aren't cheap and still centralized. Is there an example of websites that would continue to run indefinitely if there wasn't a centralization of services with IPFS? I just don't think it is likely unless you start distributing your app that bundles IPFS, so that your users end up hosting it for you. However, if someone new tries it out and only one of your users has your app open you better hope it is being cached by Cloudflare.

aaomidi3• 5 years ago

What do you think decentralized means? It means that hosting it isn't the responsibility of a single party.

If I put something online, and people think it's interesting and pin it. Even if I go down, the pinned content remains up.

Decentralized doesn't magically solve hosting availability. It just removes responsibility from a single party.

encryptluks26• 5 years ago

I would say there are very few IPFS sites where hosting doesn't still end up being the primary responsibility of a single party. That was the point I was trying to make. If you have to pay someone else to pin it and keep giving them funds then it is still your responsibility. If you have a website that you want to scale and load fast for all your users on IPFS then it will still end up being your responsibility. There may be a few exceptions.

hosh6• 5 years ago

I'm looking at integrating IPFS with sandstorm (sandstorm.io). The sandstorm (and other homelab/homeserver) community have an interest in self-hosting. IPFS allows people that are interested in seeing something stored to pin it themselves. But yes: bundling IPFS with the app is pretty much what would need to happen.

As they are now, neither the IPFS ecosystem or sandstorm are really decentralized -- or maybe I should be more precise: not decentralized in a resilient way. Yet, I think both are great technologies that enable some interesting things.

I'm not as interested in the blockchain notion of decentralized though. I'm looking for resiliency to severe netsplits. As a thought experiment, if Earth crosses a coronal mass ejection event and our power and communication grids go down, would we be able stitch things back together again when our grid fractures?

secfirstmd• 5 years ago

Very cool idea!

pm3• 5 years ago

Curious about the homelab/homeserver community. Are there any sites with further information?

random_kris• 5 years ago

I was also looking at sandstorm. I even managed to install it on my homeserver. But it is sadly not really up to date. As a replacement i found homelabOS which is a bit more complicated (I think).

And I was also thinking about integrating ipfs into my homeserver stack. But instead of plain ipfs I was thinking about sia coin. They offer cloud storage. So I was thinking about to rent 4tb of space. And whatever money I make with it I use it to buy backup from sia.

So in a sense I provide backups to others and in return I get free backup.

Sadly work is being to intensive on last month so I haven't found much inspiration in completing stuff above.

jimpick2• 5 years ago

I've been doing some experimentation along the same lines ... I've got a small proof-of-concept working where I can sync Earthstar databases between multiple Sandstorm instances - each "grain" acts as a pub, and you can use "webkeys" as pubs. I'm using powerbox-http-proxy to request capabilities for connections, so it works even with fully sandboxed client-side networking (ALLOW_LEGACY_RELAXED_CSP=false). It's maddeningly complex to do simple networking, but it's amazing to have the security layer there. I'll put together a blog post / video soonish.

https://github.com/jimpick/sandstorm-earthstar-foyer

Sandstorm is very interesting for privacy-focused peer-to-peer because it has the tools in place to sandbox networking. The networking security model is designed to prevent things from talking to each other without user approvals - so it's extra tricky to do peer-to-peer things with it.

I'm excited to try to stick some of my old IPFS/IPLD/libp2p/Dat/CRDT things into Sandstorm - there's a tonne of interesting things that could be done with it.

markhenderson4• 5 years ago

Swarm persistence is hard, if not close to impossible. Better that multiple parties pin the data than just one though. It's a step in the right direction.

encryptluks26• 5 years ago

Agreed. I started looking into setting up cluster followers, but currently you have to have enough space to pin everything and can't have it only do a percentage.

t3rr0r3• 5 years ago

I share your viewpoint and frustration with "decentralized" apps. But as you said it is indeed possible by bundling your app with IPFS and using something like orbitdb/ipfslog for structuring data. I linked this in my other comment, but take a look at this proof of concept: https://www.reddit.com/r/musichoarder/comments/lrqx7m/record...

At this point the major challenges are with UX, especially for users who are not accustomed to self-hosting. I think we have the technologies and ideas to overcome the rest.

bawolff• 5 years ago

I think that's the big problem with the "decentralized" WWW movement. There are about a billion conflicting definitions of decentralized providing very different properties, and it's never clear which definition people mean.

saurik• 5 years ago

It is decentralized in that anyone can step up and host the content on IPFS and it is considered equivalent and secure. With a normal website, if the canonical host goes down, someone else can host it, but it isn't the same website.

Geee• 5 years ago

IPFS, torrents and many old-school p2p-networks are decentralized to an extent, but they're also non-sybil-resistant and voluntarist, which makes them unsuitable for many use cases. They're not really a part of what is meant by decentralized web.

You should take a look at SkyDB to get an idea what's possible with true decentralization: https://blog.sia.tech/skydb-a-mutable-database-for-the-decen...

amw-zero• 5 years ago

I feel like you don't understand the definition of "peer to peer."

The fact that you require nodes to ultimate do the hosting of data does not make IPFS centralized.

opendomain• 5 years ago

This project may be dead

There has been no activity in over a year,and the sponsoring company has no presence in several years.

ampdepolymerase• 5 years ago

Gun DB is a good replacement.

https://gun.eco/

markhenderson4• 5 years ago

It's not dead, just in a bit of a lull due to funding, which is hopefully coming soon. We have a very active Gitter and OpenCollective.

Source: I'm one of the core maintainers

octopoc3• 5 years ago

Is there any interest in supporting other languages? Like how libp2p is being gradually ported to Rust, Go, Java. I feel like OrbitDB could become the new data model of the web if it went multi language. Or maybe was compiled to WebAssembly and embeddable using Wasmer.

t3rr0r3• 5 years ago

It's def not ded, it's been like this for as long as I remember but it just keeps slowly chugging along. I continually find myself coming back to it, especially as IPFS matures more.

hartator• 5 years ago

I wonder if it would be possible to build a decentralized search engine with a decentralized crawl on top of it.

hosh6• 5 years ago

I've been wondering the same thing too! Maybe if there is a way to share indexes, or some way to put out a call for searches through the DHT, and then the requestor aggregates the results.

A couple examples:

Web crawler -- seeded through a curated or personal bookmark list.

Knowledge base -- the permaculture community have amassed a knowledge base on growing plants: USDA hardiness zones, heat zones, humidity, soil requirements, companion planting relationships, etc. That's something I can see people adding on to, particularly with non-standardized homegrown cultivars and how well they do in their local area.

beaconstudios• 5 years ago

Limewire had an interesting protocol for exactly this back in the day, over its Gnutella network. I don't know if it scales to search engine sizes (most things don't, especially distributed) but there's at least some prior production grade work in this space.

Edit: having just double checked it looks like Gnutella predates limewire; that's just the avenue through which I came across it.

lovemenot• 5 years ago

> Maybe if there is a way to share indexes...

What does a SE index look like? Any designs in the open?

emaro• 5 years ago

There is one: https://yacy.net/ However it's slow and the last release is from 2016.

lazzlazzlazz2• 5 years ago

IPFS + (Arweave or Filecoin) behind it seems like the best decentralized web stack I've seen yet. I'm not sure how Orbit holds up.

j0hnM1st• 5 years ago

Arweave has production ready releases ?

lazzlazzlazz2• 5 years ago

Yes. Check it out.

endisneigh• 5 years ago

I wish there was something like this but with Webtorrent - I find torrents easier to grok for whatever reason.

oscargrouch• 5 years ago

I hope i can share something here more on this line pretty soon. Its not just databases and blobs over torrent but a whole web based application platform based on chrome for native applications. The SDK will start with Swift but the C interface can be used by other languages.

oilbagz• 5 years ago

I like this, but I feel its important to point out that this is effectively unusable by the very people who would benefit from it most: non-web developers.

If this were a drop-in C/C++ library, I'd use it immediately in my native apps and adopt it as a widespread platform technology. I want to be able to do this, and if it were truly a working technology, I'd put it in every single app I use.

But as a Node/JS library, it is off limits.

Please, decentralization guys, consider native languages first, and toy languages next. These technologies are never going to be embraced unless they treat native platforms as first-class systems for their manifestation.

christophilus• 5 years ago

What is the definition of a toy language? That’s a new term for me.

I (thankfully) haven’t written C++ in over 20 years, but if I had to drop back down to a low level language, it’d be C or a compile-to-C language. C is the language these things should target, in my opinion.

StavrosK• 5 years ago

What is the definition of a toy language?

A language that doesn't require as much hair-pulling to learn, so you can't feel as smug for knowing it.

kyleishie• 5 years ago

What is the definition of a toy language? That’s a new term for me.

I would imagine most non-compiled, scripting languages. They are ok when building the business logic of an application on top of faster building blocks but not ok for application building blocks such as a database.

bamfloesche• 5 years ago

Systems developer here. I made an account to answer this, because the OP is 100% right about this.

A toy language is one which you do not allow to escape the play-room/sandbox, you play with it (because hey, you can, so why not..), but you would not actually use it.

The reason this is important for decentralization efforts, is that the OS should be doing this.

Sure, paint the pretty UI, but expose the entire thing to common OS-level abstractions, please, and then host the scripts, kthxbai ..

(A non-toy language is one you are very comfortable shipping. Javascript is that for a lot of people. Not for systems programmers = it is an application execution environment, not a place for things which must operate, as part of the system, without much user hassle ..)

loxias2• 5 years ago

Amen to this! I too got really excited until I saw it was JavaScript. For me to use it I'd have to link against some sort of JavaScript interpreter (ew) and probably incur performance penalties shoveling the data too and fro.

Can't we just have nice, normal libraries like we used to? Tools that require me to install a huge new build system, package management (cough golang cough) just to run quickly lose their appeal.

MsMowz• 5 years ago

The answer is the same as it always was: you'll have "normal" libraries when somebody writes them (it could even be you). Most new application development relies on web servers, where JavaScript is very relevant and normal.

antman• 5 years ago

I would play with some Python.

loxias2• 5 years ago

Me too! But the thing is, a vanilla C library can be used by anything. It's the lowest common denominator. I can easily call into a C library from Python, Perl,... MATLAB... and I don't know JavaScript but I bet it's not hard from there either. That can't be said about any other language.

Performance is oft cited as why systems software is written in C (or C++ with C bindings), but I think a much stronger reason is the universality. C runs on everything from big iron to the smallest microcontroller.

feldrim• 5 years ago

Something like https://dqlite.io/ would fit your needs? It's distributed thanks to Raft with all consequences. It's not viable to call it P2P as it lacks node discovery et cetera. Yet, it's distributed.

steeve• 5 years ago

There is a Go implementation available https://github.com/berty/go-orbit-db

curtisblaine• 5 years ago

orbitdb runs on browsers, via WebRTC. It's much easier to embrace something that runs on a web page than something that you have to download and run.

mab122• 5 years ago

An interesting project is https://github.com/dappkit/aviondb which is mongo-db like database on top of orbitdb

Peer-to-Peer Databases for the Decentralized Web

Related Stories