Ask HN: Recommendations for Peer to Peer Frameworks?

https://news.ycombinator.com/item?id=16785878

by zilchers • 8 years ago

276 75 8 years ago

news.ycombinator.com

I'm looking for peer to peer framework recommendations to build a distributed app. In particular, a framework that supports seeds and peer discovery, message passing, maybe firewall / UPNP support, etc.

I'm not particular about language (Java, Go, Javascript, etc), since it's for a new project. I found this thread from 2016 (https://news.ycombinator.com/item?id=11145691), it had some interesting recommendations, but quite a few were out of date, and with the big push to distributed apps nowadays, I thought it might be a good time to readdress the topic.

Some of the solutions form the other thread or that I've found:

- Smokesignal (Javascript / Node, not updated in >10 months) https://github.com/marcelklehr/smokesignal

- ZeroTier: https://www.zerotier.com/product-netcon.shtml

- Maki: https://maki.io

- Telehash: http://telehash.org/

- A whole host of blockchain projects, Bitcoin, Ethereum, HyperLedger, Scorex (Scala blockchain)

Bitcoin is an interesting option, it certainly has a large user base and is well tested and secure code, but forking it has it's own set of issues.

Dowwie2• 8 years ago

How about IPFS? OpenBazaar is an IPFS case study. They recently released a revamped v2 built with Go.

zilchers4• 8 years ago

That's interesting, I hadn't thought about IPFS - it doesn't really have a strong message passing semantic so much as it has good replication properties, but maybe you could build a distributed log on top of it.

jjallen5• 8 years ago

By good "replication properties" you mean "having to run many scripts/nodes yourself" to get widely replicated that is true.

We found IPFS in general to be awesome software that just wasn't quite ready for production.

There's IPFS-cluster[0] but that's still in capital-letters ALPHA.

https://github.com/ipfs/ipfs-cluster

jjallen5• 8 years ago

They are working on this: https://github.com/orbitdb/ipfs-log

xendipity• 8 years ago

Yeah, I'd look at IPFS's networking library, libp2p. It's the peer-to-peer backbone of IPFS and has a big community and active development https://libp2p.io/. It's also implemented in both go and js (as most of IPFS is)

eeZah7Ux• 8 years ago

The wire protocol is practically almost not documented.

jjallen5• 8 years ago

We moved away from libp2p at the project I work on because of major instability problems in browsers. Our dapp that used libp2p would crash after a few minutes and was generally not easy to use.

EDIT: I'm referring to the JS version here.

jjallen5• 8 years ago

I'd love to read technical details about what they built with it and how they manage replication. Will look for some and repost if I find.

lgierth5• 8 years ago

Look at the IPFS Pubsub and Y.js integration, that e.g. PeerPad uses to build distributed text editing: https://github.com/ipfs-shipyard/peer-pad/blob/master/docs/A...

adamnemecek2• 8 years ago

Look into crdt it’s the shit for collaborative editing (idk if that’s what you ares looking for tho).

malloryerik• 8 years ago

Do you happen to know of a decent and easy-ish crdt library in Go?

adamnemecek2• 8 years ago

Check github there’s plenty.

zilchers4• 8 years ago

CRDT is definitely a good pattern for the data passing itself, but one of the really tough parts of these distributed apps is the plumbing work, just getting peers to reliability communicate. That's part of what I'm looking for.

billylo• 8 years ago

I am using https://hypelabs.io/

Pretty good for small payloads.

rauljordan• 8 years ago

Look into libp2p created by the Protocol Labs team (created IPFS). We're thinking of moving to this framework internally at Ethereum.

TACIXAT• 8 years ago

Second vote for libp2p. It looks very mature and generic enough to build whatever off of it. It has implementations in js and go.

If you end up rolling your own, I found js to have the most building blocks (pmp and upnp libraries).

Dowwie2• 8 years ago

There's an implementation in Rust: https://github.com/libp2p/rust-libp2p

fro0116• 8 years ago

Can anyone point to some higher-level libraries or open-source apps built on top of libp2p?

no_identd5• 8 years ago

We're thinking of moving to this framework internally at Ethereum.

Why not GNUnet? Check the recently released 181 pages long dissertation on it:

https://grothoff.org/christian/habil.pdf

no_identd5• 8 years ago

Link broke, here's a new one:

https://pdfs.semanticscholar.org/153d/578869863450766a870727...

Previous Hacker News discussion here: https://news.ycombinator.com/item?id=15877908

tlrobinson2• 8 years ago

GPL license is probably a non-starter.

misrab• 8 years ago

definitely the most promising out there

0XAFFE• 8 years ago

Maybe Scuttlebutt? https://www.scuttlebutt.nz/

Scuttlebutt is advertised as a social network, but in core it is an eternal distributed log. You can pass either public or private messages and assets around.

zilchers4• 8 years ago

This is a great suggestion, have started looking into it. Thanks!

freshhawk• 8 years ago

Scuttlebutt and Dat (https://datproject.org/) are definitely looking like good bets right now.

yoshuaw• 8 years ago

Also perhaps fun to note is that we're currently hacking on a Rust version of Dat. Hopefully we'll be able to bring the Dat protocol to more languages and platforms soon! https://github.com/datrs

DyslexicAtheist• 8 years ago

Kademlia DHT:

https://en.wikipedia.org/wiki/Kademlia

https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia...

petegordon• 8 years ago

I’m curious what your goal is. Do you want the complete immutability of transactions like bitcoin/blockchain; or do you want the peer/seed capability like BitTorrent and IPFS with a DHT (Kademlia)? I’d be interested in seeing a good list of the technology choices that can make up a P2P project if anyone knows of one, please share.

But you definitely should checkout IPFS and Kademlia.

dnomad• 8 years ago

I2P works well enough.

mcjiggerlog• 8 years ago

Maybe check out Dat[0] - especially useful if you need some kind of distributed data store. The main implementation is in Javascript.

[0] https://datproject.org/

robotmay3• 8 years ago

I really wish there was an additional implementation in something like Rust (personal preference, but C would be good as it's fairly easy to bind to) for DAT, because what they're doing is rather cool and the experience is tidy.

jondashkyle• 8 years ago

That’s in the works: https://github.com/datrs

indescions_2018• 8 years ago

Awesome List of P2P resources:

https://github.com/kgryte/awesome-peer-to-peer

Personal bias tilts to web tech. WebRTC will require you to centrally implement all your peer broker and logic. But take a look at something like Peer5 Video CDN to appreciate scalability:

https://www.peer5.com/

inglor2• 8 years ago

Hey, Benji from Peer5 here - our product is really cool and helps you scale your video delivery but I don't believe it works in this case.

We solve the problem of distributing HLS/DASH delivery over large p2p swarms and scaling video. Doesn't sound like it could work for OP.

Thanks a lot for keeping us in mind though :)

adam_ellsworth• 8 years ago

Hey Benji. I saw you guys setting up at NAB yesterday. Would there be a good day this week to come by and talk shop?

cjbprime• 8 years ago

WebTorrent.

wslh• 8 years ago

Re blockchain technologies: while there is an obvious intersection between the world of blockchain and P2P, blockchain technologies are slow because BFT adds many requirements to the game. Basically blockchain technologies use P2P to reach a common state and this could not be necessary in your use case.

You can take a look at OpenBazaar and Particl for P2P implementations.

no_identd5• 8 years ago

The 181 pages long GNUnet dissertation very recently got released: https://grothoff.org/christian/habil.pdf

I'd strongly recommend going for GNUnet, nobody takes me serious when I say this, but it WILL surplant IP eventually.

woah• 8 years ago

Link broken.

no_identd5• 8 years ago

https://pdfs.semanticscholar.org/153d/578869863450766a870727...

Previous Hacker News discussion here: https://news.ycombinator.com/item?id=15877908

fouc• 8 years ago

http://web.archive.org/web/20180327164208if_/https://grothof...

Ixiaus• 8 years ago

I agree. I think gnunet gets a lot right.

yid• 8 years ago

but it WILL surplant IP eventually.

The painful move from IPv4 to IPv6 suggests that this is unlikely. More likely is an overlay over IP, TCP, or even HTTPS.

no_identd5• 8 years ago

The painful move from IPv4 to IPv6 suggests that this is unlikely. More likely is an overlay over IP, TCP, or even HTTPS.

GNUnet can function as an overlay and an _underlay_ simultaneously, quoting from the dissertation:

GNUnet is an overlay network, as it is initially supposed to primarily operate over the existing Internet network. However, GNUnet does not assume that this is always the case. For example, GNUnet can also operate directly over WLAN or Bluetooth to create an ad-hoc wireless mesh network, and with the GNUnet “PT/VPN” subsystems we can run TCP/IP over GNUnet. So in this configuration, GNUnet would be an underlay network, a bit like MPLS or B.A.T.M.A.N. [NAL07]. In reality, we in fact expect to see both at the same time: some peers will run over the existing Internet, while others may connect to GNUnet on Layer 2.

hapnin• 8 years ago

Blockstack looks like it would fit here:

https://github.com/blockstack/blockstack/blob/master/README....

abreckle• 8 years ago

+1 for Blockstack

martindale• 8 years ago

Maki 0.3 will include components from Fabric: https://fabric.fm — automatically available to anyone upgrading from 0.2. :)

tribler• 8 years ago

supports seeds and peer discovery, message passing, maybe firewall / UPNP support, etc.

Our audacious 'IPv8' work is now deployed live. Might fit you needs. Includes above features in a low-complexity approach. https://github.com/Tribler/py-ipv8 Our fresh IETF Internet Standard draft of this work: https://tools.ietf.org/html/draft-pouwelse-trustchain-00

smaddock• 8 years ago

If you plan on having any collaborative editing features, Yjs [1] is a p2p framework utilizing CRDTs with support for IPFS.

[1] https://github.com/y-js/yjs

mempko• 8 years ago

How about something [completely different](http://firestr.com)

alexmorley• 8 years ago

Maybe checkout the dat project https://datproject.org/ which is whats used by BeakerBroswer etc.

JamesLeonis• 8 years ago

The WebTorrent Project [1] has a whole suite of tools for decentralized applications based around BitTorrent. It's not a framework per-se, but it breaks up the protocol into libraries for your use.

If you're interested in Ethereum, it has other non-blockchain technologies that you can use for your apps. It has the Swarm network for storing bits of data, the Whisper network for chat and messaging, as well as Public-Key signing to verify messages. I'm using the latter functionality for a demo to replace password as the security mechanism.

[1] https://github.com/webtorrent

monocasa• 8 years ago

The virtual synchrony protocols don't get enough love. Conceptually, they're sort of like if you performed PAXOS to determine ordering for a distributed finite state machine. Let's you build all sorts of distributed applications where you trust the nodes.

https://en.wikipedia.org/wiki/Virtual_synchrony

http://www.jgroups.org/

https://archive.codeplex.com/?p=vsync

shpx• 8 years ago

https://urbit.org/docs/

The idea with urbit (amongst others, it's a big system) is for everyone to have a personal server that's easier to administrate than a linux server. It's been posted a bunch of times https://hn.algolia.com/?query=urbit&sort=byPopularity&prefix...

I've also heard about https://zeronet.io/ and https://beakerbrowser.com/

tomp2p• 8 years ago

TomP2P (https://tomp2p.net/) is a library handling firewall / UPNP support, however, the focus is academic/experimental.

sova• 8 years ago

ZeroMQ http://zguide.zeromq.org/py:all ... "In One Hundred Words: ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. ZeroMQ is from iMatix and is LGPLv3 open source."

mayamatrix• 8 years ago

Worth having a look into the Holochain framework:

https://holochain.org/

Depending on the application(s) you have in mind.

PkBuzios• 8 years ago

Holochain is interesting for having a different approach then the blockchain-based frameworks, which are inefficient

jeswin• 8 years ago

Secure Scuttlebutt. It's probably the only one seeing everyday use to get things done. Patchwork, the flagship app is as good as Facebook/Twitter. Scuttlers often use git over SSB, even.

What SSB gets right and the others don't, is the incentive to mirror or replicate. Virtual currency is a weak incentive in an early stage network - but friendship isn't; you mirror your friends' data.

The basic API to program the dapp can be seen here - http://scuttlebot.io/ But for more advanced uses see https://youtu.be/f_baWUW4R8Y

w3news• 8 years ago

Check this great talk: https://youtu.be/izQFMADw70w

devxpy• 8 years ago

I recently discovered zyre. Its from the same guy that brought us the wonderful zeromq library

While it does a lot of the connection and communication part, it lacks a way to bypass nat routers.

Here is a book on zyre http://zguide.zeromq.org/php:chapter8

I also found this thing called pwnat which bypasses nat routers without the need for UDP hole punching !

https://github.com/samyk/pwnat

I feel like if someone made a high level wrapper combining these two, we can have a very good framework.

povilasb• 8 years ago

Another project you might find useful, that we at https://www.maidsafe.net are actively working on, is https://github.com/maidsafe/crust/.

It's a generic peer-to-peer networking library written in Rust.

  * it connects two peers together
  * various NAT traversal techniques are implemented: hole punching, UPnP/IGD, etc.
  * it supports both TCP and UDP. In case of UDP it uses uTP: http://www.bittorrent.org/beps/bep_0029.html
  * all messages are encrypted
  * it implements automatic peer discovery on LAN
  * it caches it's previous connections and is able to reuse them in the future
  * etc.

doomjunky• 8 years ago

Peer Name Resolution Protocol (PNRP)

- Developed by Microsoft

- Open Specification

- DHT-based

- IPv6 only

- Windows support since Windows XP SP 3

- .NET support since .NET Framework 3.5 (System.Net.PeerToPeer)

[1] https://en.wikipedia.org/wiki/Peer_Name_Resolution_Protocol

[2] https://msdn.microsoft.com/en-us/library/aa371699.aspx "PNRP Namespace Provider API"

[3] https://msdn.microsoft.com/en-us/library/bb726971.aspx "Peer Name Resolution Protocol"

zilchers4• 8 years ago

Microsoft had a few cool things, but it all seemed .net specific. Check out this in WCF https://msdn.microsoft.com/en-us/library/cc297274.aspx

Ask HN: Recommendations for Peer to Peer Frameworks?

Related Stories