Show HN: Nebula – A network agnostic DHT crawler

https://github.com/dennis-tra/nebula

by dennis-tra • 2 years ago

68 22 2 years ago

github.com

mikae1• 2 years ago

Unlucky naming collision with Slack’s networking tool Nebula: https://github.com/slackhq/nebula

rad_gruchalski• 2 years ago

Oh no, what of an unfortunate event. The slack tool uses the name already used by OpenNebula. /s

Nux• 2 years ago

Not to mention they jinxed Slack(ware) for many.

sph• 2 years ago

And the Nebula streaming platform. Which is unfortunate because I'm using both.

I get it, nebulae are cool.

doublerabbit• 2 years ago

Not forgetting the awesome Nebula game engine

pdabbadabba• 2 years ago

I'm sure that this just because I'm not the target audience, so intend only the very gentlest criticism. But I literally LOLed at how completely incomprehensible this README was for me. It has really been a while since I've read a paragraph and had literally no idea what it was talking about. But here's the winner:

A network agnostic DHT crawler and monitor. The crawler connects to DHT bootstrappers and then recursively follows all entries in their k-buckets until all peers have been visited.

Following the Wikipedia link for "DHT" yielded some clues. (Ah. Distributed hash table.) But I've still been looking at this for several minutes now and am basically just puzzled. But the graphs are pretty! Reading the word "amino" a little further down threw me off the scent for a bit. But I gather that is actually a proper noun, and we aren't really talking about proteins here.

Maybe an initial sentence that makes fewer assumptions about the reader's familiarity with the jargon would be helpful.

Chabsff• 2 years ago

This is not a particularly egregious example, but it's kind of spectacular how everything crypto adjacent revels in technobabble.

The detractors of the ecosystem (myself included, to be honest) will be quick to point out that obfuscating the tech as magic as much as possible, as well as creating an inside group lingo, is key to onboarding and retaining people into it. But it's fascinating how that percolates throughout the dev community behind it as well.

DanAtC• 2 years ago

I feel the same way about AI/LLM lingo

PedroBatista• 2 years ago

While I agree, there is a whole generation of people who know what a DHT is, it's not really that obscure.

I'm talking about of course very late 90s, 00's P2P file sharing, Kademlia, torrents but also later "eventually consistent" databases ( remember those? )

The crypto 20 somethings hype beasts came way later.

nodja• 2 years ago

A DHT is a decentralized key-value database, it's most famous use being in the bittorrent protocols, it uses a routing algorithm to guarantee that you can find the peers that can retrieve the value of a known key, granted that you at least know one peer in the network (even if that peer doesn't know the value). Essentially the network is split into buckets and it guarantees that you'll either will be already connected to a peer that knows the value for the key, or that that peer knows a peer whose bucket is closer to the key, you can then recursively ask for peers that are closer and closer until you find one that knows the key, as you do this search you keep track of the peers so the next time you ask for another key you're more likely to know a peer that is closer to it. A typical DHT implementation has you keep track of hundreds of peers to guarantee the robustness of the network.

One issue is that peers go offline and online all the time, so the network is ever changing, if you turn off your client for a week and then come back, your only hope is that at least one of the peers you know is still online, if that's the case then that's fine, if that's not the case, or you're starting the client for the first time, then there's no way for you to connect to the network and query for keys. In bittorrent this is not an issue as most torrents include trackers, the original centralized way of finding peers on the network, but it seems that each project listed on this page has it's own separate DHT network that doesn't connect to the main network (the one used by bittorrent), so for you to connect to these networks for the first time you need to use a bootstrap peer, this is just a normal peer on the network that is known to be always online, usually hosted by owner of the project, and it'll give you a starting point to find other peers in the network.

What this project does in essence, is connect to a bootstrap peer, then use the properties of the routing algorithm to efficiently find out all the peers that are currently online.

smoyer• 2 years ago

The Kademlia DHT paper is a pretty straightforward description of a DHT - https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia...

Hanschri• 2 years ago

BitTorrent's BEP regarding DHT also explains how it works quite clearly. The implementation is based on Kademlia and can be found here: https://www.bittorrent.org/beps/bep_0005.html

worksonmine• 2 years ago

When the choice is to either quickly explain why a potential user should use your project or teaching someone without domain knowledge what the project is I would choose the former every time.

The part you quote even provides links to DHT and k-buckets if you want more information. That first paragraph is important real-estate and shouldn't be wasted on someone who most likely has no use of the project.

Every domain has these words that sound alien to newcomers and you can't explain them to everyone all the time.

dTal• 2 years ago

Why is BitTorrent not supported? Perhaps I'm misunderstanding this thing but it seems like application #1.

DanielVZ• 2 years ago

Because this seems to cater to the cryptocurrency/blockchain culture.

ogurechny2• 2 years ago

My guesses:

a) Many other tools exist for that.

b) Bittorrent DHT modes are simple and interchangeable. They can give you a list of peer addresses associated with certain (torrent) hash — and only if you know the exact hash. Even client versions can't generally be collected (apart from some protocol extensions). The only thing you learn about DHT member is that it exists. On the contrary, this project is for heterogeneous networks in which peers announce various services.

c) Number of Bittorrent DHT nodes is… bigger.

d) To collect interesting data from Bittorrent DHT, one needs to observe as much third party torrent hash requests as possible. To do that, multiple nodes are needed. Moreover, they need to run for a long time, not just because it takes time to make a lot of requests to a lot of nodes, but also because of external preference for long-running nodes. Not sure how important it is, but, anecdotally, a fresh DHT node sees twice as much requests after a week than after a day.

jzm2k• 2 years ago

Looks like Nebula uses go-libp2p and all of the supported networks listed in the README use libp2p for their p2p networking. Mainline DHT doesn't support the same transport protocols that libp2p supports (such ash TCP+Yamux+Noise) which is probably why Nebula doesn't support Bittorrent

crotchfire2• 2 years ago

Because it isn't really network-agnostic.

It only supports IPFS and derivitaves thereof.

ogurechny2• 2 years ago

/me remembers various DHT views, traffic flows, client stats, graphs and other data decorations in Azureus. Now that's what I call a dashboard.

crotchfire2• 2 years ago

It isn't really network-agnostic... in fact it doesn't support the (by far) largest DHT out there, the Mainline DHT that bittorrent uses.

This is just a crawler for DHTs that use IPFS's implementation, or at least smell very similar to it.

pedalpete• 2 years ago

Can someone explain why we want to crawl and/or monitor? What is this used for?

When I think of a crawler, I think of a non-homogonous network (if that is the right term).

But with the blockchain, isn't it the case that each node has an entire copy of the blockchain, so you don't need to "crawl" it, it works more like a database.

What am I not understanding about this?

Alifatisk• 2 years ago

Instead of everyone crawling on their own, isn't it more efficient if everyone shared the same index somehow?

Show HN: Nebula – A network agnostic DHT crawler

Related Stories