Crawling BitTorrent DHTs for Fun and Profit [pdf]

https://www.usenix.org/legacy/event/woot10/tech/full_papers/Wolchok.pdf

by dgellow • 1 month ago

117 49 1 month ago

(2010)

hdgr• 28 days ago

Bitmagnet -https://bitmagnet.io/ - does exactly that. I left it running for a few weeks and then stopped the crawler. Didn't expect much, but still somewhat disappointed by the garbage it reeled in.

NegativeLatency3• 28 days ago

I've had one running for over a year now, it's replaced my usage of regular torrent sites completely, there is a lot of junk, and it gets stale, but it's still a better experience than most of the public trackers out there IMO

drdexebtjl• 28 days ago

I disabled mine because it was constantly writing to my SSD.

qingcharles6• 28 days ago

Are you running it at home?

I built one with a nice TUI to run on a VPS so I can try and find rare magazine torrents, but Hetzner were upset about it. I need to find it a new home. It was a very good citizen, but it still raised too many flags.

plusfour3• 28 days ago

Same here, for over a year. how many torrents has yours indexed?

gonzalohm• 28 days ago

How much space do you need to store the index?

qingcharles6• 28 days ago

This is a good tip, thanks. I'll probably replace my home-grown scanner for this one.

muyuu2• 28 days ago

probably worth adding some ML filter to it because yeah, most of the bulky stuff in bittorrent is always going to be garbage - a lot like the internet generally, the value is in filtering the good stuff out

infinite_spin• 28 days ago

out of curiosity, what kind of junk/garbage is typical?

naikrovek2• 28 days ago

why the hell does that require a postgres database instance? has no one heard of sqlite?

"It's just `docker compose up` dude." that's not excuse for unnecessary complexity. software for a single person should be a single binary that anyone can run how they choose. If they want to use Docker, great. If they want to run it in a terminal, great. If they want to run it on a server, great. If they want to run it on their phone, great.

developers: avoid limiting people artificially like this. There is no way that Sqlite is insufficient for this.

knowaveragejoe2• 27 days ago

The database gets quite large and the developers put in a lot of effort for tuning it for searchability.

gritzko• 28 days ago

2010. I remember those times. I was doing these things for science in 2008. Performance-wise, PEX was much faster than DHT. At least, in my setting.

This year, I was giving it as an assignment to students. Does not take much time with LLMs.

Boss0565• 28 days ago

old paper

MoonWalk4• 28 days ago

The article neglects to define "DHT" before using it.

ivanjermakov• 28 days ago

Distributed hash table - ButTorrent extension for discovering torrent's seeders by advertising its hash across known peer pool, think of it as a distributed tracker. Contrary to traditional way of asking a known tracker for peers of that torrent.

Its algorithm is very elegant, using binary search on peers' and torrents' hashes, narrowing down to peers that are more likely to be seeders (or at least know some).

https://www.bittorrent.org/beps/bep_0005.html

loeg• 28 days ago

Not a P2P innovation with Bittorrent, FWIW. Kademlia DHT (used in eMule/LimeWire/Gnutella P2P networks) long predates Bittorrent.

MoonWalk4• 27 days ago

Thanks!

drbscl• 27 days ago

The BitTorrent community has responded to these trends by deploying decentralizing tracking systems based on distributed hash tables (DHTs).

No it doesn't. Before you try to correct me, the abstract never counts ;)

MoonWalk4• 27 days ago

Downvote all you want, but a fact is a fact: The article doesn't define the abbreviation before using it.

dgellow• 27 days ago

I'm not sure why you're saying that in the HN comments. As far as I know the authors aren't there, and the paper is from 2010. It's not like it will get updated now.

One search tells you what DHT in the context of wikipedia is: https://en.wikipedia.org/wiki/Mainline_DHT

the8472• 28 days ago

Crawling has been somewhat simplified with BEP 51

https://bittorrent.org/beps/bep_0051.html

CommanderData• 28 days ago

I saw mention of Sybil's, hasn't this been patched effectively making techniques of quick discovery though many DHT nodes much more expensive and slower.

You would need a unique IP to overcome this per n nodes. That could be tens of thousands of IPs.

Crawling BitTorrent DHTs for Fun and Profit [pdf]

Related Stories