I built an IPFS crawler with libp2p

https://adlrocha.substack.com/p/adlrocha-how-i-built-an-ipfs-crawler

by adlrocha • 6 years ago

116 19 6 years ago

adlrocha.substack.com

carapace• 6 years ago

So this is crawling the nodes not the content?

mirimir• 6 years ago

Seems so.

There is https://ipfs-search.com/ [https://github.com/ipfs-search/ipfs-search].

kickscondor2• 6 years ago

Was hoping there was a link to the results - either a search box or some statistics. But hey - free code. Not complaining.

Also, if you haven't checked out Beaker Browser now that it's nearing 1.0 - great things are happening. This hasn't been broadcast much - but Beaker now has a built-in editor pane and forking mechanism. It's the best shot I've seen at a 'view source' 2.0.

There's been some healthy competition lately forking and editing the default twitter/mastodon clone. (Which is here: hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac218436f12fda830/ But forks of it are here: hyper://0e0d1685c3575cf1df94191e14b4ecf068c43bcefee787ffbbed3d1f4666cab4/ and here: hyper://b798b86cc282ae065697f36a2a07072bfa28e5fdfa1d9a8c6fac0f07ff7d6676/.) If you want to learn Javascript, this seems like an amazing way. (Kind of like jsfiddle but for actual apps.)

Malp2• 6 years ago

hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac21 has been giving me a 500

kickscondor2• 6 years ago

Somehow your link is truncated. The complete link is: hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac218436f12fda830/ (with a '30' at the end)

koeng• 6 years ago

Is beaker going to support IPFS?

anchpop2• 6 years ago

They used to, but they removed support. It makes sense to focus on just one protocol I think since they're such a lean operation. Also the guy running Beaker seems to be sort of anti-blockchain (describing himself as an "Unblockchain consultant") which might make them biased against it.

Also, IPFS urls change when the content they're pointing to changes, which makes them bad for decentralized web stuff. They are working on IPNS to handle this but I don't know what the status is.

wrnr• 6 years ago

How is the adoption of IPFS going, any example application that are using it and are themselves being used? I'm thinking of integrading it into a product, it has the best blockchain like properties without the mining madness.

momack25• 6 years ago

Going well! Check out all the awesome applications using IPFS in our editable ecosystem diagram: https://www.figma.com/file/2bVNnjoOmxh4TtID9kzqEG/IPFS-Appli...

A few highlights are Fleek, Textile, Audius, 3box, Anytype, Qri, Berty, and MagicLeap! Specifically in the blockchain space, there's Peepeth, Augur, Uniswap, Civic, Arbore, and Aragon.

momack25• 6 years ago

For anyone looking at https://arxiv.org/pdf/2002.07747.pdf to get an estimate on the actual # of IPFS nodes, note that these tools are trying to crawl the number of nodes acting as "DHT Servers" in the Public IPFS DHT. Nodes behind home firewalls or NATs don't become DHT routers because they aren't dialable, and other large users intentionally set many of their nodes into a lower-bandwidth client mode. Also, many applications run their own private IPFS networks (like OpenBazaar for instance) and don't participate in the Public IPFS DHT, so wouldn't be reached through a crawler here.

I built an IPFS crawler with libp2p

Related Stories