Database-less torrent website

https://boredcaveman.xyz/post/0x1_dbless-torrent-website.html

by etagate • 5 years ago

237 71 5 years ago

vindarel• 5 years ago

related: https://torrent-paradise.ml/ Also based on IPFS with this HTTP gateway. I find a lot of torrents in there.

etagate3• 5 years ago

Love this website too. But in this case the database and search engine is hosted on the server, it only uses IPFS to distribute a copy of a folder ready out of the box (db included) to be hosted as another instance (a backup).

mab122• 5 years ago

You actually should be able to access it using IPFS at/ipns/torrent-paradise.ml or /ipns/12D3KooWB3GY1u6zMLqnf3MJ8zhX3SS1oBj7VXk3xp6sJJiFGZXp but it seems that clicking search throws an JS error :/

jokoon• 5 years ago

any similar link you know for DHT searches?

ThalesX• 5 years ago

If any dev is here, I'm on IPFS and can't seem to search, getting console error:

Uncaught ReferenceError: passQueryToResultpage is not defined

yonixw2• 5 years ago

Enhance it with the fact you can lazy load sqlite query using HTTP Range as demonstrated in: https://news.ycombinator.com/item?id=27016630

Closi• 5 years ago

Exactly the thought I had reading this! Combining these projects would be awesome, proper semantic web! Plus it would mean near-zero loading times as the db size scaled (although ultimately some peers would still need the full database).

lovasoa• 5 years ago

Well, you could reimplement "absurd sql" over ipfs. Possible, but far from trivial.

samwillis2• 5 years ago

As the SQLite file is being downloaded vie IPFS it wouldn’t be a HTTP Range request. Does IPFS have the equivalent?

(My guess would be yes as I believe it breaks up the file for distribution, but have no idea if it’s exposed in the IPFS.js api)

EDIT:

After a quick scan of the docs, I think you can do this (but I certainly don't know enough). With at least the "The Mutable Files API" which "is a virtual file system on top of IPFS that exposes a Unix like API", you can provide an `offset` and `length` to `ipfs.files.read(path, [options])`. I don't know if that then translates to only downloading that part of the file from IPFS or not.

https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI...

EDIT2:

In fact the `ipfs.cat` api that the OP is using takes `offset` and `length` parameters too.

https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI...

samwillis2• 5 years ago

Continuing my looking into this, the SQLite range requests trick was implemented in this [0] project.

It turns out that project was inspired by a few others [1], two of which [2][3] implement a vfs for SQLite on top of bittorrent doing exactly this suggestion, but with a bittorrent hosted file rather than IPFS. They only download the parts of the SQLite files needed when querying.

0: https://github.com/phiresky/sql.js-httpvfs

1: https://github.com/phiresky/sql.js-httpvfs#inspiration

2: https://github.com/lmatteis/torrent-net

3: https://github.com/bittorrent/sqltorrent

SahAssar• 5 years ago

It doesn't have to be range requests, you could split the file instead of depending on the range requests. Essentially it's the same as the chunking instructions for hosters who have a maximum file size as is demonstrated here: https://github.com/phiresky/sql.js-httpvfs/blob/master/creat...

politician• 5 years ago

How do immutable systems like IPFS manage the problem of updates to a blog?

yonixw2• 5 years ago

You have "publish ids" which you control using private keys.. like a dns... so you can have your main page address known and all the other posts linked from that page

Gigachad3• 5 years ago

They have another system called IPNS but unless it’s improved since I checked, it wasn’t very practical.

lekevicius2• 5 years ago

What's wrong with IPNS? I'm using it for my decentralized website (via fleek), and it seems to work well.

tecleandor2• 5 years ago

I'm curious about why didn't they compress the SQLite file using gzip. Even in level 1, for superfast compression/decompression halved the file size.

etagate3• 5 years ago

I knew I could do it, I just wanted to make a demo of the whole thing to see if it would have worked in practice. But yes there's space for a lot of improvements like compression and lazy load with sqlite (cited in a comment)

tecleandor2• 5 years ago

Great! I was honestly wondering because I didn't know if there were any technical limitations/problems, but it's quite interesting nonetheless.

danka• 5 years ago

I wonder why the author ignored the option of compression in the post. Even with a simple gzip DEFLATE compression, those 10MB of plain text could get as small as a 1MB archive and possibly more, meaning that in a compressed 10MB payload you could fit much much more than 135K records.

vermilingua• 5 years ago

It isn't 10MB of plain text though, it's 10MB of binary SQLite database. I agree that compression would be useful here, but I don't think a simple gzip DEFLATE would be.

I was curious so I compressed that torrent db with a few different methods:

  11.1MB 11116544B dump.sqlite
  10.2MB 10155419B dump.csv
   6.6MB  6573399B dump.sqlite.gz
   6.6MB  6565771B dump.zip
   5.6MB  5616842B dump.rar

gzip is certainly suitable to be used in this situation, I stand corrected.

rdubz• 5 years ago

Back of envelope:

The 10MB estimated size came from [100 bytes per row] * [100k rows].

50 of the bytes per row were "description", which should compress well (2-3x, I'd guess).

40 bytes per row were the IPFS ID/hash, IIUC. I assumed this is like a Git hash, 40 hex chars, which is really just 20 bytes of entropy.

He also estimated 14 bytes for the size (stored as a string representation of a decimal integer, up to 1e15 - 1, or 1PB?). That's about 50 bits or 6-7 bytes, as a binary integer. Sizes wouldn't be uniformly distributed though so it would compress to even fewer bytes.

So if SQLite was smart (or one gzips the whole db file, like you did), it makes sense that a factor of 2 or so is reclaimable.

thinkloop• 5 years ago

IPFS requires "pinning" content for it to exist, meaning you host it, whoever pins content becomes liable for its distribution, if no one pins it, it disappears. That's why you can't freely host things like child porn on IPFS.

etagate3• 5 years ago

This is true even for a torrent btw. And (un?)fortunately you can distribute anything on IPFS as long as it's encrypted.

Gigachad3• 5 years ago

No one is going to pin encrypted data they do not understand. And IPFS doesn’t get users to seed random files, you only seed stuff you downloaded. So really it’s not much different to hosting it on a personal http server except any downloaders can also host.

only4here• 5 years ago

This will probably make illegal stuff less common, hopefully. At least outside of the darknets.

dfghdfhs2• 5 years ago

I love the internet.

lekevicius2• 5 years ago

One unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think there's a space for decentralized database format. Something that would have immutable rows (not the whole db), ranges and search, indexes, etc. Maybe there's something like this already?

gerwim• 5 years ago

Well there is Dolt: https://www.dolthub.com/

mosselman• 5 years ago

(or, updates require re-deployment of full new .sqlite db, together with a new website)

Seems like a small price to pay.

I can see new stuff living in torrents that then get committed to a sqlite database that is then deployed every now and then.

chrisjc• 5 years ago

Something that would have immutable rows

Isn't this just a log (or a stream like Kafka or Kinesis)? In fact you might even be able say every database already has this ;) (binlog, oplog, etc)

not the whole db

If rows are immutable, what part of the db is left not being immutable? If rows were immutable, doesn't that imply that any existing "ranges, searches, indexes, etc" are immutable too?

If you going to the effort of making a decentralized database, why not also decompose all of these parts from one another... no reason the tables (logs), indexes, search, etc have to live in the same database, they could be spun off as completely different parts. Basically a something that indexes logs and then something else that takes those indexes and makes the searchable.

In fact this is all the rage right now with centralized databases... all of the work being done with streaming systems just seems to be an effort of decomposing and inverting databases... every old is new again.

Anyways, I agree with the idea and don't really know enough about decentralized systems to really understand why such a distributed database can't or hasn't already been built.

oscargrouch4• 5 years ago

My current project can expose a sqlite database over torrent wrapped by the Chrome cache filesystem.

The torrent is merkle-tree based, and for each change in the SQLite memory page it also updates the merkle tree giving you a different torrent infohash at the end.

So as databases are published together with the applications the initial torrent info is the same, so theres a swarm of initial peers. Once it gets updated and the torrent "turns into another", the solution is to use the RPC api (that my solution provides) over the initial swarm of peers and use some sort of distributed algorithm like raft or gossip to define how to deal with the subsequent changes over the peers.

The important part is that all other changed torrents from all the peers are available to all peers in the network, and with a RPC working as a abstract interface to organize them, it will work for a lot of different scenarios.

smallerfish2• 5 years ago

That's interesting. Are you aware of the absurd-sql project? It may have some useful stuff that you can leverage.

Also, how do peers find each other? And finally, do you intend to open source it?

rakoo2• 5 years ago

Mutable torrents do exist (https://www.bittorrent.org/beps/bep_0046.html). Your "URL" can now be a public key, and clients will fetch the latest version of the torrent.

ddtaylor2• 5 years ago

This is actually really neat I wasn't aware of these BEPs. Do you know of any place where they are being used?

pradn• 5 years ago

I think it's possible to use IPNS to create a key-value mapping where the value can be changed.

https://flyingzumwalt.github.io/ipfs-tutorials/curriculum/fi...

all2• 5 years ago

What about a distributed hash table [0]? Yggdrasil uses it for network routing information [1].

[0] https://en.wikipedia.org/wiki/Distributed_hash_table

[1] https://yggdrasil-network.github.io/2018/07/17/world-tree.ht...

delusional• 5 years ago

I think you just reinvented the bit torrent mainline DHT.

imtringued2• 5 years ago

There should be a separation of data and the actual website that does the reading. I once thought about something similar (not for torrents), but having to download the whole database (300MB and more) was a bad idea for UX.

So what I propose is this: There is a central source of truth hosted on IPFS via IPNS or mutable torrents. The data format can be whatever is more efficient to update and store, not easiest to read and query. (An event log is fine) Then, there is a reader application, like an RSS reader. The reader application can be a desktop application or it can be a centrally hosted web app that keeps a conventional SQL database behind the scenes. The idea is that there will be multiple reader providers like there are competing email providers.

dfghdfhs2• 5 years ago

Could someone more savvy clarify....

On that page I do inspect, got console and do:

loadDBAndExec('SELECT * FROM my_table LIMIT 10;')

I get

Uncaught ReferenceError: loadDBAndExec is not defined

Why doesn't this work?

manigandham• 5 years ago

That's just example code in the blog post.

The live demo uses a webworker with different code: https://boredcaveman.xyz/demo/megacat/database-app.js

xialvjun• 5 years ago

Since it has used IPFS, why not host the content of those torrents with IPFS?

betwixthewires• 5 years ago

You can do that. And I like the idea. But for now, torrents are the go to method of sharing large pieces of data, and utilizing IPFS to move the centralized aspect of torrents to a decentralized space is a good idea.

donkarma• 5 years ago

That's not going to stop takedowns, they don't care about if it's technically on your site or not.

jtbayly• 5 years ago

Exactly. I forget where I just saw this. It seemed like a particularly absurd example, but the examples are numerous. Torrent sites don’t have the copyrighted content either, but they get shut down all the time.

roywiggins• 5 years ago

But the torrents themselves tend to stay seeded... if you stick the site itself on IPFS and anyone in any jurisdiction can "seed" it, then it's going to be hard to kill by court order.

tablespoon2• 5 years ago

That's not going to stop takedowns, they don't care about if it's technically on your site or not.

Yep.

"The underlying mistake is thinking about this like a game where you can make up rules for the government to follow." - https://news.ycombinator.com/item?id=29913036

Also the assumptions here kind of remind me of domain fronting: https://www.zdnet.com/article/amazons-aws-latest-to-give-up-.... Basically, it's assuming IPFS will protect it from the authorities, but what might actually happen is it makes IPFS a target of the authorities. Now that probably won't happen with this because the authorities don't actually care that much about torrenting/piracy, but the error is still there.

Chris20482• 5 years ago

The "rules for the government to follow" are the laws, and these aren't being made up, but rather pre-exist (safe harbour laws) with some pretty strong backers (goog et al)

Chris20482• 5 years ago

Do they also not care whether they technically have the authority of law or not?

imtringued2• 5 years ago

They can change the law to whatever is more profitable...

ineedasername• 5 years ago

The headline doesn't make much sense. There's still a database, what changes is simply where it's stored when in use. "Torrent Website w/o DB Server" might work better.

That aside, would this method scale? 135,000 torrents doesn't seem comprehensive, so I would expect real world use to have many more. Maybe a different SQLite db for different categories?

moolcool• 5 years ago

Since it's search is just based on prefixes, you could probably just alphabetically partition the databases.

senjin• 5 years ago

Makes me wonder if you could host a pouchdb in ipfs

SuchAnonMuchWow• 5 years ago

Reading just the title, I was expecting storing/serving magnet files on DNS, as someone did a few years back if I remember correctly (just as a joke/poc, not as a serious solution).

I'm pretty sure it was posted on hw, but I can't find the link anymore.

ronyfadel• 5 years ago

I'm disappointed that I don't find result for "La vita e bella" but I do find results for "La vita è bella".

kderbyma• 5 years ago

love these kind of things. I am interested in distribution methods for smarter clients without needing a dedicated server to host from.

MuchHustle• 5 years ago

Theres already like 10 crypto projects implementing that. The most prominent is probably Bit Torrent Token (BTT). https://www.bittorrent.com/token/btt/

Hakashiro• 5 years ago

These projects while technically impressive and interesting, I'm sure wouldn't make much of a difference to copyright holders. I can imagine something like this: - Please take down our IP

- I can't because I am not hosting it

- But you are displaying it on your website, take your website down or filter what your website displays to avoid showing copyrighted material

And that's the end of it. It doesn't matter where the data is stored. It does matter, a lot, where the data is displayed. If your website displays illegal content, you're facilitating access so you're breaching the law. And, honestly, I'm not sure there's an easy solution to that other than using onion or I2P sites which are only marginally more safe against copyright takedown requests because, due to their very nature, they are obscure and difficult to access.

Database-less torrent website

Related Stories