Dropbox-as-a-Database

http://blog.opalang.org/2012/10/dropbox-as-database.html

by cedrics • 14 years ago

84 32 14 years ago

blog.opalang.org

I've used .json files in place of a database before on a read-only site. It worked very well. I do believe that we over-engineer some times and use databases when flat files will do just fine.

codewright• 14 years ago

Using sqlite as the "this is just an idea" phase positions you better for scale-out/build-up if it becomes necessary. Helps keep your head straight but doesn't slow you down.

I say this as someone who worships Crockford for inventing JSON.

I've seen too many projects get fucked hard because they used flatfiles.

Edit: Seriously though, managing synchronization/atomic changes to flatfiles manually is a Bad Idea (TM).

It's somewhere up there with, "I have a performance problem. I'll use pthreads!" Now you have 1,02-SEGMENTATION FAULT problems.

MatthewPhillips2• 14 years ago

Like I said, in my case there were no changes, I only read from them. So I let the server cache the hell out of them and the entire site was fast because of it.

IsaacL• 14 years ago

You're not alone. Guess which famous tech website uses flat files as a database? Hint: orange header, heirarchical comments...

habosa• 14 years ago

Is this true? Is there some post about this?

tarr11• 14 years ago

It is a fun exercise to do these kinds of things. Obviously, they are very inefficient. But that is not the point! Perhaps there's some residual value related to some Dropbox feature (such as automatic replication) that makes this useful . But that is countered by getting your account rate-limited by Dropbox.

I wonder, is there some way to create a key-value store out of any physical or virtual object that can store state? I know there is a lot of work done with quantum computing to store state in quarks. I can see that has value.

But, what is the most inefficient data storage mechanism you can come up with? Here are a few I can think of:

Instagram-as-a-database - You issue a request to save your data, we encode it using some visual encoding scheme, take a picture, and then upload it somewhere. You get returned a URL, and can then decode it or update it.

MTurk-as-a-database - The possibilities are endless here. Someone could write down your request on a piece of paper and then you could send another request where they type it back.

Twilio-as-a-database - Encode your data as an audio stream. Store it in random voicemail boxes. Retrieve it later by calling up, listening to your voicemail, decoding the messages.

Jabbles• 14 years ago

quantum computing to store state in quarks

Atoms or electrons, but not quarks.

Erwin• 14 years ago

Humanity as a database: after infiltrating a religious order mandate arranged marriages encoding your input data in predicted genetic mutations of offspring. Ensure the religious order is hermetic so no outside data appears and that its teaching require all members to appear at a certain location at a certain time. Retrieve your data 1000 years later by appearing at the preordained spot for the Holy Query Ceremony, and examining the eye/hair colours of those gathered. Good error correction protocols are required. (This might well already be part of a Neal Stephenson novel).

plaguuuuuu• 14 years ago

A Minecraft game would make a hilarious database. The fact that it's probably pretty trivial to CRUD it programmatically outside the Minecraft client by manipulating save files is a great bonus.

Start using the DB. If you get bored enough you can examine the data physically in-game.

fiatpandas• 14 years ago

Why not go full-hilarity and not touch the save files? The DB API simply performs clicks, keydowns, and mouse movements on the game client, so you see the character moving on the screen. I guess you would have to initially read the save file though, to know what you start with. Would be hilariously slow.

gdp3• 14 years ago

It seems rather generous to call this a database. It's files on a filesystem. The lack of any kind of transactional consistency guarantees (Dropbox is notoriously awful at this), and the lack of any sane mechanism for querying this "database" means that it's just "data", not a "database".

yefrederic4• 14 years ago

I think he calls it "database" because it is strongly integrated with the database syntax of the Opa language. You can declare and use this "dropbox-database" just like a Mongo one, independently of the notion of safe-ness of the database.

gdp3• 14 years ago

Consistent with the complaint leveled against my previous comment, I should think that this is really consistent with the DBMS syntax of the Opa language</pedantry>

mistercow• 14 years ago

Transactions are not a requirement for databases, and querying is the domain of the DBMS, not the database itself.

gdp3• 14 years ago

You're quite right. I think my complaint was that it reduces the definition of "database" to "anything that is used to store data", rather than the database being the json file(s) themselves, at which point we have degenerated to about the same terminological point where it would be reasonable to call saving CSV or MS-Access files into a Dropbox folder "Dropbox-as-a-Database", which seems an unhelpful label, if nothing else.

sehugg• 14 years ago

Reminds me of the GMail filesystem: http://www.viksoe.dk/code/gmail.htm

bduerst2• 14 years ago

trevorcreech• 14 years ago

Ha, I remember using this for backups before I had even heard of ftp.

mattmanser• 14 years ago

Your blog design is bad btw, popout at side overlays scroll bar in Windows/Chrome.

yefrederic4• 14 years ago

It's one of the design proposed by Google, blame them :)

bduerst2• 14 years ago

Google isn't the one using it to demonstrate their work as a developer ;-)

ehutch79• 14 years ago

Plus, i can barely read the text.

formatjam• 14 years ago

Actually I have the exact same idea like a few weeks ago. The potential for this idea is big. It can be widely used by small web apps (Chrome App). Instead of hosting a server and a database, every thing can be done and stored in the client, isn't that neat?

yefrederic4• 14 years ago

You should have a look at filepicker.io then

ananddass• 14 years ago

as yefrederic pointed out you could use filepicker.io to build static js applications. We handle all the server side logic for you.

taskstrike• 14 years ago

Dropbox as a database is a actually great idea. It takes away scaling problems from the developer and puts the data control back to the user.

I'm actually working on NimbusBase, a product that allows you to write clientside javascript applications where there is no server code and data is stored inside of Dropbox. It also integrates with backbone.

Shameless plug: sign up for our beta! http://nimbusbase.com

pencilcode• 14 years ago

is this part of the app open-sourced? it's a great idea.

yefrederic4• 14 years ago

You can find the source code here : https://github.com/cedricss/server-monitor/

hbbio• 14 years ago

The list of commits is interesting: https://github.com/cedricss/server-monitor/commits/master

And also shows its really a one-day app!

cedrics• 14 years ago

http://server-monitor.herokuapp.com/ is open-source (MIT), and use the Opa framework (also open-source, MIT). The Dropbox-as-a-Database is integrated in Opa by default.

shykes• 14 years ago

"update: Heroku is down [...] the demo in now on dotCloud."

Dropbox-as-a-Database

Related Stories