Datastore API (beta)

https://www.dropbox.com/developers/datastore

by gluegadget • 13 years ago

207 84 13 years ago

For the iOS library specifically, I wonder if it would be possible to build a library that would (nearly) automatically map a core data store in to this Datastore API. If so, including such functionality would make the library a no-brainer for many.

ZitchDog• 13 years ago

I think with NSIncrementalStore it would be totally doable, and a great alternative to Core Data Sync.

frankcaron• 13 years ago

I wonder how Apple will react to @dropbox's new Datastore sync in the long run. Hopefully it won't pull the rug out a la IAP, UDID, OF, ...

jemeshsu• 13 years ago

I would rather they react positively by opening up their iCloud API for web application for third parties. Apple already has such in place as can be seen from their cloud apps at icloud.com.

duaneb2• 13 years ago

Apple is a hardware company.

callumjones• 13 years ago

I highly doubt they would do anything to stifle Dropbox's syncing functionality in iOS. Apple is still focused on selling iOS devices and ensuring people keep buying iOS devices, they will do anything in their power to keep you happy.

Limiting Dropbox functionality on iOS would be met my swift user opposition, something that doesn't guarantee recurring sales.

obilgic• 13 years ago

What app.net wanted to accomplish.

WalterGR• 13 years ago

At first glance it also looks like WinFS for the Cloud.

jimsilverman• 13 years ago

this is basically what iCloud was supposed to be.

BadassFractal• 13 years ago

What is the long term vision here? Is the idea that consumers will have little apps here and there that will procedurally interact with the owner's data on Dropbox? Or, is Dropbox making a product along the lines of S3 so that backends can consider using it too? I guess given how expensive space is at this point, that's not really an option.

chacham15• 13 years ago

dhouston has said more than once that he wants dropbox to be the filesystem of the internet.

mmahemoff2• 13 years ago

Something I don't get about Dropbox API: Why can't I just throw URLs at it?

If a user wants to save <thing at URL>, I have to pull it to my server and push it into Dropbox. Dropbox could just have grabbed it directly and in many cases, won't even have to thanks, to system-wide caching.

(This is based on the last time I looked at the API; maybe this is supported now, but I don't think so.)

timdorr• 13 years ago

You're describing the Saver Drop-In: https://www.dropbox.com/developers/dropins/saver

natch3• 13 years ago

Can the Saver Drop-In be used with arbitrary third-party URLs? To me this is what was being described, so if not, then no, it's not describing the Saver Drop-In.

mmahemoff2• 13 years ago

Great to know they got around to this and did so in a way that leverages the existing HTTP caching protocols. Thanks!

JosephRedfern2• 13 years ago

How would system-wide caching work unless the HTTP server dropbox requested the file from sent a checksum of the file being requested? Just because something has the same URL doesn't mean that it's the same file at the other end (http://wordpress.org/latest.tar.gz for instance)

varenc• 13 years ago

It obey's HTTP caching controls. Checkout the end of the docs: https://www.dropbox.com/developers/dropins/saver

joejohnson• 13 years ago

So Dropbox is becoming a more-direct competitor with iCloud. Meh.

I'd rather they work on an end-to-end encryption system with a focus on ease of use.

Afforess3• 13 years ago

Hacker News is a tiny audience. I don't see any business sense in focusing on features that 0.1% of users want.

albertyw• 13 years ago

Not to sound self-righteous, but the HN audience (or more plausibly, some subset within) has a high influence on what the rest of the world sees and wants.

Any Dropbox employees read HN?

afarrell• 13 years ago

If they did want to serve this audience, they would actually want to make it easy for someone (possibly the employees themselves) to build a Free encryption layer on top of them.

iamclovin• 13 years ago

I think this is exactly the direction they should take.

oellegaard• 13 years ago

Such an awesome idea. However, in light of the recent events the first thing to come up in my mind when I see "datastore" and a US corporation is "PRISM integration". I'd like to see proper encryption of Dropbox as the next feature, so that we - innocent users- are not uploading our data to the US government.

redblacktree2• 13 years ago

I combine dropbox with TrueCrypt for sensitive stuff. For me, that's just digital copies of important documents.

aasarava• 13 years ago

So are you just syncing entire truecrypt .tc files to Dropbox? Does that work incrementally or does the entire .tc file have to get resent every time you modify a file within?

philip1209• 13 years ago

Yes - in my dropbox public folder, I keep a truecrypt bucket with scans of everything in my wallet, should I lose it.

Afforess3• 13 years ago

Can't you just encrypt the data before storing it in your app? Why does Dropbox need to do this for you? Would you even trust Dropbox if they did add it themselves?

hcarvalhoalves• 13 years ago

The right approach is to encrypt before pushing down the wire - it's useless if Dropbox just encrypts on their end.

peterkelly6• 13 years ago

Well, the nice thing about PRISM is that it's API is so simple that you don't even need to muck around with SDKs, REST, JSON encoding etc. Integration is completely straightforward.

lovemenot• 13 years ago

Expect to see the NSA as a back-up-of-last-resort cloud solution provider any time soon?

r00fus• 13 years ago

Sadly, you're going to need to encrypt before uploading. Just as the webserver should never trust the client, the user should never trust the server with sensitive data.

Try boxcryptor - works with more than just Dropbox.

duaneb2• 13 years ago

Proper encryption meaning metadata is encrypted as well? Isn't that the whole appeal of http://mega.co.nz/?

jonaldomo• 13 years ago

After quickly clicking through the api documentation I don't understand why I would use this over a hosted NoSql database like mongohq. I still like having additional options though.

forgotAgain• 13 years ago

Off-line access.

icefox• 13 years ago

Perhaps you don't want to host a db just for user data.

peterkelly6• 13 years ago

The biggest problem right now with online file storage services (sorry, I meant "The Cloud") is that while they all do basically the same stuff, there's no standard protocol for accessing them. It's a bit like if every website used a custom protocol instead of HTTP, and web browser vendors had to explicitly collaborate with each web site (or use the website's SDK) in order to be able to retrieve content.

Imagine this: All the major players in the space (Dropbox, Box, Google Drive, Skydrive, Sugarsync etc.) as well as app developers got together and came up with a standard way of doing this that makes it possible to use a single protocol & API for all online storage providers. I see this being a good thing for everyone - users can choose whatever service they want, app developers can easily provide their customers with a ton of options, and storage providers don't have to explicitly produce their own SDK or court developers to do integration with their service. We did have WebDAV, but that doesn't seem to be widely adopted by any of the major players.

I've been facing this problem myself with my own app; it's a lot of work to support all the major services and this is really just plumbing stuff that everyone would rather be taken care of so they can focus on the more important stuff. I had a meeting with Box recently where I proposed the idea of an open source framework for iOS which would provide a common native API that abstracts over all the different REST APIs used by different storage providers. The people I spoke to seemed quite open to the idea and agreed that it would be useful. I don't think a library like this would necessary be considered a bad thing by any of the major players, as it evens out the playing field and makes it easy to support different services.

If you're interested in taking part and/or using such an API in your iOS app, I'd love to hear your thoughts on the matter. I have some code already (which I haven't released yet) and am looking to clean up the API a bit, but I think it would be useful to a lot of developers.

icebraining• 13 years ago

Big players aren't interested in anything that reduces lock-in. Seamless integration is one of the keypoints they have to get you to use their services.

Obligatory presentation by Bruce Schneier on the online feudalism: http://www.infoq.com/presentations/cloud-security

peterkelly6• 13 years ago

I found Box to be the opposite of this. While I obviously can't speak on their behalf, and I was discussing the issue with their developer relations people (not senior executives), they said they didn't see a problem with this idea.

DTE• 13 years ago

exactly...it is unlikely they would be interested in data storage being treated as the commodity it is.

davidjgraph2• 13 years ago

That's exactly the idea of Remote Storage [0]. The problem is here that the lowest common denominator of functionality isn't actually very much.

Anything but fairly trivial apps quickly run into problems with application data (how relevant), file patching, file intents (Google have this),

Google have, IMO, the most advanced solution in the space, so you could argue have the most the lose from standardising. But in fairness, some of their stuff (I'm thinking the real-time collaboration part of the Drive API) is so complex, there's no way you'd ever see it in a common standard.

That said, it'd be really nice to see a common API over the lower stuff.

[0] http://remotestorage.io/

peterkelly6• 13 years ago

Thanks for the link, this does look very interesting.

I believe filepicker.io have something along these lines as well, but haven't had a chance yet to look into the details of it.

nly3• 13 years ago

By Jove, you're right! Perhaps what we need is some kind of universal protocol for transferring files. We could call it the 'File Transfer Protocol' or something.

peterkelly6• 13 years ago

LOL yes ;)

What happened to FTP anyway? I know there were a number of issues with it (security, working through NAT, lack of a standard directory listing format etc); but these were all problems that people managed to solve in different ways.

It just goes to show how far how our industry is willing to embrace change for the sake of change (with the results sometimes being inferior) every time a new buzzord appears. I reckon a modernised version of FTP would do the job pretty well.

What I'm mainly concerned with at the moment is building something that recognises a widely-adopted protocol is going to be very difficult politically to achieve (and get implemented), but a library which abstracts over the different protocols would relieve most of the pain.

roc• 13 years ago

As it turns out, the protocol for actually transferring the files is utterly irrelevant compared to the protocol for getting updates, keeping various local stores synchronized, retaining versions, managing conflicts, handling client-side bugs and surprises, etc.

brettcvz• 13 years ago

Check out https://www.inkfilepicker.com/

taskstrike• 13 years ago

Hey, this is exactly what NimbusBase (http://nimbusbase.com) wants to do. It's a work in progress to integrate all the services, but we got the major 2 Dropbox and Google Drive.

johnrob• 13 years ago

Grammar check: "These days, your app need to store and sync more than just files."

munificent• 13 years ago

I'd like to coin a modern update of Zawinski's Law[1]:

    Every software company attempts to expand until it
    offers a cloud-based platform API. Those companies
    which cannot do so are replaced by ones which do.

[1]: http://en.wikipedia.org/wiki/Jamie_Zawinski#Zawinski.27s_law...

modarts• 13 years ago

Why would I use this over something like Parse?

joeblau3• 13 years ago

I haven't looked into it but it may be cheaper. That being said, both Parse and Azure support more platforms than this offering. I think that even Firebase has an service in this space as well.

fireboi2• 13 years ago

are there any kind souls that could help please explain the difference between dropbox datastore, firebase and parse? thank you.

sankethkatta2• 13 years ago

In comparison to firebase, this solves the offline sync problem, if your user is offline, you can still save to dropbox datastore locally, when you get a connection, it will automatically sync and take care of conflict resolution.

mikelehen• 13 years ago

Firebase dev here. :-) Firebase does handle offline syncing / conflict resolution.

If you have a Firebase app open and you lose network connectivity, the app still continues to work fine and any modifications you make will be synced back to Firebase when you regain connectivity.

What we don't (quite yet) do is handle the case where you make changes, kill the app (without regaining network connectivity), restart the app, and then get network connectivity, but that's coming soon.

As for other differences, Dropbox Datastore seems to be tightly tied to Dropbox, so the end user for your app must have a Dropbox account (and if they want to collaborate with other users, those users must also have Dropbox accounts, etc.). So it's really for building apps on top of Dropbox.

Firebase and Parse are just generic backends, with no ties to other services, so your end users don't need anything to use your app.

As for differentiating Parse and Firebase, Firebase deals with data in real-time, pushing updates to apps as soon as data changes. Parse is a traditional request/response model, where your app has to explicitly "refresh" to get new data.

jimbokun• 13 years ago

"List – A special value that can contain other values, though not other lists."

https://www.dropbox.com/developers/datastore/tutorial/ios#re...

Seems like a very significant limitation. No mechanism for storing hierarchical data structures. For example, this is certainly not capable of storing arbitrary JSON data as a Record.

est• 13 years ago

There's also limit on Postgres hstore and arrays. No nested sturcture.

joeblau3• 13 years ago

Does anyone know of an OSS alternative to this similar to http://helios.io/ that supports Android/iOS?

jchrisa2• 13 years ago

Couchbase Lite is an embedded JSON database with sync capability. It's Apache licensed and available for iOS and Android:

https://github.com/couchbase/couchbase-lite-ios

https://github.com/couchbase/couchbase-lite-android

joeblau3• 13 years ago

Thanks for the link to this library. It seems like I would still need to implement a lot of features like login, but it's a great start.

makeramen2• 13 years ago

As someone who has been playing around with Parse and Firebase SDKs (on Android), this seems to be the first one that offers full 2-way sync. I haven't actually used it yet but it seems that way due to the presence of conflict resolution [1]. I'm personally really excited about that for mobile side-projects and MVPs.

Parse has an offline cache, that only caches query results and not actual objects. So any local saves/updates are not reflected until you get network access again, unless you manually implement your own cache.

Firebase is better at handling realtime changes, but is lacking in any realtime capabilities at the moment. Also Firebase's mobile authentication SDK still isn't released yet so no way to handle users yet either.

[1] https://www.dropbox.com/developers/datastore/tutorial/androi...

pavpanchekha• 13 years ago

Yeah, the Datastores API does two-way synchronization. Also, offline caches are done for you and offline changes are propagated (with conflict resolution) when the internet comes back up.

makeramen2• 13 years ago

Correction: "lacking in any realtime capabilities" should read "offline capabilities". And by that I mean offline without manually managing cache.

Also by 2-way sync I mean 2-way cache sync. Firebase is actually really good with 2-way sync, but without cache/offline (yet) it's kind of a dealbreaker for me.

sigil2• 13 years ago

So now app developers have a choice: they can use iCloud or Dropbox to manage distributed app state. Any iOS app developers (I'm not one) care to compare & contrast the two options?

It looks like Dropbox may be using Operational Transformation [0] to merge changes to lists:

Lists are special field values. They contain an ordered list of other values, though not other lists. Lists can be manipulated via four list operations: put (i.e. replace), move, insert, and delete. These list operations allow Dropbox to handle merging changes to the structure of the list automatically. [1]

iCloud, on the other hand, uses the coarser, state-centric approach of whole document versioning. [2] (Edit: okay, key-value property versioning is also available.)

But iCloud lets the app control conflict resolution, while the Dropbox Datastore API only lets you choose from a predefined set of conflict resolution strategies. [3]

Which approach provides greater value to the app developer? Has Dropbox realized here that developers are just not comfortable implementing a custom merge strategy for entire documents, and not having to worry about this is worth some loss of control?

[0] http://en.wikipedia.org/wiki/Operational_transformation

[1] https://www.dropbox.com/developers/datastore/tutorial/ios#re...

[2] http://developer.apple.com/library/ios/#documentation/DataMa...

[3] https://www.dropbox.com/developers/datastore/tutorial/ios#co...

hboon• 13 years ago

Not answering your question, but developers can also use http://parse.com or similar.

Datastore API (beta)

Related Stories