Build An Opensource Dropbox Clone

http://fak3r.com/2009/09/14/howto-build-your-own-open-source-dropbox-clone/

by tommynazareth • 16 years ago

86 58 16 years ago

res0nat0r3• 16 years ago

I've looked at this project before, it looks like a nice implementation of rsync with inotify, but it seems to be just for a typical use case for rsync, ie, push your local changes to a remote host.

I think if you try to use this like a real dropbox, ie changes occurring on multiple hosts at or around the same time you are going to run in to issues. Also does lysnc handle deleting data on the remote end and propagting those changes out (rsync --del). Also I'm wondering how it will handle conflicts and open files (which it says isn't recommended). I see the .dropbox dir using some type of uuid's or sha1 sigs so I'm sure its doing something more sophisticated to keep things in sync. I've noticed my daemon has died on my linux box every once in a while, and it somehow tracks multiple changes from multiple hosts and replays those transactions correctly and everything rolls up fine.

tommynazareth• 16 years ago

I don't plan on ditching Dropbox, but I'm definitely interested in having a similar sort of automatic backup with diffs that I can host myself. This looks like it might be an option.

Does anyone else handle this in a different way?

patrickaljord• 16 years ago

I just use http://rdiff-backup.nongnu.org/

gvb2• 16 years ago

I use rsnapshot (also built on top of rsync/ssh) http://www.rsnapshot.org

aidanf• 16 years ago

I've used Unison (http://www.cis.upenn.edu/~bcpierce/unison/) to get a dropbox type setup running across several machines.

larelli• 16 years ago

I've thought about that as well. Could you give some details on the topology used? Did you run unison in automatic mode via cron?

dododo• 16 years ago

lsyncd does the brunt of the work. it uses inotify, which is a facility of the linux kernel, to tell which files should be copied. but i think there is a problem.

inotify is not guaranteed to see every file system access. in fact, it often misses them when the file system is busy because there is an upper bound upon the number of inotify events that can be queued, set in /proc/sys/fs/inotify/max_queued_events.

i.e., you could get data loss using this mechanism alone. you need to also run rsync every so often, i think.

the lsyncd guys are aware of this: http://code.google.com/p/lsyncd/

(maybe this is how dropbox works under linux too, if so it presumably has the same problem..)

beza1e1• 16 years ago

Well, Ubuntu One seems to rely on checking every so often alone ...

noja• 16 years ago

Any files not moved elsewhere within X seconds can be handled by a cleanup daemon.

rw-• 16 years ago

inotify is also unable to inherit handlers. after a "mkdir -p a/b/c/d" it is very likely that "d" is untracked. therefore lsyncd is useless on real systems. :(

brlewis3• 16 years ago

I would guess Dropbox has this same problem on Linux, though I can't say with certainty. What I can say with certainty is that they're thinking about the problem.

As is often the case, you learn a lot about a company through its jobs page. Their latest challenge for applicants looks very much like this problem.

xxpor2• 16 years ago

I don't understand why dropbox doesn't open source their client. Their business isn't built off of their technology, which, while nice, isn't exactly novel. They make their money from selling storage. Open sourcing their client doesn't change the fundamental fact people need the space to store stuff.

edanm4• 16 years ago

Showing it doesn't necessarily hurt Dropbox isn't enough. You have to show that it actually helps Dropbox to open-source the client. Otherwise, why should they?

xxpor2• 16 years ago

2 things that I can think of immediately

1. Would increase adoption on Linux clients 2. Increased security (the whole security though obscurity doesn't work thing)

pgbovine• 16 years ago

Their business isn't built off of their technology, which, while nice, isn't exactly novel.

well, be careful with such claims. many technologies don't seem novel at first glance, but i'm sure there are lots of novel tricks they play under-the-hood to get their system to run quickly and reliably.

e.g., google's pagerank is now well-known and public, but i'm sure that they have lots of 'secret sauce' internal algorithms to make their search index work even better, and they definitely don't want to share those secrets

tofumatt• 16 years ago

Open sourcing the client could mean I run it off my own server, sitting in my apartment, for free. Granted, mostly geeks would be doing that, but I get the impression that many of Dropbox's user base _are_ geeks.

Like someone else said: it's not just whether or not it would HURT Dropbox (though, as I've said, I'd argue it could), but whether or not open-sourcing their clients would actually HELP them. I doubt it would.

dagw2• 16 years ago

I get the impression that many of Dropbox's user base _are_ geeks.

This is not my impression at all. Basically everybody I know uses dropbox, and very few of them are geeks. I was actually introduced to Dropbox by a non-geek boss I had who thought it was the greatest thing she'd ever used and insisted I also get it so we could share files.

Dropbox seems to have done a very good job of penetrating the general market, thanks in no small part to their pretty amazing and minimal UI. It's at the point now where I hear 'normal' people saying things like "can you share it to my dropbox" with the automatic assumption that the other person will know what it means.

mrduncan• 16 years ago

From Drew's original YC application [0]:

# Why would your project be hard for someone else to duplicate?

This idea requires executing well in several somewhat orthogonal directions, and missteps in any torpedo the entire product.

For example, there's an academic/theoretical component: designing the protocol and app to behave consistently/recoverably when any power or ethernet cord in the chain could pop out at any time. There's a gross Win32 integration piece (ditto for a Mac port). There's a mostly Linux/Unix-oriented operations/sysadmin and scalability piece. Then there's the web design and UX piece to make things simple and sexy. Most of these hats are pretty different, and if executing in all these directions was easy, a good product/service would already exist.

I'd find it hard to believe that there isn't some very tricky stuff being done in the clients which gives them a big competitive advantage over competitors that pop up.

[0] http://files.dropbox.com/u/2/app.html

rmc• 16 years ago

If it's open source, anyone can run a "DropBox sever", and charge what they want. This would mean DropBox does all the hardwork and is then undercut by someone else charging less than them.

dagw2• 16 years ago

Dropbox does however have a network effect. One of the big points, at least for me and most dropbox users I know, is sharing folders. That only works if you're on the same server as everybody else is.

wrs3• 16 years ago

I've been involved in a few consumer-oriented file synchronization projects, and I can assure you that the client technology is not at all simple or easy. Consider that all existing applications have no idea that you're changing the semantics of the filesystem, but you have to make them act in a way that makes sense to the end user (not a filesystem expert). Not to mention all the little issues of conflicts, deletion, renaming, crazy stuff that apps do for "atomic save", and on and on.

It's a combination of having to understand app-filesystem interactions as well as user expectations, both of which are largely undocumented and unspoken -- but if you mess with either in the slightest way, somebody will freak out.

huhtenberg3• 16 years ago

> crazy stuff that apps do for "atomic save"

Care for an example?

please• 16 years ago

except that it misses 90% of what dropbox does. like multi client sync, cross platform, shared folders, web interface and so on

fluidcruft• 16 years ago

except that it covers 99% of what I actually use dropbox for.

kgo• 16 years ago

Don't forget thirty days of backups and redundant hardware...

gvb2• 16 years ago

What about using unison http://www.cis.upenn.edu/~bcpierce/unison/ instead of lsyncd? It seems like it would be more resilient. It does look like unison would require a cron job or human trigger where lsyncd is triggered by a file/directory change. On the plus side, it seems like unison would work better in an offline mode.

res0nat0r3• 16 years ago

I've tried to use this on syncing my mp3 directory between two boxes and it seemed to be horribly slow. It needs to stat each file for a second or two before it even begins to sync, this takes way too long.

dagar• 16 years ago

  -fastcheck
    do fast (and slightly unsafe) update detection on windows

mikeyg• 16 years ago

Zimbra's open source edition's WebDAV sharing works well once you get the URLs figured out. You get a lot more fine grained permissions. It's cross platform, there are shared folders, etc.. if that's what you're looking for. You can make folders public to share with others, and so on. With bandwidth as fat as it is these days DAV's heavyness doesn't feel so heavy anymore. Am I the only one that shares these sentiments? Did I mention it works on MacOS 10.6, Linux, and Windows 7?

If you're using Dropbox to back up some of these other solutions might make more sense. Crafting up a script using hard links and rsync can give you daily snapshots of any filesystem at only the storage cost of the delta (which can be easily turned into a cron job) .. here's a decent resource for that:

http://www.mikerubel.org/computers/rsync_snapshots/

khingebjerg• 16 years ago

Old news. http://news.ycombinator.com/item?id=1196421

telemachos2• 16 years ago

This is an off-topic short rant. Vote as you like (obvious enough), but skip reading this if you want on-topic discussion.

Shoot me if this is how the web is heading. (Recent browsing suggests maybe it is...) We get not only the insufferable "bottom bar" (is there a standard name for those yet?), but also a "top bar", a hover "Click Here to Share" box and a pop-up "Learn More, Instantly" box (not to mention the general clutter and overdense design).

Man, that's foul.

stevelosh• 16 years ago

I activated Safari Reader as soon as I saw those walls of 12px text, so I had no idea what you were talking about until I went back and let the page finish loading.

"Foul" is not a strong enough word for that. Maybe "vile" is better?

mkramlich• 16 years ago

Same here. And I love Safari Reader aka Readability!

A good rule of thumb I have is that if you "have to put a bag over the head" of something in order to make that thing pleasant to digest or access (whether it's a UI or a bureaucracy or paperwork or a person or physical widget or whatever), then really the source thing itself should be designed that way in the first place. In this case, my own ideal style of web design is very close to what Readability/Reader produces. Thus making it redundant to the extent I've achieved my ideal.

samstokes• 16 years ago

Those bars are even more irritating on a mobile device. The fixed height and position means they obscure half of the actual content and you can't scroll away from them. Do not want.

jpr2• 16 years ago

They also break the "scroll one whole page at a time" paradigm that many are used to.

EDIT:

I just realized that Chrome devs have apparently already decided that it's not worth fighting the idiotic bars, so Chrome scrolls less than one page.

Groxx• 16 years ago

I think I'll call them "Mercuries", now that I've thought about it (get it? floating footers!).

w1ntermute2• 16 years ago

Add the following to your hosts file:

  # Web toolbars/widgets
  127.0.0.1       wibiya.com
  127.0.0.1       cdn.wibiya.com
  127.0.0.1       apture.com
  127.0.0.1       www.apture.com
  127.0.0.1       cim.meebo.com

In this case, the top toolbar is from Apture and the bottom from Wibiya.

telemachos2• 16 years ago

Thanks for taking the time to track these down. (I love a rant with a happy ending.)

sprout• 16 years ago

Or simply install NoScript, and worry about whitelists, not blacklists.

I can get at the content there with no trouble, and don't have to worry about any of those toolbars.

spirulina• 16 years ago

Thanks for the list

Just added these to the "Adblock Plus" Preferences in Firefox (under Tools).

Worked like a charm.

dochtman• 16 years ago

I thought http://www.sparkleshare.org/ was more interesting.

Build An Opensource Dropbox Clone

Related Stories