Syncer: fast stateful file/disk data syncer written on Go

by stargrave • 11 years ago

56 30 11 years ago

gionn• 11 years ago

Nice, but in which use cases it may be useful?

stargrave8• 11 years ago

I have got fast SSD that can break down every day. I have got USB hard drive. I want to sync data every day, twice a day between them as fast as possible (raw dd takes an hour). If SSD is broken then I just plugin USB hard drive and continue working (with some minor data lost).

ZFS has incremental snapshots, it is fast, but I need to restore bootloader, partitions and everything corresponding to be able to bootup and work again.

rocky1138• 11 years ago

Scratching your own itch is the correct path to success, in my books. Keep up the good work.

josteink2• 11 years ago

I assume the title has a typo and that it's supposed to say "written in Go".

With that said, as a user of a tool/library/service, why should I care what language it's written in?

I want to know if a data-syncing tool is reliable or not. I know rsync is, but OK that's not always super-duper fast.

I know volume-streaming with btrfs and ZFS works wonders, without the need to do manual book-keeping of a secondary state and hoping you ALWAYS keep that secondary state up to date. I doubt this out-performs the FS just doing the job itself, and it sounds much more prone to user error.

So what does this offer over those? It sounds like it has less features and is not as reliable. Being written in Go offers me nothing I don't already have.

stargrave8• 11 years ago

I answered in another comment on that question. The main point of course is not a language. ZFS incremental snapshots and its send/receive are great, but my task is to create binary identical image of hard drive with bootloaders, partitions and so on. I have got SSD that can breake and slow USB connected HDD. I want to sync them several times a day and be able to trivially switch them in the case of failure. ZFS is only a filesystem. I need to prepare harddrive with bootloaders to use ZFS send/receive. I do not want it. Raw dd is what is needed, but it is slow -- the only reason this utility is written.

andor2• 11 years ago

It should be enough to copy partition table and boot loader only once. After that, it's safer and probably faster to backup with consistent btrfs/zfs snapshots. Copying raw blocks from a writable mounted filesystem will lead to inconsistent backups.

josteink2• 11 years ago

ZFS incremental snapshots and its send/receive are great, but my task is to create binary identical image of hard drive with bootloaders, partitions and so on.

If you use UEFI, bootloader is just a normal file on a normal (FAT) partition so no special magic is needed, but fair enough.

ZFS typically handles partitions its own way so no need to sync that.

But really: If you just want to keep to full drives in sync, at block-level so one can at any point be a stand-in replacement for the other, it sounds like what you want is RAID0. Isn't that what you want?

And with UEFI you can boot those RAID-volumes, even though they are soft-raid volumes.

Not saying your need doesn't exist, but that existing solutions which are more standardized and requires less administration seemingly already exists.

andor2• 11 years ago

In my experience, btrfs send does a lot of random reads and is therefore rather slow when reading from mechanical disks. Backing up to a mechanical USB drive is fast enough, restoring from that drive can take forever (but you can use seeding). This tool reads the disk sequentially and should reach much higher throughput.

On the other hand, btrfs send skips empty and unchanged blocks. If the disk is mostly empty, send could still be the faster option.

stargrave8• 11 years ago

But btrfs is supported on GNU/Linux platforms, ZFS on FreeBSD/Solaris ones. So they are not portable, OS-specific -- and it can be disadvantage.

LoSboccacc2• 11 years ago

well I take 'written in go' is more than a warning than a bonus at this point, especially after looking at some similar tech and finding all this https://github.com/syncthing/syncthing/issues/9 (ended up using glusterfs in the end)

besides, this lib only cover one sync scenario and only one and does it only on demand. might be fast because it stores the hash somewhere, but that also means you'll run in issues if the mirrored slow drive content changes by any mean.

howeyc• 11 years ago

I don't see what your issue is with syncthing. syncthing is meant to be cross platform, so using inotify was an obvious problem.

Plus syncthing is an entirely different use case from glusterfs. Think Dropbox, not one-filesystem-across-many-machines.

jerf• 11 years ago

You can get inotify-or-equivalent support in Go for specific OSes reasonable well with bindings (simple enough even to write your own if push came to shove), what's missing is a cross-platform, simple API. Which, arguably, is because it turns out that despite the problem sounding simple, a cross-platform, simple API is all but impossible. You can sort of get one for a single file or a single small directory with no recursive subdirectories, but beyond that, you're surprisingly boned.

This hasn't got much, if anything, to do with Go specifically; in particular, I've been aware of everything I write in this post since before I was even aware of Go. The OS support is just too variable and shot through with subtle-yet-critically-important semantic differences, in addition to the fact the problem is harder than it looks at first. Trying to write a lowest-common-denominator API leaves you with a really low LCD. Trying to write a higher level API that offers the more convenient stuff leaves you with an API that makes it trivially easy to write something that works really, really well on one platform and is completely non-performant on another (in particular, trying to automatically "recurse" on the directories). It turns out there's no happy middle here for the general case.

coldtea• 11 years ago

With that said, as a user of a tool/library/service, why should I care what language it's written in?

Because it tells you a lot with what other ecosystems is compatible, how much of a fuss it would be to interface with it (in case it's a library and you use e.g. Java, you'd normally prefer a pure-Java implementation instead of a C+JNI one, etc), if it's a good fit for your existing dependency management, if you know the language to go in and fix an issue or adapt it, etc.

mobiuscog• 11 years ago

It would therefore also be useful to state the platforms it would work on - in this case it seems to require a block-level device and is unlikely to be usable on Windows.

tete2• 11 years ago

With that said, as a user of a tool/library/service, why should I care what language it's written in?

It makes sense to mention it for people who for example for learning purposes want to read code written in Go.

furyofantares• 11 years ago

HN is not really a "new products for you to use" site, and I find it a bit strange when people read and reply to posts purely from the perspective of a user or potential user.

If you approach the post with a more open curiosity I think it will make more sense to you why the language is a relevant detail, even if it still doesn't interest you personally.

kevan2• 11 years ago

It's late and maybe I missed something looking through the code, but what happens if you don't have write permissions on the directory that the statefile is in? The copy operation is performed but then the state is lost when the statefile rename fails.

stargrave8• 11 years ago

At first, temporary file created in current directory. So if directory is not writable -- temporary file creation at the very beginning will fail.

kevan2• 11 years ago

Ah, that's what I missed. I guess there's still a possibility of permissions changing during the transfer, but I don't think there's anything you'd want to do to handle that other than error out.

tete2• 11 years ago

Suggestion: Don't set a default for dst. That looks really dangerous.

stargrave8• 11 years ago

Agreed. Fixed. Thanks for the suggestion.

mercora• 11 years ago

A former colleague has written something quite similar[0] few years ago, which we used to backup encrypted disks (actually snapshots of encrypted disks to be precise).

[0] http://chunksync.florz.de/

perlpimp• 11 years ago

this is great. I wonder how it would perform when I need to sync 10mil. files. Rsync and anything else breaks down at this point, well it does work but it takes forever and rsync bloats to 10-20GB memory foot print and after a few hours starts streaming updates.

stargrave8• 11 years ago

syncer works on block level, with raw byte sequences. It knows nothing about file systems. Everything is limited with sequential read/write speeds of your hard drives.

It takes blocksize-size memory block for each CPU in your system. I have got 4 CPUs and work with 2 MiB blocks: program will take 8 MiB of RAM.

Syncer: fast stateful file/disk data syncer written on Go

Related Stories