Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

http://zsync.moria.org.uk/

by pmoriarty • 9 years ago

215 46 9 years ago

zsync.moria.org.uk

On a LAN or UDP-based "Layer 2 overlay", there is mrsync for this purpose. One could efficiently distribute regular updates to some data that everyone on the network needs, e.g., "domain names" and IP addresses.

bruxis2• 9 years ago

This makes the claim that you don't need to run any additional software on the server, other than having an HTTP/1.1 compliant host.

How does zsync diff against the local file without downloading the contents from the server?

arbie• 9 years ago

"zsync is only useful if people offer zsync downloads."

It works by expecting a .zsync metadata file which gets downloaded first and is used to guide the differential download.

deelowe3• 9 years ago

It uses a special file that has to be installed on the server. Perhaps you thought (as did I) that this was supposed to work with any file over http? That does not appear to be the case.

http://zsync.moria.org.uk/server

bruxis2• 9 years ago

I see, this really threw me off:

"Rsync over HTTP — zsync provides transfers that are nearly as efficient as rsync -z or cvsup, without the need to run a special server application. All that is needed is an HTTP/1.1-compliant web server. So it works through firewalls and on shared hosting accounts, and gives less security worries."

The other child comment also touches on something that I seem to have skipped over, the .zsync metadata file which contains pre-calculated rsync hashes. Using this means software isn't needed, but during upload the file needs to be processed to produce this metadata.

oliwarner2• 9 years ago

Installed on? Uploaded to.

It's just a pair of files. The big thing you're trying to transfer and the zsync file that details the content of that first file, to guide the downloader.

takeda• 9 years ago

This reminds me about the demo utility that comes wit rsynclib called rdiff, except they approached this problem in less practical way, although it shows rsynclib better.

It works as follows:

- Let say you already have file that is older version, or perhaps corrupted, you use rdiff to generate its signature

- you go then to the place which contains proper file and use the signature file to generate a patch file

- then you use the patch file to fix your local file

feelin_googley3• 9 years ago

Not that I would know any better but I always saw a user-controlled approach built around rdiff as a better alternative than surrendering files to a non-transparent third party such as Dropbox (who, go figure, used librsync originally).

There is an elegant simplicity to rdiff, IMHO.

teraflop• 9 years ago

http://duplicity.nongnu.org/ implements this approach, FWIW.

bch• 9 years ago

ISTR they may have started w librsync but even years ago, their (modified, actively developed) lib bore little resemblance to the canonical librsync.

js22• 9 years ago

2005. So I guess this didn't catch on.

cbhl• 9 years ago

Ubuntu still provides zsync of its installation media: http://cdimage.ubuntu.com/ubuntu/releases/17.04/release/

That said, I'm not sure I know of any other major users of it -- most people just use a .torrent (which similarly has checksums of each piece so you know which pieces need to be downloaded).

vog• 9 years ago

Yes, this seems to have been replaced mostly by bittorrent.

On the other hand, HTTP has the advantage that it works through corporate proxies and that is usually not blocked by over-cautious admins.

dividuum2• 9 years ago

Not a major user, but we're using zsync for system updates of our Raspberry Pi based digital signage operating system (https://info-beamer.com/hosted). It's pretty great and offers a few things we couldn't do with bittorrent: Every time we have a new release we put together an install.zip file of everything required (kernel, firmware files, initrd, squashfs). Users can download this file directly and unzip it on their SD card and it will boot our OS. For updates we use a previous (see below) version of our install.zip already available on the device and only download the changes. We then unzip that into a new partition for an A/B booting.

Zsync is awesome as we can specify any number of existing files already available on the device (with the -i command line option) and zsync will try to make use of them to minimize downloads. We really use this feature to our advantage: zsync by default will keep the previous version of a file if it's going to overwrite it. So we have two versions of install.zip on a device. When switching between OS releases (stable / testing...) we can switch back and forth with zero additional downloads as both versions are available and zsync makes use of that. Similarly after a user installs our OS, we just have the unpacked artefacts (kernel, etc.) on the SD. We can quickly recreate an initial version of the install.zip file on the device by seeding the download with those files. It's usually just 500k to construct an initial install.zip file we then later use to minimize all future updates.

arjie• 9 years ago

A long time ago on a crappy restricted internet I used this and jigdo at different times to download the Ubuntu ISO. Being able to do so for Ubuntu and not other distributions was another factor in my using Ubuntu back then.

pmoriarty• 9 years ago

The AntiX Linux distro also provides zsync downloads:

https://sourceforge.net/projects/antix-linux/files/Final/ant...

eeZah7Ux3• 9 years ago

which similarly has checksums of each piece so you know which pieces need to be downloaded

Can you achieve differential downloads with bittorrent?

js22• 9 years ago

Thank you to the sibling comments for mentioning that you're using zsync. I'm sorry my comment was so negative.

just_astounded• 9 years ago

If you like rsync and javascript, I wrote https://github.com/claytongulick/bit-sync which is kinda fun.

Doman• 9 years ago

We are using this for addon synchronization in our community through Arma3Sync. On server side we need to "build" repository - it just generates .zsync files, then clients are downloading just diff. Update size came down from 10-15GB to <3GB.

Bromskloss3• 9 years ago

I've been thinking that we should replace "cp" with "scp", and then replace both of them with "rsync". Is that a bad idea?

Piskvorrr2• 9 years ago

For what use case? We do use `rsync` instead of `cp` in some capacity, even for local-to-local file copy - as there is slightly more verification of a successful copy, and the destination is a quirky flash medium. Not sure how SCP would help here.

Bromskloss3• 9 years ago

For all use cases.

Not sure how SCP would help here.

I mean that my train of though was first "scp, it seems, can do everything cp can, and more, so let's drop cp and use scp instead", then "but hey, rsync, it seems, can do everything that scp can, and more, so let's drop scp too and use rsync instead".

e12e• 9 years ago

There's also xdelta which is just an algorithm / program for calculating and applying binary diffs. I suppose the advantage of zsync is that you can always point "new" users to "/current.file", and use zsync to patch "up" to the latest version - with xdelta people would need to explicitly get "my..current.xdelta".

http://xdelta.org/

rakoo• 9 years ago

The big difference is that xdelta (much like any diff program) needs both the old and the new versions to create a patch. With zsync, the server only needs to have the new version (which is the only one of interest), and the clients then gets only the parts they want, because they only have the old version.

zsync also does all the fetching stuff directly.

iFire2• 9 years ago

I'm not related to Itchio.

If you are looking for a maintained system for online systems that provide software updates I would look into https://github.com/itchio/wharf-spec.

Wharf is used for Itchio to sync folder structures differentially / incrementally. It uses the latest compression algorithms. It has a reference server.

Alternatively, a port of zsync is https://github.com/salesforce/zsync4j written in Java.

I had trouble compiling zsync for windows.

eeZah7Ux3• 9 years ago

How is the wharf protocol better than zsync?

iFire2• 9 years ago

https://itch.io/docs/wharf/design-goals.html Describes the goals.

https://itch.io/docs/wharf/algorithms/diff.html and https://itch.io/docs/wharf/algorithms/apply.html describe patching

The most important thing is that Itchio runs a business on the usability of this system.

blacksqr• 9 years ago

Fossil (https://www.fossil-scm.org) is a SCM tool that uses the rsync algorithm for syncing repositories. It has a built-in web server, and can also be accessed via CGI from any CGI-capable web server. It also has a SSH option.

Its features make it very handy for a number of file transfer/sync tasks, over and above its chief SCM role.

a4dev• 9 years ago

Lennart Poettering has an interesting tool, casync , which overlaps with zsync. Claims to be a better solution for image syncing. Reasons given here: http://0pointer.net/blog/casync-a-tool-for-distributing-file...

Zsync: Differential file downloading over HTTP using the rsync algorithm (2010)

Related Stories