Apache Helix – Near-Realtime Rsync Replicated File System

https://helix.apache.org/0.6.8-docs/recipes/rsync_replicated_file_store.html

by GordonS • 6 years ago

296 191 6 years ago

helix.apache.org

bigbubba2• 6 years ago

How is this using rsync? Does it invoke an rsync process, or implement the rsync wire protocol? One of my gripes with rsync is I've not found another way to integrate it into other systems. librsync doesn't implement the wire protocol and parsing the output of rsync itself is fraught with peril (for instance the formatting of --no-h has changed before.)

Edit: it seems to invoke an a rsync process but doesn't parse stdout for progress. A bit disappointing but I suppose that's all which is needed for this application.

pengaru4• 6 years ago

librsync has absolutely no connection to the rsync utility, it's a distinct implementation of the underlying differential algorithm. The rsync utility doesn't use it, and they don't even share a common data format at any level.

bigbubba2• 6 years ago

Yes I'm aware, but usually when I voice my gripes with integrating rsync somebody unfamiliar with the matter quickly googles 'rsync library', finds librsync, and assumes it must be the answer to my problem. Then I have to explain to them that it's not. I thought to head that off this time by mentioning upfront that librsync is not the answer.

strbean• 6 years ago

rclone might hold some promise as an alternative, if you don't mind using Go.

rakoo• 6 years ago

Indeed there is no widespread documentation on how to play with rsync itself. A long time ago I played with the idea of replicating rproxy (https://rproxy.samba.org/), which was just a cool idea. The results are here (https://rproxy.samba.org/) but are very very basic, it was just a toy. I did have to invent my own protocol, but I did it over HTTP.

varispeed• 6 years ago

I used to run distributed system based on a few bash scripts and rsync to work in real time. It has become more complex when each "cell" exceeded the node capacity. The file naming system was not flexible enough to split these cells into smaller nodes and I abandoned writing a router for Nginx to direct requests to right server based on the filenames. I just decided to throw a couple of big machines at the problem instead and that removed all complexity. It wouldn't be possible just a couple of years ago though. For a couple of years the rsync solution worked very well.

3np3• 6 years ago

I'm using a distributed filesystem for persisted files (config, assets and resources, etc - anything that'd need to be in a docker volume and persisted between restarts) in my container workloads. This mostly works fine, except for sqlite applications that insist on using WAL - this completely breaks and results in corrupted database files.

Maybe this could be an alternative for those!

teraflop• 6 years ago

"Near-realtime" isn't good enough for SQLite. It has a strong dependency on the consistency guarantees provided by a POSIX-like filesystem.

The approach described in this page does make some attempt to provide consistency. The master generates a single stream of file updates, and all of the replicas consistently apply those changes in the same order. But rsync is not guaranteed to observe changes to files on the master in the same order that they were written.

In particular, SQLite (by default) uses a rollback journal to recover from failures. While a transaction is in process and some parts of the database might have been partially updated, the journal stores the old contents of the modified pages. Since the old data is fsync'ed to the journal before the new data is written to the database file, the data on disk at any moment in time is recoverable.

But there's no clean way for rsync to read an atomic snapshot of both the database file and the journal at the same instant in time. If it reads them at different times, a database change might be partially applied or partially rolled back, which has a high likelihood of corrupting the database.

jlokier3• 6 years ago

This won't work for database files, and it won't work for any other application that requires a filesystem to behave consistently over a network either.

The property you are looking for in a distributed filesystem is called strong consistency, or cache coherent (different name, same effect).

If a distributed filesystem does genuinely provide that property, it should be ok for SQLite and other applications, because it means the filesystem behaves the same as if it were multiple processes on the same local system accessing it.

Most distributed filesystems don't provide that guarantee though. It's a very nice guarantee, but it comes at a complexity cost, usually a performance cost (although there are clever ways to approach the perforance of a non-consistent system), and in particular it means the filesystems can't be used when the network connection is down.

There is another property called durability, which you might also care about. That affects whether you get corruption when devices fail or are rebooted suddenly.

3np3• 6 years ago

I actually don't need strong consistency here - I'll only have a single instance of the container accessing the sqlite file at a single point, but they can be running on any host. As long as I can ensure replication is synchronized before process start when a container gets rescheduled we good.

It goes over my head a bit exactly why (related to POSIX lock mechanisms?) but sqlite doesn't work reliable on NFS/ceph/gluster/networked filesystems, even when only accessed from a single host, distributed or not.

https://sqlite.org/forum/forumpost/4b340b81eb

pbowyer• 6 years ago

Most distributed filesystems don't provide that guarantee though. It's a very nice guarantee, but it comes at a complexity cost, usually a performance cost (although there are clever ways to approach the perforance of a non-consistent system), and in particular it means the filesystems can't be used when the network connection is down.

Which distributed file systems do provide this guarantee? I can't think of one, so hopeful to learn something new today.

traceroute662• 6 years ago

Minor rant: The trouble with all these lovely Apache projects is their dependency on Java.

If these things were written in C, Go or Rust then I'd leap at the opportunity to explore them and maybe use them in production.

But Java dependency just brings so much baggage with it. Its all well and good if you've got a Java pro or two on your team. But if you don't, you usually end up spending hours and days troubleshooting obscure JVM problems or somesuch.

ktpsns• 6 years ago

I second this so much! I have the feeling that the ASF tries to be so enterprisy with Java. I have no idea.

(Note that there are a lot of very high quality Apache java projects.)

pjmlp11• 6 years ago

One of their best features.

jaaron• 6 years ago

You have to go back almost 20 years to understand the relationship between the ASF and the Java community. Java at the time was positioned to be a core web technology (originally as applets, then as as backend tech via J2EE) and Apache was interested in ensuring the web ecosystem as a whole wasn't locked up in proprietary tech. The ASF was rather influential in working with Sun at the time to foster was eventually became a vibrant open source community.

I know the ASF isn't as trendy these days, but that's never really bothered those of us who volunteer. The focus has always been about a stable, open source ecosystem, primarily, but not exclusively for the internet and its underlying infrastructure. The goal is measured in decades.

It wasn't about being "enterprisey" or anything like that. The ASF as a foundation doesn't really care about the language or tech stack. Being a more mature non-profit, many corporations have found it easier to interface with the ASF than many other open source organizations out there, which does lend itself to seeing a lot of enterprisey donations.

buryat• 6 years ago

java is very easy to troubleshoot compared to c, go, and rust

traceroute662• 6 years ago

java is very easy to troubleshoot compared to c, go, and rust

All the worst and most time-consuming troubleshooting experiences of my life have involved Java. Whether staring at pages of obscure stacktraces that the JVM decided to vomit all over my screen, or hours on end with Oracle tech support troubleshooting why Java decided to throw a tantrum (followed by editing obscure Java XML config files).

These days I just stay well away from Java.

candiddevmike• 6 years ago

Can you expand on this? Why do you think that?

late2part2• 6 years ago

No, it's not. My experience is that for poor programmers, Java is easy because they lack the fundamental skills, and are familiar w/ the mechanics of Java. Of course this is a broad generalization and terribly unfair empirical anecdata.

seibelj• 6 years ago

Java is extremely old and stable. The obsession with Rust as open-source savior is overwrought. Rust may be good but Java has been around 25 years, give Rust another 10 years and 3 major language revisions before I will trust it.

johnisgood• 6 years ago

Thank you for pointing this out. I read the title and it sounded nice, but if Java is anywhere near close to it, then no thanks. Thank you, you saved me from wasting time. :)

chuckSu• 6 years ago

While agree that some of the Apache projects would be better written in something other than Java but the comment about spending hours debugging obscure Jvm issues is huge exaggeration

kinow• 6 years ago

You can check if there is a project like that in the incubator, or suggest one if you have an idea or some code to donate.

Most new projects are related to existing ASF projects, or use ASF libraries, or are created by someone already in the ASF that maybe use/prefer Java.

But there is no requirement on a project being Java.

See Airflow, Arrow, Log4net, httpd... it's just a question of people creating the proposals that must show a possible community of users/devs to use/support it.

geodel• 6 years ago

Well Apache and Java are like GUI and JS. I'd be shocked if most of collaboration, productivity, editing tools etc is not JS/Electron today.

Not saying that either is good, but it is just fact of life.

raz32dust2• 6 years ago

What baggage is specific to Java? You'd want some experts for the language your critical services are written in, regardless of it being java or C or Rust or whatever. Likewise, a java shop wouldn't want to use a critical service written in C.

klodolph5• 6 years ago

For Java you need people in operations who are good at things like JVM tuning working. Just a fact of life, speaking from experience working on teams that do that.

Languages like C and Rust do not have any equivalent to JVM tuning.

nfg3• 6 years ago

The JVM that Java shop is using is likely written in C, which should make you question the straight equivalence you’re setting up. There is an operational difference between getting a native executable and having something that sits on top of Java (or any other managed platform).

rodgerd3• 6 years ago

For me, at this point, the biggest baggage is dealing with Oracle and funneling them money if you want an LTS JVM, alongside the shifting sands of their licensing rules. Unfortunately it can still be a bit of a wheel of fortune as to how well (if at all) something runs on OpenJDK.

hedora5• 6 years ago

Between the open-world assumption the classloader uses, and the lack of a performant, general purpose garbage collector, you can’t seriously consider deploying Java at scale without having experts on hand.

I’ve deployed things written in many languages at extremely large scale. They’ve been written in Java, C, C++, python, go, sh, perl, and other languages I’ve forgotten. Java is, by far, the least operable of those languages.

I’m an expert Java developer, along with most of those other languages (where expert is defined by me as “over 10 years experience”).

In fairness to java, it’s not my least favorite language on the list.

shock14• 6 years ago

It would be useful to you to recognize that you are letting an emotional reaction stopping you from using useful technology. I understand that Java makes you uncomfortable, but it is a powerful and stable language and you're missing out by avoiding it. It would take much more time to implement a similar project in C, and it would likely have lower stability. Yes, implementing it in C would use a lot less memory and if that's what you're optimizing for going with something else than Java is a wise choice most of the time.

Most technology is useful, depending on what you're optimizing for. I urge you not to optimize for being comfortable.

breakfastduck4• 6 years ago

It's not an emotional reaction they were expressing at all - they even explicitly stated it's because of the need to bring in someone savvy with Java and the JVM to be able to support it. Sometimes there aren't the resources.

0xEFF• 6 years ago

OPs points are valid. Filesystem work requires experience with system calls and I/O behavior. Java strives to abstract the underlying system away from the developer. System level tooling written in Java frequently requires twice as much expertise because there are two big systems in play instead of one, hence the unnecessary baggage.

inopinatus• 6 years ago

The language isn’t the problem. The runtime is the problem. It’s a dependency mess, a licensing lottery, and a tuning crapshoot. The overhead of simply making it work, and maintaining that status across heterogeneous environments, is more than non-Java shops want to undertake.

Having spent much of the first half of my career writing C and Perl and then Java, I was very happy to set aside the latter two - for completely differing reasons.

67868018• 6 years ago

Java is a significant legal risk. Do not let it into your company.

Uberphallus• 6 years ago

Java is great, the JVM is hell.

Our shop has about the same Cassandra DBAs, as JVM experts to keep it running.

Every time a node goes down because of JVM shenanigans the ScyllaDB migration discussion pops back.

jayd162• 6 years ago

I don't think Java adds any more things you must tune so much as it adds a lot of things you can tune.

You can just treat these appliances as a black box if you can't be bothered to learn the tooling.

Out_of_Characte• 6 years ago

Untill its not, Then I have a problem because I dont understand java

UMetaGOMS• 6 years ago

Cannot upvote this enough. I'm sure a frequent Java user will come along to tell me how easy it is - but at this point I'm not interested in delving into their world. If a new application I'm looking at is a single binary I can drop on my system (a la Go/Rust) then great - otherwise it has to be something I really need for me to not just resort to "whatever is in the package manager, or find a different solution".

francislavoie4• 6 years ago

I avoid ElasticSearch for that reason.

a_imho• 6 years ago

If you use Docker or similar for deployment you can have ~static executables ootb.

pjmlp11• 6 years ago

That exactly what I like from Apache projects, one step at a time to make C less relevant.

StreamBright• 6 years ago

This is not the most serious problem, by far.

rodgerd3• 6 years ago

If JVM optimization is "too hard" then you're going to have a bad time trying to run a distributed computing system.

x87678r• 6 years ago

True but if it was written in C it never be completed and would likely crash all the time. :) Just be thankful they're not Python.

zmmmmm2• 6 years ago

Funny how I'm the complete opposite. For me, seeing it is written in Java is a big plus.

As soon as I see Java I know it's going to have a whole set of properties that will make it highly manageable to deploy and run. And when they go wrong I have a lot of hooks and tools to understand and resolve the issue that are completely unavailable to me with native applications. You apply the pejorative "baggage" to these as if they are valueless but sometimes you just have to develop enough experience before the utility of things becomes apparent.

(Writing an application in Java however ... I would go straight for a JVM language like Kotlin/Groovy/Scala)

vips7L2• 6 years ago

Personally I never want to touch Scala again. Implicit and operator overloading hell. Let alone the shit show that is sbt and c++ compile times from scalac.

spenczar5• 6 years ago

Both of you are right. You happen to be a “Java pro” who knows how to use the tool well. It’s a bit uphill for shops without Java experience - more uphill than with other languages, perhaps.

chrisjc2• 6 years ago

One of the most important and valuable (at least to me) things about the Java and the JVM is its ecosystem. And Apache is a part of that ecosystem. BTW a lot of the Apache projects aren't even Java or depend on the JVM.

If these things were written in C, Go or Rust...

Do these languages have similar ecosystems?

francislavoie4• 6 years ago

Go and Rust have very good package management (Go's kinda sucked but go modules is helping a lot), so yes. They both work much nicer than the occasions I've used maven etc.

jwr• 6 years ago

Unless you are happily using a JVM-hosted language of course (Clojure in my case), in which case it is a huge advantage. I can use this immediately and easily in production.

jjtheblunt2• 6 years ago

I wonder if folks are more averse to Maven than Java

shock14• 6 years ago

I don't see a reason for users to be aware of Maven, similarly when you get a C, Go, app, you are not aware of the build tool that was used to create it.

lmm5• 6 years ago

Java is pretty much the only language I'd want to see low-level libraries written in. Go libraries can only be used from Go, Rust libraries can only be used from Rust, and any nontrivial C library contains an arbitrary code execution vulnerability when built with a newer compiler. Java is a lowest-common-denominator language but it's memory-safe, cross-platform (in a way that C# isn't really yet), and there's a decent range of languages that you can run on the JVM and use Java libraries from.

francislavoie4• 6 years ago

Go libraries can only be used from Go

https://github.com/draffensperger/go-interlang

Rust libraries can only be used from Rust

https://rust-embedded.github.io/book/interoperability/rust-w...

And any language that supports an FFI interface can call rust code. For example you can write a rust lib and call it from PHP using it's recently released FFI https://www.php.net/manual/en/book.ffi.php, e.g. https://dev.to/verkkokauppacom/introduction-to-php-ffi-po3

FridgeSeal• 6 years ago

I agree completely.

I love using things written in Rust, because if nothing else I know it's going to be resource efficient and the deployment will be nice and easy.

Java applications: I hope there's enough spare memory, I hope the setup instructions aren't littered with "set this obscure JVM config in some obscure way, that isn't the application config, but we won't tell you how/where, you've just got to figure that one out"

shmerl• 6 years ago

I also wonder why are they using Java? Filesystems should be written in something like Rust. Using languages like Java for systems programming always looked bizarre to me.

ir123• 6 years ago

Curious, what obscure JVM problems have you encountered? My experience with Java has been that it is easy to get started and work with a codebase.

lutoma• 6 years ago

Also, every single Java application I've ever used has used an entirely unreasonable amount of memory for the job it's supposed to do. I'm sure you can write things in Java that are lightweight on resources, but I've never seen it.

redm• 6 years ago

I wonder how near-realtime this is. It seems that will dictate the types of applications this will be applicable for.

I currently use Gluster which has its downsides, but is what I consider near-realtime, but with full filesystem features in a nice fuse wrapper. I'd love it if someone would contrast the performance and more of the features.

late2part2• 6 years ago

In the mid-2000s (the "oughts?") I went to my doctor at PAMF with a sore throat and he said he would do a strep test.

I asked him if we would get "real time results?"

He stopped, and looked at me puzzled with a tilted head... "What other kind of time is there?"

Your question is highly reasonable, but I heard a colleague today say that the issue could happen only in a very, very short time period of, say, 2-3 minutes. We routinely work with corner cases of 1-2 ms; so the context is relative :-)

jjtheblunt2• 6 years ago

"context is relative" made me smile, thinking "PAMF" is Palo Alto Medical Foundation, but why would anyone in general have the context to understand that

StavrosK• 6 years ago

What are the downsides of Gluster?

rodgerd3• 6 years ago

Gluster's performance with metadata heavy workloads is very poor. If you're running files with a high ratio of data:metadata, it works well; if you're running something like a git repo on Gluster it is not a great experience.

It's also thrashing a bit development wise; there's a lot of features being pruned, and I'm a little concerned what its future looks like as Red Hat push harder down the Ceph route, since they're the main source of code contributions to Gluster.

par• 6 years ago

Looks like a great way to keep my photos, videos and documents backed up :)

coffeeri• 6 years ago

I wouldn't consider this as a backup solution, more as one for high-availability of data.

mceachen• 6 years ago

Please don't use this as your only backup. Consider it an experimental cross-machine RAID.

You'll adopt the oft-repeated "RAID isn't a backup" mantra as soon as any of these issues hit you (like several have hit me): https://photostructure.com/faq/raid-is-not-a-backup/#why-isn...

gfodor2• 6 years ago

This might be an appropriate thread: does anyone have a OSS alternative to EFS they have run in production? (A NFS-mountable, elastic storage system.)

cyberpunk• 6 years ago

Leofs

JoshTriplett• 6 years ago

Does it specifically need to be NFS, or just mountable?

gfodor2• 6 years ago

Just mountable

gluteousminimus• 6 years ago

In the HPC space, Lustre.

pengaru4• 6 years ago

A major problem with rsync is it relies on filenames matching at the source and destination for an accelerated delta transfer to occur.

If a file gets renamed or copied, modified or not, rsync will transfer the whole file.

They added --fuzzy to try improve on this situation, but it generally only helps with unmodified files copied/renamed within the same directory.

You can build robust solutions around rsync that work efficiently most of the time, but pathological cases such as users having a huge file regularly regenerated using something like a $(date --iso-8601=seconds) filename (think mysqldump backups) are surprisingly common once you have enough users. Even if you can convince them to uncompress such files for delta transfer sake, something as common as a versioned filename prevents rsync from automatically finding the relationship for use in the delta transfer.

catmanjan2• 6 years ago

How else could they do it? All systems I've seen that have relied on files on a file system use the file name as the "primary key"

I believe you can use journalling to detect file name change events but I don't think it scales well

pengaru4• 6 years ago

Who is "they"? Are you asking about rysnc developers or filesystem developers?

For a filesystem the developers are free to innovate on any form of filesystem-wide block checksumming and content-addressed deduplication approach. This kind of global block-level content awareness is entirely decoupled from filenames and paths.

Using rsync to underpin a filesystem strikes me as a gross hack done by developers punting on the real challenges of modern filesystem development.

WRT rsync doing better than --fuzzy, the options are more limited since it's a userspace tool working within the confines of filesystem APIs like POSIX.

ComodoHacker• 6 years ago

For example, zpaq splits files in chunks and hashes them, so it can deduplicate across files.

BatteryMountain• 6 years ago

So we need a new tool that is basically git and rsync rolled into one. Might be easier to use something like zfs and use it's features to build a tool that behaves almost like git. Then a (zfs + rsync + [zfs-git hybrid]) monster is born, which may or may not work.

Fyi have a look at zfs. It has some neat traits/behaviours. So my thinking is, someone should write a database that uses gazzilions of files (instead of a file per table), a tiny file for each cell/field in a table. Then use zfs for versioning of data (or git). Zfs can already do software mirroring to drive pools on the machine, so maybe add rsync for remote pools. Optimize it for SSD's and you have a filesystem-database-replicating-versioned -based store.

Maybe just need a nice cli/gui for zfs to make it easier/safer to expose more of it's interesting features.

zcw100• 6 years ago

“So we need a new tool that is basically git and rsync rolled into one.”

That sounds a bit like IPFS

jiggawatts• 6 years ago

Windows Server has a variant of the Rsync algorithm that it uses for its Distributed File System Replication (DFS-R) that can look for up to 4 similar files. It works okay-ish in practice, and is better than nothing.

ysleepy• 6 years ago

This is a cluster mgmt system that also has a file sync feature built in, probably to deploy binaries and data to run jobs.

https://helix.apache.org

The link here is to the docs of version 0.6.8 while 1.0.1 is current (after 0.7, 0.8 and 0.9).

This also explains the heavy weight architecture, you probably are running a ZooKeeper anyway in that environment.

tashbarg• 6 years ago

To facilitate this, the master logs each transaction in a file and each transaction is associated with an 64 bit ID in which the 32 LSB represents a sequence number and MSB represents the generation number The sequence number gets incremented on every transaction and the generation is incremented when a new master is elected

So, what happens after 32bit transactions without re-election?

fulafel• 6 years ago

How do I mount this? Or is it like s3, inaccessible as a normal FS?

mperham• 6 years ago

I use Resilio Sync in production with my servers. I like it more than this solution because it does not require a primary and replica design but instead uses bit torrent where all servers are equal. I can add a new server and drop an old server and it handles everything gracefully.

GordonS2• 6 years ago

Haven't come across this before. At a glance, it looks like it's a file sync tool for small scale use, rather than a "serious" tool for replicating a file system.

Curious to know more about your production use case?

minaguib• 6 years ago

It looks like a more distributed alternative to lsyncd ( https://axkibe.github.io/lsyncd/ )

orasis• 6 years ago

What is the point of rsync if the files aren’t updated?

hedora5• 6 years ago

It’d be nice if something like this was built atop unison.

stephenr2• 6 years ago

Unison is only intended for two "points" isn't it? i.e. it's meant for pretty narrow range of "client : server" sync jobs, and is essentially unusable without exactly the same patch version on both ends.

For purely server-side "cluster" sync I've found csync2 to be a reasonable solution.

riejo• 6 years ago

When it's 2020 and you label your boxes master/slave.

butterisgood• 6 years ago

Yeah a lot of people are moving away from this terminology thankfully!

anaganisk• 6 years ago

Im not sure if you're being sarcastic or if it really matters that we call it matters and slave. Are we so easy to get triggered?

stephenr2• 6 years ago

As I haven't seen it mentioned yet, if you want bi-directional sync of changes with relatively low overhead, I've found csync2 to be reasonably good for this task.

Apache Helix – Near-Realtime Rsync Replicated File System

Related Stories