A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes

https://juicefs.com/en/blog/engineering/reduce-metadata-memory-usage

by LittleCat38 • 2 years ago

140 66 2 years ago

headline.append(" Per File")

    Uncaught TypeError: headline.append is not a function

thwarted• 2 years ago

This is implied by "average", assuming it's understood that a filesystem's primary unit is the "file".

LittleCat38• 2 years ago

Agree

rat9988• 2 years ago

Actually, the average here just implies that tested in different configurations and scenarios, it should cut by 100 bytes in averages. Net per file. Adding "per file" does have a benefit.

tonyhb• 2 years ago

I'd like to learn more about JuiceFS, but from their architecture diagrams I'm struggling to see what benefit they provide if they're a layer over blob store systems like Ceph, MinIO, etc.

It looks as though you need to set up the underlying storage engine that has its own recordkeeping (one point of failure), then layer JuiceFS on top — another point of failure, and I don't know what it gives you.

It's good that it's fast, but if it's pointing to another blob store you'd... expect it to be fast? Is this only needed if you want properly faked file system primitives over blob stores, if you can't use blob stores directly? Definitely need to spend some time reading.

daviesliu7• 2 years ago

JuiceFS is similar to HDFS/CephFS/Lustre, so it MUST has a component to manage metadata, similar to NameNode of HDFS or MDS of CephFS, this point of failure is the problem we have to address.

The underlying blob store systems is similar to DataNode or OSD in other distributed file system, could be slower than them a little bit because of the middle layers, the overall performance is determined by the disks.

So we can expect similar performance comparing to HDFS/CephFS, the benchmark results also confirm that.

thinxer• 2 years ago

For “cloud-native” apps, JuiceFS is not needed.

S3 is not designed for intensive metadata operations, like listing, renaming etc. For these operations, you will need a somewhat POSIX-complaint system. For example, if you want to train on ImageNet dataset, the “canonical” way [1] is to extract the images and organize them into folders, class by class. The whole dataset is discovered by directory listing. This where JuiceFS shines.

Of course, if the dataset is really massive, you will mostly end-up with in-house solutions.

[1] https://github.com/pytorch/examples/blob/main/imagenet/extra...

snerbles2• 2 years ago

Is this only needed if you want properly faked file system primitives over blob stores, if you can't use blob stores directly?

Semiconductor EDA comes to mind, where users are at the mercy of their tool vendors and they expect something that really hasn't evolved much past an office network of Unix workstations from 1999. Object storage is almost completely alien to the tooling, and despite significant file & block storage costs there is little interest from EDA tool vendors to adapt their tools to object storage. This is a challenge for semiconductor firms operating in or moving to a cloud environment.

S3FS/rclone can of course act as a shim, but are very slow when it comes to metadata operations in a typical shell. But if you were to move your metadata away from the distant object store and closer to to your compute environment things actually start becoming usable - this is the case with JuiceFS. Of course there are also tiered storage systems like Weka would have better overall performance, but is more complicated to set up and more expensive to operate than JuiceFS.

KaiserPro5• 2 years ago

S3's metadata speed is horrifically slow. Part of the reason fake filesystems on S3 also perform badly is that checking anything to do with where a file is located, takes >250ms.

Now if you think about how many files are in a git repo, and how many are touched when you commit, you can see the problem.

I'm not sure JuiceFS is the answer, given that the metadata is stored in memory. but it is an answer.

personally you're better off with their managed lustre offering

Geisterde3• 2 years ago

Is this something thats a deficiency with S3, or is there not a more purpose built offering?

seungwoolee5183• 2 years ago

I've put this on table for distributed storage on Kubernetes.

It's quite simple in front, simple to explain that split object and metadata storage.

but it's too hard to select and tune each storage.

So I (or We)'ve decided to move to Ceph.

daviesliu7• 2 years ago

Agreed, the all-in-one solution (Ceph) should be better, if you have to setup all the components.

If you already have the infra (databases and object stores), then JuiceFS is the easiest solution to have a distributed file system.

KaiserPro5• 2 years ago

Ex HPC person here.

1) never cluster unless you have to.

2) never use distributed Filesystems unless you need a global namespace

3) distributed filesystems mean that you have a single point of failure, its just more complex and subtle form of failure.

4) You have to think about how partition works, and how you want to recover.

Sharding standard NFS fileservers is by far the easiest way to scale, have redundancy, and partition your failure zones. using autofs on a custom directory you can mount servers ondemand. /mnt/0 /mnt/1 /mnt/2 etc etc etc.

You then have the ability to chose your sharding scheme for speed, redundancy or some other constraint.

Nowadays the performance that distributed systems allowed are commodity. A single 4u storage server can saturate a 100gig network link with random disk IO.

seungwoolee5183• 2 years ago

We've used a some of Distributed storage (Longhorn, OpenEBS) things for Kubernetes and had no luck.

I agree with you at ALL the things you've listed. When we've started to used Ceph for the first time - (At that time I was an user, not a Manager) all the thing was a total mess. All the things are set to default and Use all JBODs on Node. (And I've started to face Corrupted WALs and OSDs came from nowhere.)

The handful of documentations was very helpful to cleans up and make it to run correctly.

Maybe, when I go back to the starting point, I'd like to say "move to cloud" or "do nothing" or following your guidance.

seungwoolee5183• 2 years ago

Anyway, thanks for a helpful guidance to keep it simple on cloud Era.

mike_d2• 2 years ago

I'll never understand the "we are open source but lock $cool_but_essential_features behind an enterprise license."

Unless you plan to charge less than what a single developer makes in a quarter or two, any of your potential customers could at any point patch in the feature and open source the changes for everyone.

(Traefik and Riak are the two biggest offenders I've ran into)

MoOmer• 2 years ago

Those patches are unlikely to be officially supported, though. I really support this mode of monetization, because community/free users get something, well, free, while others get something more for a bit more. It’s [hopefully] sustainable for the engineers and team behind the product, too!

mike_d2• 2 years ago

So sell support? Seems to work really well for a lot of companies.

1propionyl• 2 years ago

You don't pay for features. You pay for support.

dspillett• 2 years ago

Well, yes. But also no. If you need/want a feature that is only included in the non-F/OSS release, then you are not just paying for the support.

umanwizard• 2 years ago

In theory you’re right, but in most companies the friction of spinning up a team to manage the custom patch set of each piece of open-core software they use would dominate the actual effort needed to implement the features. It’s just not in most company’s DNA to operate this way.

(Also, a lot of SaaS is in fact cheaper than what a dev makes in half a year).

nijave• 2 years ago

Cloud providers are making a killing on their ability to spin these types of teams up

amluto4• 2 years ago

JuiceFS, written in Go, can manage tens of billions of files in a single namespace.

At that scale, I care about integrity. Can someone working in this space please have a real integrity story as part of the offering? Give each object (object, version pair, perhaps) a cryptographic hash of the contents, and make that hash be part of the inventory. Allow the entire bucket to opt in to mandatory hashing. Let me ask the system to do a scrub in which it verifies those cryptographic hashes.

If this blows up metadata to 164 bytes per object, so be it. But the hash can probably get away with being stored with data, not metadata, as long as there is a mechanism to inventory those hashes. Keeping them in memory doesn’t seem necessary.

Even S3 has only desultory support for this. A lot of competitors have nothing of the sort.

snissn• 2 years ago

do you use the hash of the items you're storing as the key/filename?

daviesliu7• 2 years ago

JuiceFS relies on the object store to provide integrity for data. Besides that, JuiceFS stores the checksum of each object as tags in S3, and verifies that when downloading the objects.

Inside the metadata service, it uses merkle tree (hash of hash) to verify the integrity of whole namespace (including id of data blocks) between RAFT replicas. Once we store the hash (4 bytes) of each objects into metadata, it should provide the integrity of the whole namespace.

amluto4• 2 years ago

Does JuiceFS allow the user to specify the hash of a file when uploaded? And then to read that hash back later?

Otherwise there’s no end-to-end integrity check.

jorticka• 2 years ago

RocksDB supports hashing at multiple levels (key, value, files) because Meta also realized the importance of integrity. It also supports verifying them in bulk.

Presumably filesystems built over rocksdb also support this.

fh973• 2 years ago

The Quobyte DCFS does end-to-end CRC32 for each 4k block of data. All metadata and communication is also CRC protected, although one other frame boundaries.

EGreg• 2 years ago

IPFS, MaidSAFE and even BitTorrent have had this from day 1, and they are distributed and not under the control of one company.

FileCoin is used to pay for storage, but for now it seems storage is mostly free because of block rewards.

kfrzcode3• 2 years ago

And they are slow, expensive and subject to node-based 33% attacks

Hedera Hashgraph is none of these

KaiserPro5• 2 years ago

The next bit:

Its metadata engine uses an all-in-memory approach and achieves remarkable memory optimization

Sounds like a fun thing to recover after a crash.

ted_dunning• 2 years ago

They have a blog on that:

https://juicefs.com/en/blog/engineering/increase-performance...

daviesliu7• 2 years ago

The recover process is similar to what a database does after a crash, it loads the recent snapshot in disk and apply any newer transaction logs.

kfrzcode3• 2 years ago

I think you could achieve this on Hedera's hashgraph - ABFT DLT

yjftsjthsd-h• 2 years ago

I was under the impression that minio with erasure coding did that? ( https://min.io/docs/minio/linux/operations/concepts/erasure-... )

amluto4• 2 years ago

Looks like minio added this in 2022:

https://github.com/minio/minio/pull/15433

styluss• 2 years ago

Techniques like memory pools, manual memory management, directory compression, and compact file formats reduced metadata memory usage by 90%.

It is interesting that high performance systems in Go end up needing these kind of performance optimizations.

Intermernet• 2 years ago

High performance systems in any language require these optimisations. Go provides a solid foundation to create the platform before you start optimising. Not saying it's the best or only choice, but it's not a bad choice.

Karellen• 2 years ago

High performance systems are where these kinds of optimisations make the most difference.

If most of your overhead is due to the low performance of the language/runtime you're using, these kinds of optimisations won't make as much of a difference. That is, if you can even implement them at all. I mean, good luck trying to use memory pools and manual memory management in Python or Javascript.

afr0ck• 2 years ago

You're implementing a Slab allocator. That's exactly what any Slab allocator does, including gmalloc, tcmalloc, jemalloc and SLUB (Linux kernel). But, there is probably much much more you can do on a modern machine to squeeze even more performance. Some of the things that comes into my mind are: reducing tlb stalls with hugepage awareness[1], and reducing false cache sharing on SMPs [2].

For the compression part, have you thought about OS-managed memory compression [3][4]?

[1] https://google.github.io/tcmalloc/temeraire.html

[2] https://people.freebsd.org/~jasone/jemalloc/bsdcan2006/jemal...

[3] https://dl.acm.org/doi/pdf/10.1145/3297858.3304053

[4] https://www.kernel.org/doc/Documentation/vm/zswap.txt

gnarlouse• 2 years ago

In production, 10 metadata nodes each with 512 GB of memory collectively manage over 20 billion files.

That doesn’t sound all that impressive—is it impressive?

hansvm• 2 years ago

They're only using 36% of that (so they have slack capacity). It's in-memory, which has some perf wins (though I'd question if distributed systems are worth it when you can get a PCIe splitter and just mmap some 4TB SSDs on whatever cheap desktop computer somebody has lying around).

100 bytes per file is less than you would expect to support all that data if you coded it "naturally" without an eye toward space efficiency. You'll need 16-32 bytes or so just to handle a tree structure with internal string indices to be able to support finding any given file's location quickly. Toss in created times, modified times, permissions, attributes, the fact that they have a chunking structure to support both random seek and efficient/easy allocation, .... It adds up to more than 100B if you're not pretty careful. They hit that 100B mark.

Is that impressive? I dunno. Their competitors didn't do it, and they did, so that's something.

biomcgary• 2 years ago

As a computational biologist, I have a lot of embarrassingly parallel problems, so the lightweight concurrency of Go is really easy. In addition, my algorithms and data perform comfortably in GC-land 99% of the time, reducing my mental overhead (for science!). I had to reach for unsafe.Pointer() only once and that was to use Other People's Code (TM).

I consistently find my code runs 20-80x faster than the equivalent in Python and I'm not about to cram C++ or Rust into my head along with all the biology I've memorized for ~2x performance improvement and >5x learning curve. On HN, I shouldn't need to explain why I don't use Java despite a similar set of tradeoffs to Go (other than memory use, historically).

ramses0• 2 years ago

Competitive in this space is also SeaweedFS, which I've taken out for a local spin. It feels kindof like "memcached-for-fs/s3"... you add 30GB chunks of storage and it coalesces them for you without much ceremony. ~20 bytes storage overhead per file and performance seems pretty decent.

tehcopec• 2 years ago

I’ve been using community edition JuiceFS with Percona XtraDB cluster for Metadata and MinIO multi-node for a couple of years for large archival and backup data storage. That setup has worked really well.

A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes

Related Stories