Demystifying OpenZFS 2.0

https://klarasystems.com/articles/demystifying-openzfs-2-0/

by vermaden • 5 years ago

159 52 5 years ago

klarasystems.com

john37386• 5 years ago

What is the difference between Klara and IXSystems? Are they competitor, partners or something else?

vermaden• 5 years ago

Just two different independent companies. Like Netflix and Apple TV for example.

magicalhippo7• 5 years ago

To add a bit more, iXsystems makes enterprise NAS systems, and develops TrueNAS (formerly FreeNAS and TrueOS) which has ZFS as one of it's main features.

Klara systems is more of a development and support consultancy firm for FreeBSD and ZFS.

willis9362• 5 years ago

Competitors, but they both participate in OpenZFS development. Yet another success story for open standards.

https://www.ixsystems.com/blog/openzfs-2-on-truenas/

thijsvandien• 5 years ago

Competing how, exactly?

bhouston9• 5 years ago

EDIT: There is a now an incorrect response to this post of mine stating that I forgot to mention the ZIL SLOG -- but the poster doesn't understand how the ZIL SLOG actually works unfortunately. :(

I applaud OpenZFS for these major improvements, but its performance is not competitive with some of the commercial offerings, even if OpenZFS is a more easily administered solution.

The main issue with OpenZFS performance is its write speed.

While OpenZFS has excellent read caching via ARC and L2ARC, it doesn't enable NVMe write caching nor does it allow for automatic tiered storage pools (which can have NVMe paired with HDDs.)

This means that large writes, those which exceed the available RAM, will drop down to the HDD write speed quite quickly instead of achieving wire speed. Which means I get write speeds of 300 MBps versus 10 GBps wirespeed.

The solutions competitors have implemented are:

- SSD read-write caching, usually using mirrored NVMe drives to ensure redundancy.

- SSD tiering, which is including your SSDs in your storage pool. Frequently accessed data is moved onto the SSDs while infrequently accessed data is moved to the HDDs.

Without these features, my QNAP (configured with SSD tiering) kicks the pants off of ZFS on writes that do not fit into memory.

References:

[1] QNAP Tiering: https://www.qnap.com/solution/qtier/en-us/

[2] QNAP SSD Read-Write Caching: https://www.qnap.com/solution/ssd-cache/en/

[3] Synology SSD Read-Write Caching: https://kb.synology.com/en-my/DSM/help/DSM/StorageManager/ge...

Intermernet• 5 years ago

Is QNAP doing something outside the ZFS spec here? If not, it's almost certain that solutions like TrueNAS can replicate this behaviour. If they are, I'd be concerned about vendor lock-in and data recovery.

I'm assuming that QNAP are using ZFS, which a cursory search supports.

bhouston9• 5 years ago

I'm assuming that QNAP are using ZFS, which a cursory search supports.

QNAP has two NAS OSes right now. The newer one is QuTS and it support ZFS. The other older one is QTS which doesn't use ZFS. Only the older QTS OS supports SSD Tiering.

Is QNAP doing something outside the ZFS spec here? If not, it's almost certain that solutions like TrueNAS can replicate this behaviour. If they are, I'd be concerned about vendor lock-in and data recovery.

I am using TrueNAS for my ZFS solution and it doesn't offer anything out of the box to address the issues I am bringing up.

tw046• 5 years ago

The main issue with OpenZFS performance is its write speed.
While OpenZFS has excellent read caching via ARC and L2ARC, it doesn't enable NVMe write caching nor does it allow for automatic tiered storage pools (which can have NVMe paired with HDDs.)

Huh? What are you talking about? ZFS has had a write cache from day 1: ZIL. ZFS Intent Log, with SLOG which is a dedicated device. Back in the day we'd use RAM based devices, now you can use optane (or any other fast device of your choosing including just a regular old SSD).

https://openzfs.org/w/images/c/c8/10-ZIL_performance.pdf

https://www.servethehome.com/exploring-best-zfs-zil-slog-ssd...

magicalhippo7• 5 years ago

Huh? What are you talking about? ZFS has had a write cache from day 1: ZIL.

The ZIL is specifically not a cache. It can have cache-like behavior in certain cases, particularly when paired with fast SLOG devices, but that's incidental and not its purpose. Its purpose is to ensure integrity and consistency.

Specifically, the writes to disk are served from RAM[1], the ZIL is only read from in case of recovery from an unclean shutdown. ZFS won't store more writes in the SLOG than what it can store in RAM, unlike a write-back cache device (which ZFS does not support yet).

[1] https://klarasystems.com/articles/what-makes-a-good-time-to-...

bhouston9• 5 years ago

Huh? What are you talking about? ZFS has had a write cache from day 1: ZIL. ZFS Intent Log, with SLOG which is a dedicated device. Back in the day we'd use RAM based devices, now you can use optane (or any other fast device of your choosing including just a regular old SSD).

You are spreading incorrect information, please stop. You know enough to be dangerous, but not enough to actually be right.

I've benchmarked ZIL SLOG in all configurations. It doesn't speed up writes generally. It speeds up acknowledgements on sync writes only. But it doesn't act as a write through cache in my readings and in my testing.

What it does is it allows a sync acknowledgements to be sent once a sync write is written to the ZIL SLOG device.

But it doesn't actually use the ZIL SLOG for reading at all in normal operation, instead it uses the in-memory RAM to cache the actual write to the HDD-based Pool. Thus you are still limited to RAM size when you are doing large writes -- you may get acknowledges quicker on small sync writes, but your RAM limits the size of large write speeds because it will fill up and have to wait for it to be saved to the HDD to accept new data.

Here is more data on this:

https://www.reddit.com/r/zfs/comments/bjmfnv/comment/em9lh1i...

reacharavindh5• 5 years ago

Isn't that what ZIL(Intent Log)/SLOG is for? I have a huge ZFS server (> 1 PB usable ZFS storage) that runs with 768 GiB of memory and 480 GB of Intel Optane as SLOG. All our writes are sync by virtue of using NFS from our compute cluster. I'm sure we see better write speeds, and not wait on spinning disks for write IOs.

formerly_proven• 5 years ago

Are you using a single ZFS server as the primary storage system for the cluster? If so, how do you handle availability? Do nodes have local scratch space or is it a do-everything-on-the-cluster-FS kind of setup? How is this setup treating you (administration, "oh no something is slow / hanging / broke, whatdoido", availability, performance, especially on those heavy days)? Is this some kind of COTS solution supported by one of the usual vendors or did you roll it yourself? Being ZFS I'm guessing it is far better at dealing with small files, and with directories containing many files than clustered FS which fall over pretty easily in these cases?

e: Not doing market research here, I'm just seeing (sometimes, feeling) all the pain involved with cluster filesystems and wondering how your solution stacks up.

magicalhippo7• 5 years ago

Isn't that what ZIL(Intent Log)/SLOG is for?

No, the ZIL is just for integrity[1].

[1] https://news.ycombinator.com/item?id=29266628

myrandomcomment• 5 years ago

https://jrs-s.net/2019/05/02/zfs-sync-async-zil-slog/

ZIL and SLOG are NOT a write cache. They are 100% about not loosing data. A ZIL is part of a zpool where data is written to when the file protocol requires sync. When it is written the sync is cleared. In this case you are sharing the zpool with all its others needs to write and read, so more load on the pool. The data is also in RAM given to the write aggregator process. The RAM is only written latter. If the system crashes before the aggregator writes the data, on restart the data written to the ZIL is read into the aggregator and then is written out to the zpool. By default the ZIL is never read. An SLOG is just a ZIL that is on a separate device from the zpool. So a sync write comes in and is written to the SLOG and not part of the spinning rust zpool so no shared resources with the pool. The sync is marked as cleared when the data is written to the SLOG. The SLOG should be really fast as the goal is to ack the sync quickly. Remember at this point the data is NOT written to the zpool, it’s RAM and the aggregator will write the data out later. Same thing applies on crash. Reboot, SLOG is reviewed for uncleared items, sent to RAM, written to pool.

TLDR - zil/slog are write devices where the data is written when sync is required and almost never read back. Data is alway in RAM cache and written out to the spool from RAM. ZIL/SLOG is not a buffer. Speed of ack sync will be as fast as data can be written to ZIL/SLOG.

gorkish• 5 years ago

ZLOG/ZIL devices journal writes; they have always been in zfs. SSD/NVMe with power loss protection are recommended. Yes, it requires dedicated devices, but so does L2ARC.

bhouston9• 5 years ago

It is commonly misunderstood that ZIL SLOGs are write-back caches, they are not. They do not speed up large writes that do not fit into memory. I have tested this myself. It seems to be an incredibly popular misconception that has spread very widely.

xoa2• 5 years ago

The main issue with OpenZFS performance is its write speed.

At first I thought you were talking about actual raw read/write speed and how things like ARC or write caches can actually become bottlenecks when using NVMe storage, which can easily get to 200 Gbps with a small number of devices. That's being worked on via efforts like adding Direct IO.

Instead though I think you've fallen into one of those niche holes that always takes longer to get filled because there isn't much demand. Big ZFS users simply have tons and tons of drives, and with large enough arrays to spread across even rust can do alright. They'll also have more custom direction of hot/cold to different pools entirely. Smaller scale users, depending on size vs budget, may just be using pure solid state at this point, or multiple pools with more manual management. Basic 2TB SATA SSDs are down to <$170 and 2TB NVMe drives are <$190. 10 Gbps isn't much to hit, and there are lots of ways to script things up for management. RAM for buffer is pretty cheap now too. Using a mirrored NVMe-based ZIL and bypassing cache might also get many people where they want to be on sync writes.

I can see why it'd be a nice bit of polish to have hybrid pool management built-in, but I can also see how there wouldn't be a ton of demand to implement it given the various incentives in play. Might be worth seeing if there is any open work on such a thing or at least feature request.

Also to your lower comment:

I am using TrueNAS for my ZFS solution and it doesn't offer anything out of the box to address the issues I am bringing up.

While not directly related to precisely what you're asking for, I recently started using TrueNAS myself for the first time after over a decade of using ZFS full time and one thing that immediately surprised me is how unexpectedly limited the GUI is. TONS and tons of normal ZFS functionality is not exposed there for no reason I can understand. However, it's all still supported, it's still FreeBSD and normal ZFS underneath. You can still pull up a shell and manipulate things via the commandline, or (better for some stuff) modify the GUI framework helpers to customize things like pool creation options. The power is still there at least, though other issues like inexplicably shitty user support in the GUI show it's clearly aimed at home/SoHo, maybe with some SMB roles.

bhouston9• 5 years ago

You are correct -- I am in a niche area. I have less than 12 HDDs in a RAIDZ2 configuration. If I had more HDDs in the RAID I would see better write speeds. But I think most home users with TrueNAS ZFS do not have massive arrays, thus my envy of the tiering or true SSD caching.

You are correct that prices of small SSD are near HDD prices, but most people would want larger drives like 12 to 16TB drives in their array, and SSDs can not compete with these on price at all.

Maybe we are in the twilight of HDDs? And SSDs will compete on price across the whole storage capacity range soon? Maybe...

walrus013• 5 years ago

Basic 2TB SATA SSDs are down to <$170 and 2TB NVMe drives are <$190.

You sure wouldn't want to use them in any write heavy zfs application however, because you'll kill them quick in their ultimate TB write endurance. Those are triple level and quad level cell SSDs. Double the dollar figures there for anything with a ultimate lifespan TB written figure that you would want to try in a ZFS array.

walrus013• 5 years ago

Without these features, my QNAP (configured with SSD tiering) kicks the pants off of ZFS on writes that do not fit into memory.
Synology

I am perfectly willing to sacrifice some performance in exchange for not being locked into a vendor-specific proprietary NAS/SAN hardware platform or closed source software solution that's proprietary to that one vendor.

danudey• 5 years ago

While I like the idea of moving to a proper self-managed system, my Synology NAS is the least proprietary of all the proprietary systems I've used.

That is to say that yes, it does have a bunch of proprietary stuff on top, but it also:

1. Runs Linux

2. Uses Linux systems, like BTRFS and LVM, to pool data

3. Provides SSH access

4. Has tons of packages that you might want to use (Squid, Node, Plex, etc.) available

5. Supports Docker, among other things, so you can run what you like.

So yeah, they have a lot of proprietary offerings on top of a generic Linux storage solution, but if you change your mind you can take that generic Linux storage solution somewhere else without being locked in to anything Synology provides. The only caveat is if you choose to use their all-the-way proprietary systems, like their Google Docs equivalent or their cloud music/drive/etc. stuff, all of which is still stored on the same accessible drives. You might lose your "music library" or "photo library" or "video library", but your music and photo and video files will still be accessible.

xoa2• 5 years ago

Looks like a decent basic rundown of what was another nice jump forward (2020 fwiw, 2.0 wasn't just introduced it launched about 1 year ago, current version is 2.1.1). Worth noting a number of these have significant implications for tuning performance in common use cases, so some resources from even a few years ago are now a bit out of date. I know a lot of people for example have followed JRS' "ZFS tuning cheat sheet" [0], and a lot of that still applies. But given the increasing usage of enormously faster but pricier solid state drives, commonly a relative surfeit of processing power, and the performance of ZStandard, I think most people should now consider it over lz4 by default when using solid state drives unless using a very weak system or every last cycle is needed for other tasks (VMs perhaps). A bunch of detailed performance graphs comparing different compression algorithms were posted in the ZSTD feature introduction [1] which can help give a better idea of the right balance. HN had a long discussion on the kind of improvement it can make at scale just a few days ago [2]. Persistent L2ARC has also potentially changed the math there in some cases, though that one in turn may well be obsoleted entirely for SSD pools. And not new here I think but somewhat more recent, for those who do still have things that make use of atime ZFS now includes the Linux style 'relatime' support which does a nice job of splitting the difference between noatime and full atime.

Also, since a reasonable number of people may use ZFS via TrueNAS (formerly FreeNAS), note the GUI there doesn't expose a ton of useful features itself but can still be told to do them since it handles interfacing with ZFS via a plain Python. Currently, GUI pool/fs creation stuff is handled via:

  /usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py

'options' handles pool flags and 'fsoptions' fs ones. Modifying that (reboot seems needed to make it take effect afterwards though perhaps reloading something else would do it) will then allow all the GUI setup to work and slice things the way TrueNAS wants while still offering finer grained property control. Particularly useful for things that can only be set at creation time, like pool properties such as ashift/feature flags or fs properties like utf8/normalization.

----

0: https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet/

1: https://github.com/openzfs/zfs/pull/9735#issuecomment-570082...

2: https://news.ycombinator.com/item?id=29164727

todd8• 5 years ago

This is a great discussion on ZFS. I thoroughly enjoy reading HN! For those who aren't real IT people here is how I set up my first ZFS system at home.

My background includes a few years at IBM as an OS architect in charge of, among other things, the file systems and network services for AIX. However, this was a long time ago (~1988). So I understood NFS, inodes, what it meant to mount a device and the classic Unix commands for managing filesystems.

The first ZFS problem for me was just the vocabulary. What's a pool? What's a VDev? Fortunately, there are lots of high-level descriptions of ZFS on the internet that cover the high level concepts.

Getting the hardware set up was the second hurdle. There are a lot of discussions on forums that cover hardware issues and there you will find answers to many hardware issues, like: is ECC memory required, how fast does the processor need to be, are SATA drives recommended and what drive controllers are compatible with ZFS. Naturally, the answers to these questions depend on how critical is the system you are putting together. I suggest for most people to try setting up a ZFS on some old hardware just to get practice with it. I had a low end Dell tower server model that I wasn't using so it became my first FreeBSD/ZFS server. It had four hard drives.

The real difficulty for me was just the lack of a deep understanding of how ZFS works. ZFS has lots of features totally new to me so working with only the man pages for the ZFS related commands was not giving me much fun. I ended up buying two small books that I can't recommend enough, FreeBSD Mastery: ZFS and FreeBSD Mastery: Advanced ZFS by Michael W Lucas and Allan Jude, see [1] and [2].

These books are each around 200 pages long. The chapters of the first book are:

1. Introducing ZFS

2. Virtual Devices

3. Pools

4. ZFS Datasets

5. Repairs and Renovations

6. Disk Space Management

7. Snapshots and Clones

8. Installing ZFS

Each of the chapters is quite detailed, and I found these two books were just what I needed.

[1] https://read.amazon.com/kp/embed?asin=B00Y32OHNM&preview=new...

[2] https://read.amazon.com/kp/embed?asin=B01E40YIRM&preview=new...

Demystifying OpenZFS 2.0

Related Stories