ZFS Profiling on Arch Linux

https://www.binwang.me/2023-12-14-ZFS-Profiling-on-Arch-Linux.html

by wb14123 • 3 years ago

183 33 3 years ago

This is a great read. It explains in detail the process of enabling debug symbols and using perf to root cause a performance issue.

I've never done this type of work, but probably will in the future now that I have an example to follow.

tjoff• 3 years ago

It is refreshing to see the author explain how and why all the pieces were gathered. Most guides/instructions/bug-reports only lists the end result, omitting information that helps the reader to get a better understanding and be better equipped to solve related issues.

sneak• 3 years ago

I heard a rumor that ZFS native encryption is somewhat abandoned by its maintainer following it having landed in the release, that it was one person and they moved on/lost interest. I don’t know if this is true or not.

I’m still using ZFS on LUKS2 for now, given the issues like this. (You also can’t specify multiple keys in ZFS native encryption.)

xoa• 3 years ago

Sadly there have seemed to be a number of rough edges that have stuck around with ZFS native encryption, which is unfortunate given that the cross platform and remote replication advantages are fundamentally very compelling. Honestly it's kind of curious that polishing that experience hasn't been more of a priority, for commercial interests like iXsystems if no one else. Speaking of which, for people running their ZFS off a NAS the idea of just being able to offload all the encryption and FS computation to that is a compelling part of the value equation beyond sheer storage in principle too. Though that in turn (if one does it with iSCSI) runs into another seemingly neglected area of ZFS (zvols). Nothing is perfect.

zrav2• 3 years ago

What makes you say zvols are neglected? In fact the people that did most of the work porting ZFS to Linux, i.e. LLNL, use zvols for their HPC cluster and they still employ a number developers and the primary maintainer of ZoL. I also remember a quote from them stating that zvols were in fact more mature than vfs.

justinclift• 3 years ago

ZFS on top of LUKS seems to have it's own issues though. :(

https://github.com/openzfs/zfs/issues/15533

PaulCarrack• 3 years ago

Since LUKS is exposed as block device like a hard drive, it's really no different than saying that they are just having issues with certain kind of hard drives.

There is a PR out for this: https://github.com/openzfs/zfs/pull/15588

zrav2• 3 years ago

See this discussion about the state of native encryption and the linked list of open bugs: https://discourse.practicalzfs.com/t/is-native-encryption-re...

wb14123• 3 years ago

More details on this discussion: https://github.com/openzfs/zfs/issues/6824#issuecomment-1817... . Basically he is too busy to continue developing the encryption features but is able to review the related works.

There is also discussion about using ZFS on LUKS in the same thread: https://github.com/openzfs/zfs/issues/6824#issuecomment-1819...

predmijat• 3 years ago

On Arch, zfs-linux package has an exact version of linux as a dependency. linux package gets updated fairly often, and by the time zfs-linux package catches up, linux is already on a newer version and you are unable to upgrade.

Sometimes it takes quite a while before those two are in sync and you can upgrade your system.

hnlmorg• 3 years ago

Is there lts kernel support for zfs-linux? I found I had to switch to the lts kernel on an Arch gaming rig because of issues with Nvidia drivers.

aperum• 3 years ago

There is zfs-linux-lts for that.

mzi• 3 years ago

If you reorder your repositories so the zfs repository comes before the default one in the configuration you won't have that problem.

commandersaki• 3 years ago

Huh, interesting, will have to try that out, thanks.

tobias2014• 3 years ago

In addition to the other comments: you could use the zfs-dkms package which builds it from source on every update

79a6ed87• 3 years ago

Targetting the linux-lts package as a dependency instead of linux should reduce the impact of this issue

enormousness• 3 years ago

You can install the specific older version of the Linux package that the zfs package needs. You can also pin it so it won't get updated by regular upgrades.

b33j0r4• 3 years ago

Channelling Erik Dubious: “it’s ok, you see the error message, and you navigate to etsy.

This config we find is the answer. I lean into neovim this time, today, because it’s all a lessons. And then we type update, and it’s all okay!”

The thing he humbly leaves out is “why aren’t you an arcolinux fanboy yet!?”

b33j0r4• 3 years ago

I am like disturbed that two people thought complimenting Erik’s work was worth a downvote. Who the heck hates nice people so much to spend that.

I’m an arch contributor, your’re barking up the wrong tree.

dfc2• 3 years ago

I think the downvotes had more to do with people not knowing what/who you are referring to. Without the context that you have your comment seems like word salad.

Modified3019• 3 years ago

Perhaps it's a non-native english speaker thing, but it's very hard to understand your prior post.

It looks like a bot post, from the time before AI generation actually became good. The post appears like random disconnected concepts are just thrown together with no clear context or relevancy to the topic of debugging ZFS.

nisa2• 3 years ago

Another fix for that is to switch the ZFS license to GPL before building:

Just put this line:

sed -i "s/CDDL/GPL/g" META

in the prepare() section of the PKGBUILD.

This will use the in-kernel FPU functions for cryptography and other things like raidz calculations that got EXPORT_SYMBOL_GPL'ed later if the original plumbing is still in the codebase. It's probably not offically recommend and I'm not sure how much these codepaths are tested anymore but I assume that quite a few people that don't need to redistribute anything but need the performance use a hack like this or something similiar. It works fine for me.

Personally I lost a ton of respect for the Linux kernel developers after introducing that kernel fpu EXPORT_SYMBOL_GPL thing and backporting it to all kernels and Linus badmouthing ZFS and somehow assuming Oracle is still involved in OpenZFS which is mostly FreeBSD and some sponsors spread out over the world but not related to Oracle.

nine_k• 3 years ago

This evokes in me a distinct feeling of applying a crack to run a pirated game.

nisa2• 3 years ago

I guess it violates the CDDL licence - at the time some distributions like NixOS just patched the EXPORT_SYMBOL_GPL out of the Linux kernel (https://github.com/NixOS/nixpkgs/pull/61076/commits/7b77c27c...) which could even be totally legal for redistribution as the GPL probably allows this?

nicman23• 3 years ago

it is probably fine as it is just the instructions. i have done it before in a raspberry for some reason i do not remember

Sodel• 3 years ago

I mostly agree with your points except to say that, even if Oracle is not active in the project, the risk is real as long as Oracle owns any of the copyright for any ZFS code.

The USL vs BSDi litigation over BSD ultimately came down to a handful of source files out of 18000. ( https://en.m.wikipedia.org/wiki/History_of_the_Berkeley_Soft... )

I wouldn't talk badly against ZFS, but I would not roll any dice with Oracle as a copyright holder.

tiberious726• 3 years ago

So the answer is crime?

ZFS Profiling on Arch Linux

Related Stories