ZFS 2.1.6

https://github.com/openzfs/zfs/releases/tag/zfs-2.1.6

by lysp • 4 years ago

176 106 4 years ago

rjzzleep• 4 years ago

Can someone explain what the relationship between openzfs and the zfs in opensolaris is? Are changes to openzfs backported to zfs in illumos or have they completely diverged? I think one thing that broke opensolaris' neck is, is how hard it is to set up a development environment compared to just building the linux kernel.

ghoward3• 4 years ago

IIRC, OpenZFS is the ZFS in Illumos. It's just not the ZFS in Oracle's Solaris.

OpenZFS was created as a way to combine development of all the versions of ZFS across Linux, FreeBSD, Illumos, and others. If you're using a (fairly) recent Illumos, you're using OpenZFS.

Disclaimer: Not an OpenZFS developer; just a happy user.

spijdar2• 4 years ago

Not exactly, take a look at the list of supported platforms in the releases notes of TFA -- OpenZFS 2 as a unified codebase is currently only ported to Linux and FreeBSD. Illumos is still using the older ZFS code with fewer features.

The whole history is pretty convoluted, since I'm fairly certain "OpenZFS" has been used to refer to several completely different source trees, with the Linux, FreeBSD, OS X, NetBSD, and Windows ports of ZFS all being separate forks off of the ZFS code from early OpenSolaris releases.

Despite the licensing drama and general distaste most distros and Linus have/had for ZoL, the Linux port had sort of become the de facto "standard" for ZFS development, with more work being done on that version of the source tree.

I only followed the "drama" from a distance, don't use ZFS or FreeBSD very frequently, but I know there was an internal discussion a while back within FreeBSD where they decided to pull commits directly from the ZoL tree instead of Illumos, because that tree was more active, and otherwise they had to wait on any improvements from Linux to be merged to the upstream Illumos tree.

The elephant in the room was the lack of activity in Illumos and the gradual decline of industry sponsors and development over the years. I don't know what the current state is, and I'm not saying Illumos is "dead", but it's definitely a shadow of its prime back when Joyent and a lot of companies were really pushing and developing on it. Several big names dropped out of Illumos development, including some NAS companies IIRC.

Anyway, OpenZFS 2, AFAIK, is essentially a rebranding of ZFS on Linux, but with CI/CD pipelines testing every commit against FreeBSD as well, basically unifying the work for Linux + FreeBSD.

The other versions (Mac, NetBSD, Windows, others?) still use separate forks AFAIK.

nix23• 4 years ago

If you have a bit of time, it's about OpenZFS and it's history, zfs-version vs. feature-tags etc.

https://www.youtube.com/watch?v=gmaHZBwDKho

neilpanchal6• 4 years ago

I build a zfs based NAS last year[1], after 10 months of usage, I have nothing but positive things to say. One thing I didn't understand conceptually at first is that ZFS is standalone and not dependent on the host OS. This was not clear to me and it is not explained anywhere. You can wipe out the entire OS, move the drives to a new system, mount them on a new OS and run `zpool import` to import them. All the information (raid config, snapshots, etc) about ZFS is stored on the pool itself and not on the host OS, especially if you have an alias setup for invidivual drives in `/etc/zfs/vdev_id.conf`.

There is no need for FreeNAS or any of that stuff, I don't use all those features. Just run like 3 commands to create a zpool, install samba service and that's all I need from a NAS. Cron job to run a scrub every month.

If ZFS file system made a sound, it'd make a satisfying and reassuring "CLUNK!" sound.

[1] https://neil.computer/notes/zfs-raidz2/

aborsy7• 4 years ago

TrueNAS ensures that different versions of different software fit together. So you update your OS, and your setup doesn’t break.

The common tasks are made easy: snapshot scheduling, snapshot replication, Samba share with a few clicks, backup to cloud, users management, one click install of applications, SMART tests, etc. Config management can be a headache.

neilpanchal6• 4 years ago

I use zfs-auto-snapshot for snapshots, it works great. I think the only thing I would want is early warning for a disk failure. I'll probably automate that. TrueNAS feels extremely heavy with too many features I don't need. There is a web server, database, etc which feels overkill for a simple home NAS IMO and having to manage the management system.

vladvasiliu2• 4 years ago

As GP said, all these features are nice, but if you don't need them, they're just overhead.

I'm in the same boat, all I have on my NAS is Samba, zrepl (ZFS snapshots and backups) and node exporter (monitoring agent for Prometheus - handles SMART, etc). It's running Arch, and I've never had my setup "break" in more than 10 years of using this distro (though this particular NAS is not that old).

I don't care for a DB, web server and god forbid random applications doing who knows what on my NAS. But if you want your NAS to do everything, it should be great. To each their own, I guess.

Also, TrueNAS is FreeBSD which may or may not work on dinkier hardware. I'm specifically thinking of random watchdog problems on some older Realtek Gigabit NICs, where you had to use a specific patched driver which, at one point, didn't support the latest version.

bombcar• 4 years ago

My NAS is a zfs-root Gentoo system. I went with it because I couldn’t beat freenas and friends into doing what I wanted.

asveikau4• 4 years ago

Is this where we're giving testimonials?

I've been a ZFS user for maybe 6 years. I had data loss this year.

It was quite interesting. Writes would fail on multiple disks at the same time. That's why it was data loss. A normal disk failure wouldn't look like that, failures should be spread out and redundancy would help.

It turned out to be a bad power supply. I was able to predictably get it to corrupt writes with the old PSU. Then I replaced the PSU. No longer failed after that.

I wouldn't have guessed to suspect the PSU. It was a frustrating experience but in the end, ZFS did help me detect it. On ext4 or ffs I wouldn't have even been aware it happened, let alone could confirm a fix.

aborsy7• 4 years ago

Copy-on-write file systems such as ZFS should be resilient to power outages.

How does power fluctuations or cutoff cause data loss in ZFS?

I frequently force shutdown my laptop and PCs with ext4 when they hang, and never lost data.

abrookewood• 4 years ago

What did that look like in terms of error messages etc? I'm guessing ZFS would try to write the checksum, which wouldn't work and then throw an error? I assume it never impacted data that already resided on disk?

guilhas• 4 years ago

Jesus same happened to me so frustrating! Everything working ok, then slowly increase of writing failures and power resets, and strange clicking, eventually all disks drop, scary every time it happened

I swapped every server part before the PSU. Updated Linux, downgraded, tried kernel options... At some point I thought btrfs was the problem so I created an mdadm ext4 raid but got the same problem

It was a btrfs raid 1, lots of fs errors, and files missing until restart, as some disks were down. But didn't lost any data (take that ZFS!) besides the one being transferred as the disks/array went down anyway

jen20• 4 years ago

Writes would fail on multiple disks at the same time. That's why it was data loss.

This doesn’t sound like data loss but rather a fault preventing writing… it would be data loss if confirmed writes were lost.

bbojan• 4 years ago

I just recently had a case where writes to an M2 SSD on a newly built computer would frequently fail. Reads were OK.

After replacing the SSD twice and then rebuilding everything using a different motherboard, it turned out it was the PSU.

Really hard to diagnose problems like this.

bubblethink• 4 years ago

One thing I didn't understand conceptually at first is that ZFS is standalone and not dependent on the host OS.

This is true for every mature/production file system. You can do the same with mdadm, ext4, xfs, btrfs, etc. The only constraint would be with versions in that it would be a one way street. Can't necessarily go from something new to something old, but other way round is fine.

neilpanchal6• 4 years ago

Right, what was not clear to me was that raid configuration and snapshots are also part of the file system. Usually, that's done through hardware cards or software raid where the configuration sits not on the file system (?) (Intel VROC, vSAN, etc), at least that was my wrong conceptual model. I used to make multiple copies of the USB stick of FreeNAS 8 back in the day because I didn't want the USB drive to fail and not knowing how to recover the zpool. Messing around with the file system directly cleared up everything.

magicalhippo2• 4 years ago

ZFS stores mount points and even NFS shares in its metadata[1], so more than most others.

[1] https://openzfs.github.io/openzfs-docs/man/8/zfs-set.8.html?...

yjftsjthsd-h2• 4 years ago

Oh, it's better than that! If you're crazy enough, you can even multiboot multiple OSs on the same pool:) You do need to ensure that the pool features are sufficiently compatible between ZFS drivers (ex. you can't create a fully-featured pool with OpenZFS 2.1.6 on Linux and import it on OpenIndiana, last I checked), but it does work, empirically;)

neilpanchal6• 4 years ago

A friend of mine suggested this and I didn't like the idea of putting the OS on there especially that it is a virtual machine. There are a couple of things that made this setup nice though: PCIe bifurcation (4x4x4x4) lanes straight to the CPU and PCIe passthrough in ESXi. From what I read that ZFS needs direct access to the underlying hardware. So, I just treat the OS as totally disposable and not worth backing up. I can spin up a VM in less than 2 mins if something were to happen to it.

phpisthebest• 4 years ago

no need for FreeNAS or any of that stuf

I would encourage anyone that does not have an system admin background to use FreeNAS(truenas now) or something

Their are default security and maintenance things (like periodic scrubs of the pool) that these nas operating systems set up by default

barrkel2• 4 years ago

ZoL on Ubuntu adds a cronjob for zfs scrub; shipped in xenial (16.04).

jiveturkey• 4 years ago

standalone and not dependent on the host OS.

The OS needs to implement zfs, like any other filesystem/host combination. Any filesystem can be "moved" to another host simply by attaching the drives, assuming you have hardware level compat and filesystem level compat.

OK the exception is when you have host-based hardware level encryption (ie, key in TPM or other security chip). In that case, zfs and otherwise, you can't just move the drives.

waynesonfire2• 4 years ago

Anyone know if there is a way to install this on a Ubuntu 16.04 system? Would like to update from 0.8.4-1 but jonathonf hasn't build anything newer for this version of Ubuntu.

SkyMarshal2• 4 years ago

You can install the Nix package manager on other Linuxes (or Mac) and use it to manage some packages. It keeps ZFS up to date and that may be an option for installing it on Ubuntu. I’ve never done this before so not sure, ask in the Nix community to verify.

remexre• 4 years ago

I would suspect Nix wouldn't be the optimal way to manage Ubuntu kernel modules; I suppose you could use it to build them, but I don't think Nix without NixOS will e.g. configure DKMS for you.

ratorx2• 4 years ago

This isn’t a good idea.

ZFS needs to be kept in sync with the kernel version. If you use Nix on Ubuntu, Ubuntu manages the kernel, so Nix can’t do anything.

ZFS on NixOS is pretty neat though. The exact version pinning means that ZFS and the kernel can be updated together and only once both packages build at a new version.

There’s also an open issue for more extensive checks, so hopefully eventually NixOS CI will be able to check not just that the ZFS and kernel are compatible (header-wise), but actually go through the motion of creating test ZFS dataset and make sure it actually works.

yjftsjthsd-h2• 4 years ago

I think you can just download the source and build it as if Ubuntu didn't support ZFS. https://openzfs.github.io/openzfs-docs/Developer%20Resources... I haven't done this on Ubuntu, but I can attest that it works on Debian so unless Canonical did something "interesting" to break things it should Just Work™.

mustache_kimono8• 4 years ago

At the very least, you can do (and I have done) something similar with `apt source --build zfs-linux`. I wasn't terribly worth it for me (it takes some work?), but it's definitely possible/reasonable create backported native packages relatively easily.

Conan_Kudo2• 4 years ago

There's a repo on the openSUSE Build Service: https://software.opensuse.org//download.html?project=home%3A...

_joel• 4 years ago

Wouldn't it better updating OS. I mean, I'm all for stability and if it's isolated no problem it not getting patches, but it if you're going in for kinda hackish updates to 16.04 kernels then maybe bumping to a later OS version might be better? You're already adding volatility and potential issues, but at least with a supported build it reduces the latter (and former really)

mattjaynes• 4 years ago

I've had good results with this PPA for using later versions of ZFS on earlier Ubuntu versions (though it's only at zfs 2.1.5 at the moment, but it should be updated in not too long): https://launchpad.net/~jonathonf/+archive/ubuntu/zfs

gigatexal• 4 years ago

supports kernels up to 5.19 but 6.0 and 6.1 are on the horizon :(

boomboomsubban• 4 years ago

There are always kernel releases on the horizon, my experience has seen ZoL supporting them pretty quickly.

thulle• 4 years ago

There was some discussion about waiting for 6.0 in the pull request for 2.1.6, but they chose to ship this and instantly open a branch for 2.1.7 pull requests.

https://github.com/openzfs/zfs/pull/13886

puffoflogic• 4 years ago

This concern doesn't make any sense. There are always new kernels on the horizon bydl definition, and there's nothing special about 6.0 at all. By this logic nothing should ever be released as it should always be waiting for the "on the horizon" kernel.

Octabrain3• 4 years ago

Anyone using ZFS on a daily-driver desktop computer?

Fedora + BTRFS + Deja dup daily backups have been my default for months now, but I am tempted to give it a go to perhaps Ubuntu JJ + ZFS + Deja dup. Not that I have any issues with my current setup. It's just for the sake of, you know, try new things.

gnfargbl4• 4 years ago

I have been using ZFS on my daily-driver Ubuntu laptop for six months or so. Also been using ZFS as the backing store for a heavy update postgres database for a few years before that. Haven't had any issues with it at all, generally it seems like a very solid bit of software.

I'm not familiar with Deja dup, but the usual method of backing up a ZFS filesystem is to snapshot it and then stream the snapshot (just a file, can be compressed) into the remote store of your choice.

Octabrain3• 4 years ago

Thank you for answering! Deja dup it's just a GUI over duplicity (https://en.wikipedia.org/wiki/Duplicity_(software)). It allows me to have encryption, differential backups and restoring files in a granular way (even from Gnome context menu) with no effort. Very pleasant experience similar to Time machine in MacOS. Anyways, I'll definitely give it a go to ZFS.

aborsy7• 4 years ago

Streaming ZFS snapshots to a file and storing that in a remote of your choice is not the usual way of backing up ZFS data at all (it’s actually recommended against).

The usual way is ZFS send and receive to another ZFS server. But almost no cloud storage provider supports that, other than rsync.net (and zfs.rent), which starts with $60/month. So you have to set up and manage another ZFS server with secure remote access, PIA if you ask me.

paulgdp2• 4 years ago

I have been using ZFS as root FS since the first release of ZFS on Linux (2013) on all my machines (laptops, NAS, server) and I can't imagine going back.

flatiron• 4 years ago

What are the advantages of it on a laptop. Do you have more than one drive in it?

I honestly use ext4 for everything based on ignorance and familiarity.

nisa• 4 years ago

using it on my notebook with native encryption and I'm quite happy - there are some things missing / work in progress for quite a while - unfortunately it looks like it's not a priority at the moment for the current sponsors or there is a lack of time - not a criticism of zfs but I'd love to see these features merged sometimes in the future:

- no overlayfs yet (https://github.com/openzfs/zfs/pull/9414) - the docker zfs driver is very slow and buggy - best to create a sparse zvol with ext4/xfs/btrfs and use this for /var/lib/docker

- no directio yet (https://github.com/openzfs/zfs/pull/10018) - not sure if this is so useful beyond specialized databases that utilize that.

- no idmapped mounts yet (https://github.com/openzfs/zfs/pull/13671) - this would be useful for lxc/lxd

- async dmu (https://github.com/openzfs/zfs/pull/12166) - complicated patchset would transform a lot of operations into callbacks and probably increase performance for a lot of workloads.

- namespace delegation was recently merged: https://github.com/openzfs/zfs/commit/4ed5e25074ffec266df385... with idmapped mounts it should be possible to have zfs datasets / snapshots inside a unprivileged lxc-container which could be super cool for lightweight container things.

caveats:

- no swap on zvol at the moment (https://github.com/openzfs/zfs/issues/7734) - this is some hairy memory allocation problem beneath it and I don't really use swap - just something to be aware off.

- arc vs. pagecache / performance (https://github.com/openzfs/zfs/issues/13736) - zfs is usally very fast but there are some edge-cases where it's not at the moment.

infogulch2• 4 years ago

namespace delegation

That's great. I had a conversation with the owner of rsync.net about why they require zfs-enabled customers run on a separate bhyve VM (thus requiring a large minimum). Iirc it boils down to zfs destroy permissions somehow affecting other tenants. I see mention of zfs destroy in the linked patch, I'm curious how that relates to rsync's experience and requirements.

https://news.ycombinator.com/item?id=32019105

barrkel2• 4 years ago

I would worry a little about going above 90% space usage. ZFS slows down fairly dramatically when it starts running out of contiguous space and there is no defragmentation tool.

On the other hand it's not that hard to export and re-import a ZFS file system from a workstation or laptop to defragment manually - it's harder to do that with a NAS with lots of drives when you don't have a second set of drives just hanging around.

aborsy7• 4 years ago

On a laptop, with Ubuntu. No issue.

matheusmoreira• 4 years ago

I want to use it but I'm waiting for RAID-Z expansion.

UI_at_80x242• 4 years ago

Have you heard about ZFS feature called "dRAID"?

https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAI...

Expand your ZFS array over a network. It's not expansion, BUT it can be used to grow a pool size.

trissylegs• 4 years ago

I'm using ZFS on my desktop (Arch Linux). 1 'Pool' is just an SSD that my OS root drive. (Booting off ZFS is a bit harder but putting /boot on your ESP works).

And a hard disk array with 4 drives in raidz (like raid 5).

Main advantages: * I moved my root drive to another drive and did it entirely online without needing to cp or dd everything just add new drive to pool, remove old drive (and wait until it's done, but you can keep using it while it copies) * It's way faster than MD, md spends 10s of hours initializing or resilivering big drives. zfs just formats it and your good to go. * Subvolumes and snapshops are great. You can give a folder different file system properties. (Eg make /var/log compressed and disable it for /var/cache/pacman because it's already compressed) * Docker is quicker. * Although BTRFS has snapshots I do find ZFS's more intuitive. (eg subvolume can be nomount in ZFS which allows you to organize and apply options recursively easier)

Check for bugs in Ubuntu ZFS before installing updating. They do it differently than other OpenZFS options.

I'm not currently using ZFS send for backups. So I can't compare it to Deja dup.

UI_at_80x242• 4 years ago

Yes, for 5+ years. FreeBSD.

sir_brickalot• 4 years ago

Using it on MacOS for years (not for system) and fairly recently for ZFS on root on Ubuntu.

I love it but two problem I ran into on Ubuntu:

1. Docker failed bcs of the custom implementation in Ubuntu (Don't remeber the specifics).

2. I repeatedly run into the problem that rpool (root) or bpool are supposedly full and automatic snapshots break.

I hope those problems get fixed soon.

wb14123• 4 years ago

I've used ZFS on my daily desktop computer for almost 3 years. I'm using it with ArchLinux. The overall experience is great but here are some notes:

* The cpu usage can be higher if you use compression.

* I used to setup the main pool with SSD cache with HDD data store. It's okay for most use case except slow first time open of applications. But for demanding games it usually has very bad performance.

* ZFS also manage cache by itself so it needs some memory. Personally I'd like to set a cap for it, because otherwise when I open some memory hungry program it tries to write cache onto disk which sometimes makes the system slow. This happens when I only use SSD as cache. Not sure if it can be better with all SSD solution.

* I ran into problem with docker as well. But after change some configurations it works perfectly.

* I use zfs snapshot and zfs send/recv for backups and it is really awesome. I've written a blog about my personal offsite backup solution that you may find useful: https://www.binwang.me/2021-09-19-Personal-ZFS-Offsite-Onlin...

mnw21cam• 4 years ago

Yes, I have been using it for the bulk storage part of my desktop for the last 7 years, in a 3-drive RAID-Z configuration. I don't think any of the drives in the pool are the originals, as they have been swapped out for larger models as and when they started showing errors, with a scrub happening once a month. I have OS, swap, /home, /var, /tmp and ZFS log on an NVME drive separate - the ZFS pool is only used for the large stuff.

Separately, at work I managed to persuade them to configure our two new servers with ZFS, each with 168 drives for a total (after RAID) of 1.5PB each. Works like a charm, and is quite happy to saturate a 10Gb ethernet connection when accessing from another server, with transparent compression turned on. I would not hesitate to recommend ZFS to anyone.

ghoward3• 4 years ago

I have, for more than two years, and my remote backup has been an encrypted ZFS for a year and a half. [1]

The scripts in that post are a little outdated, so if you would like to see updated ones, contact me [2].

The end result is a setup that:

* Has parallel upload to saturate my slow upload connection.

* Allows resume with `make -j<cores>`

* Allows me to backup things that are only on my server, such as my Gitea instance [3], to my desktop, thus having my server and desktop serve as backups to each other.

* Allows me to delete snapshots older than a certain number of snapshots (set to 60).

* Has automatic loading and unloading of encryption keys, to reduce the exposure time. (However, I wish raw sends on encrypted datasets worked. But it is nice that I can load the keys and download individual files instead, which I've already had to do.)

Perhaps I should add an update post to [1]. If there's enough interest, I will.

Edit: I should make clear that my ZFS is only my home directory. I do not use it as a root or boot filesystem because it runs off of a mirror of hard drives, and my root/boot drive is an SSD and can be destroyed without pain because all of my configs are in a repo in my home directory. Thus, it would be trivial to rebuild my system.

[1] https://gavinhoward.com/2021/02/adventures-in-backing-up-dat...

[2] https://gavinhoward.com/contact/

[3] https://git.yzena.com/

drbawb• 4 years ago

Yes, I have Arch on a ZFS root pool. rEFInd + ZFSBootMenu to boot it. The only thing I don't use ZFS for is a Windows (gaming) VM. I tried numerous different setups to keep the NTFS disk image on a zpool, but it's ultimately pretty hard to beat the performance of PCIe passthrough for a modern NVMe drive. I use syncoid/sanoid + ZFS delegations for backup management.

No real complaints except that it complicates updating Arch: kernel updates usually have to be delayed 24-48 hours while the ZFS packages get rebuilt, or in some cases patched upstream. syncoid/sanoid also poses a problem because it relies on a few Perl packages from the AUR that are a pain to rebuild.

I know the Arch devs loathe people doing partial updates, but occasionally delaying a kernel update for the sake of ZFS has yet to bite me in the ass. I would strongly advise against DKMS, though. Use a binary module or build the package yourself. The majority of problems I had with Arch+ZFS boiled down to misuse of zfs-dkms.

matja• 4 years ago

Specifically zfs-dkms + the stable kernel package ("linux") can often be an issue because I have seen the kernel version updated by Arch without ZFS support for that version (obviously not something that the Arch kernel package maintainers should care about, but it's a bit inconvenient).

zfs-dkms + "linux-lts" (longterm kernel) seems to work better than zfs-dkms + linux (stable kernel) if you're intending to do frequent system updates - I've been running this on several machines for years with no issues.

linsomniac• 4 years ago

I installed Ubuntu 20.04 with ZFS for the root and user storage, with ZFS encryption, and frankly found the performance to be annoying. Keep in mind: I'm a huge ZFS fan. One of the packages I had installed, I forget which, made basically all apt sessions take much, much longer because of snapshotting, and my system load was basically always 1-3. I ended up reinstalling with ext4+LUKS when 22.04 came out. This is on a Dell XPS 15 with i7-10750H and 32GB RAM and "PC611 NVMe SK hynix 1TB" SSD.

aflag• 4 years ago

I've been enjoying my experience with zfs this far. I just wish removal of vdevs was a thing for raidz setups. I could never fully understand why removing a full vdev that happens to implement raidz isn't as easy as removing a mirror one (which is supported). I suppose vdevs don't fully abstract the away the underlying mechanism.

ZFS 2.1.6

Related Stories