Is ZFS a suitable replacement for other Linux filesystems?

https://clusterhq.com/blog/zfs-suitable-replacement-file-systems/

by lewq • 12 years ago

70 84 12 years ago

ryao18• 12 years ago

I wrote the above article. I will try to answer any questions that I see here.

notacoward2• 12 years ago

Nice article. I think you understate the importance of O_DIRECT, since it's the only way many kinds of software can work around the utter lameness of Linux local filesystems' durability semantics, but other than that I think it's extremely informative and useful.

otterley• 12 years ago

O_DIRECT isn't about durability -- a broken device that falsely reports that data was committed will still be broken when O_DIRECT is used.

O_DIRECT is about bypassing the kernel's buffer cache (and, presumably, zfs's ARC) so that the software can implement its own buffering.

jfindley• 12 years ago

As another commenter noted it's not a complete durability solution. However it is useful in a number of places - and not just to do with durability.

AIO is also very extensively used, and bmap is pretty important.

It's really important when writing comparison pages like this to not play down the weaknesses of the thing you're trying to sell - it comes across badly. O_DIRECT, AIO and to a lesser extent bmap are feature that are required of a general purpose linux filesystem. I'm sure you'll have them all working in ZoL soon enough, but please don't try to pretend they aren't important.

I'm also not sure why you have NFS in there - network filesystems are not local filesystems, and aren't generally going to have identical feature-sets. It feels a bit like you just stuck NFS in there so that there'd at least be something with more missing features. The same could kind of be said of JFS - I haven't heard of anyone using JFS on linux in a serious way in a very long time.

justincormack• 12 years ago

I can't read the tables on my phone as can't shrink them down (Firefox Android)... So as you don't seem to spell it out is it the case that SElinux should be fine on ZFS but PAX might not be?

ryao18• 12 years ago

They both work, with two caveats:

1. ZoL is currently incompatible with kernels built with either PAX_USERCOPY_SLABS or PAX_KERNEXEC_PLUGIN_METHOD_OR. The next release should fix PAX_USERCOPY_SLABS and might fix PAX_KERNEXEC_PLUGIN_METHOD_OR. The former should be fixed by a kmem-rework that the project plans to merge before the next release:

https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/zfs/pull/2411

The fix requires both pull requests, but I have yet to refresh the second pull request with the fix. I plan to refresh it before I leave for LinuxCon Europe 2014.

The latter might be fixed by changes to drop support for pre-2.6.32 kernels that the project lead would prefer to merge for the next release:

https://github.com/zfsonlinux/spl/pull/395

2. There is a bug specific to ZoL where unlinking a file with a directory extended attribute (Solaris-style) will trigger a full traversal of all objects (i.e. inodes) within a dataset. It manifests itself as the zfs_iput_taskq kernle thread consuming excessive CPU time. It is fixed for the next release:

https://github.com/zfsonlinux/zfs/commit/0d5c500d6cf4dd9e133...

purplerails• 12 years ago

According to this article, ZFS is sub-optimal for databases due to fragmentation of the db files. Could you please comment? Thx. http://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-...

nabla9• 12 years ago

Oracle advice on how to configure ZFS and Oracle Database

http://www.oracle.com/technetwork/server-storage/solaris/con...

Some picks:

Free space is important 10-20%.

In an environment with a high rate of data update (or churn), it is advisable to maintain a certain amount of free space. ZFS is a copy-on-write file system and relocates all writes to free disk space. Keeping a certain amount of free blocks allows ZFS to easily find space in big chunks, allows ZFS to aggregate writes and reduce the write IOPS demand on hard disks. Streaming aggregated writes can be 50-100 times less demanding on hard drives than doing small random writes. Therefore, the incentive for keeping free space is high, especially in a high churn environment, such as an active OLTP database.
The number one rule for setting up an Oracle database on ZFS is to set ZFS recordsize equal to the database block size for the file systems that contain the Oracle data files. recordsize = db_block_size

jwildeboer• 12 years ago

Who offers production Level support? Certification? No one AFAICS. I wonder why you didn't mention that.

ryao18• 12 years ago

It was not the purpose of the blog post. Anyway, you mentioned a couple of different things here. First, there are a few companies that offer production level support. SoftNAS is one. For full disclosure, my 2014 tax return will include consulting fees earned from fixing bugs for SoftNAS. It is in use at various large companies, including Netflix and Boeing:

http://www.softnas.com/

Another is PogoStor, which advertises that the hardware that they sell has been certified by Nexenta for use with ZFS:

http://www.pogostorage.com/products/nexenta/overview/whycert...

They also have their own Linux distribution based on ZoL, for which I assume that they offer commercial support:

http://www.pogostorage.com/products/pogostor/pogostor-overvi...

As for certification, it is not clear if you mean hardware certification or system administrator certification. I am not aware of any filesystems that have system administrator certifications. As for hardware certifications, ZFS is designed for use with commodity hardware. It probably does not make sense to try to certify it for the full range of hardware it supports (although a subset is certified by Nexenta). Specific hardware guidelines are available on the official Open ZFS wiki:

http://open-zfs.org/wiki/Hardware

Thaxll2• 12 years ago

No.

ferrantim• 12 years ago

Care to elaborate?

valarauca13• 12 years ago

Zpool is a great feature. On top of the dozen(s) of amazing features in ZFS. The problem is, and ZFS admits this: If you don't run ECC ram Zpool can "accidentally" your whole hard drive (bit of a joke there, it can corrupt your drive as it attempts to correct bit rot that never happened).

This happens a lot more often then we really care to think about. (Ram corruption)

For most day-to-day linux users who are just using standard consumer desktop/laptops they don't have this protection. And ZFS can do more harm then good.

References:

https://pthree.org/2013/12/10/zfs-administration-appendix-c-...

Thaxll2• 12 years ago

The article is missing many things that make a FS suitable for production, it's listing every features but lack the reliability / cons / waknesses ect... ZFS is nowhere near production ready for Linux. I know XFS and ext4 under heavy load with different scenarios, could you tell the same for ZoL?

Qantourisc2• 12 years ago

No, it doesn't has shrink, a future you want on a desktop imo. And occasionally on server when the customer doesn't provide enough storage (on the account of being cheap) and you have move things around to get things done.

icantthinkofone• 12 years ago

I'm a bit confused. I always thought that proper use of ZFS required multiple drives just like RAID. Then I read an article where RAID was used with ZFS. Now I'm really confused so I'm taking the lazy way and asking here: is ZFS appropriate for a single drive system?

Qantourisc2• 12 years ago

Sure, COW(copy on write), snapshots, error checksum on files, compression, support for SSD's. You just don't have redundancy.

FooBarWidget• 12 years ago

I thought the error checking (scrubbing) only works if you have multiple drives set up in a RAID.

prakashsurya6• 12 years ago

Yes, I run ZFS on Linux on my single drive laptop currently.

ZFS doesn't _need_ multiple drives to work well, but it is capable of using them if they're available. One can also run a HW raid solution underneath ZFS, where the RAID engine just presents ZFS with a small number of LUNs. It's all up to the admin; and what sort of performance, redundancy, and maintenance guarantees are required.

ChrisInEdmonton2• 12 years ago

I have heard that ZFS (at least, ZFS on Linux) requires about 1 GB of memory for every 1 TB of storage. Is this an accurate statement? That's certainly a critical flaw for many use cases, though completely irrelevant for many others. If accurate, what's behind this requirement?

dedward2• 12 years ago

All the extra stuff ZFS does, and does performantly, comes at a cost. That's the cost (I can't vouch for the amount.. but the principle is right. CPU power matters too.)

You don't get all those features for free... but you do get them.

ryao18• 12 years ago

While this is true, ZFSOnLinux has made progress on reducing CPU overhead. Additional improvements will be in the next release. In particular, the performance of NFS exports of ZFS datasets will increase considerably (some are seeing a factor of 2 increase) while CPU utilization has dropped:

https://github.com/zfsonlinux/spl/pull/369#issuecomment-5839...

Already__Taken• 12 years ago

another anecdote: I've only heard this mentioned specifically about using de-duplication.

prakashsurya6• 12 years ago

No, that's flat out not true.

I've seen that metric thrown around when talking about the "dedup" feature of ZFS, but honestly, don't use dedup unless you know what you're doing. It's way to easy for things to go wrong otherwise.

ChrisInEdmonton2• 12 years ago

Ah ha! Thanks. Data deduplication. Well, yeah, that makes sense to me. Thank you very much for clearing that up.

beagle3• 12 years ago

How much memory does ZFS (and/or ZoL) need per 1TB of storage when dedup is off?

Also, bup ("it backs things up!") efficiently dedups across an ssh connection (using bloom filters) Scales are differrent, but it might work for ZFS as well.

ryao18• 12 years ago

On ZFS, all file data is stored using B-trees, where the leaves store X bytes, where X is the value of recordsize at file creation. When a write is done to a file in a dataset with dedup=on, a lookup is done on the data deduplication table. If the lookup finds an entry, it increments a reference. If it does not find an entry, it creates one. This involves 3 random seeks and consequentially your total write throughput is [average record size * IOPS / 3 random seeks]. If you have sufficient memory that the entire DDT can stay cached, then we avoid this limit.

The 1GB of RAM for every 1TB of storage is a rule of thumb for avoiding this limit that someone made a very long time ago. Unfortunately, that rule is wrong because it is impossible to estimate memory requirements by such a simple rule, but it has stuck with us for years despite being wrong.

The amount of memory needed to store a DDT is a function of the average record size. If your dataset has the default recorsize=128K (e.g. you are storing many >1MB files), then you can multiply your system memory by 153.6 to determine the total amount of unique data that you can store before hitting the limit I described. If you are storing many small <128KB files, then you need to calculate the average file size and then calculate [average file size * system memory * 12 / 10355] to obtain the total amount of unique data. This assumes the module settings for zfs_arc_max and zfs_arc_meta_max were left at their ZoL defaults. It is important to keep in mind that the unique data is different from total data, especially in the case of duplicate files. It also excludes metadata required for maintaining indirect block trees, dnodes (i.e. inodes), directories and the DDT itself.

The total amount of data that you can store on a pool before hitting the limit is [unique data * deduplication multipler], where unique data is what I described how to calculate and the deduplication multipler is a number that is either 1 or greater. A measure of the deduplication multipler is provided by `zpool list` as DEDUP, so you should be able to see that yourself. If your pool has data that was written without dedup=on, then any duplicates in that data will be counted as unique data for the purposes of that calculation. To provide a simple example of the deduplication multipler, imagine a pool with only two files that store the same data. The deduplication multipler for that pool would be 2, provided that both were written with dedup=on set on their dataset. If one or both were written with dedup=off, then the deduplication multiper would be 1. You can use zdb to calculate the theoretical deduplication statistics for an entire pool by running `zdb -D $POOLNAME`. Note that this will require significant memory because it constructs a full DDT in userland memory.

dedward2• 12 years ago

This should be fairly obvious...

ZFS has it's own requirements... to work well it needs lots of resources (cpu/ram) and EEC memory.

If you have those, then sure, it gives you many advantages.

If you don't, then you are opening yourself up for struggle.

atoponce5• 12 years ago

ZFS doesn't need any more "resources" than any other kernel space filesystem. Deduplication on the other hand, does need a good amount of RAM, if you want to prevent the deduplication table (DDT) from spilling to slow spinning rust. However, if you have a fast, and large SSD L2ARC, the DDT spilling over to the L2ARC might not have drastic consequences.

Regardless, unless using the DDT feature of ZFS, it doesn't need anymore resources than ext4 or XFS.

po4• 12 years ago

I found an old ZFS disk laying around the other day (a few years old.. from when I was playing with zevo community edition ZFS and I had forgotten what I had stored on it) and I was interested in trying https://openzfsonosx.org so I installed it and mounted my disk up. It worked like a charm. I was pretty impressed.

I kind of screwed myself over though because I to do a bunch of file cleanup that I was hoping to perhaps roll back if it didn't go well so I created a snapshot and then got to work with some copying…

I wasn't paying attention and I ended up running out of space on the disk. The way ZFS snapshots work is that ZFS still does what it would normally do on a copy: writes data, then verifies the write is completed before deleting the old data. When there is a snapshot, it just skips the second step and leaves the old data around. This is how snapshots have zero performance impact on data write.

Turns out that I couldn't even delete the old snapshot (or rollback the new one) because I didn't have enough space to make any changes to the disk. I couldn't even delete files because they would stay in the snapshot.

Aha! I can add a disk to the pool! So I added a small disk to the zpool and then I could delete the snapshot ok. However, I forgot that you can't remove a disk from a zpool. I ended up having to move the data to another disk, drop the entire zpool and recreate it, and move it back.

Honestly, everything worked rather well except for forgetting a few of the limitations of ZFS. Given how often I have issues with HFS+ I really think it's pretty tempting for OS X.

prakashsurya6• 12 years ago

There is work in progress to add the ability to remove a device from a pool. So, hopefully in the not too distant future, that work will land in illumos and migrate to all the downstream implementations (e.g. linux, mac, and freebsd).

ryao18• 12 years ago

This issue was fixed in Illumos a few months ago, although the fix has not made its way to O3X yet:

https://github.com/illumos/illumos-gate/commit/4bb7380495217...

The workaround is to find a file that was written that is outside of a snapshot and truncate it. `zfs diff` can be used to do that. That allows you to delete an old snapshot.

ryao18• 12 years ago

I ported this fix to ZoL and opened a pull request to get it merged into HEAD for the next release:

https://github.com/zfsonlinux/zfs/pull/2784

O3X merges fixes from ZoL, so it should get this fix soon after it has been merged to ZoL.

antman• 12 years ago

I have not used zfs because when I read about it I found the expand-contract procedure overly complicated. (1) Caveats when planning for expansion should be addressed.

(1) http://arstechnica.com/information-technology/2014/02/ars-wa...

ryanqian• 12 years ago

zfs is a great FS, which add many modern concept such as zpool, and also put volume manager and fs together. storage and fs should be easy as this way. and it's in good quality on Solaris, I miss Solaris.

snw• 12 years ago

try one of the Illumos distributions like OmniOS or SmartOS. The legacy of open-solaris is well and alive.

robmccoll• 12 years ago

For private use, yes. I've been using ZoL for over a year on my home server with a simple ZFS raid pool and a number of volumes. Relatively stable. Snapshots every 15 minutes or so. Other than a few briefly scary kernel upgrades - nothing actually broke, just had to rebuild modules. Anytime your system comes up without pools mounted, it's scary. If only the ZFS license were compatible. Anyway, those weren't real breakages, so no problems.

aidenn0• 12 years ago

Can anyone point me to a guide for migrating from software RAID to ZoL?

ryao18• 12 years ago

I suggest emailing the zfs-discuss mailing list for this one:

http://zfsonlinux.org/lists.html

The answer to your question depends on your existing setup.

barrkel• 12 years ago

ZFS on Linux requires more memory than most other FSes, due to the Solaris compatibility layer. The cache is a bit less responsive to low memory conditions.

I run it on a NAS with 4G of memory, and I've had to limit usage and flush caches occasionally, otherwise I get to the point that I can't run new programs.

higherpurpose• 12 years ago

I think F2FS is.

jeffdavis• 12 years ago

The license is pretty concerning. If anyone using ZFS sues Oracle for a patent issue, then Oracle can revoke their license to use ZFS in 60 days. (Not a lawyer, my interpretation might be wrong.)

Probably OK for home use, but not much more. Even a small startup with no patents could end up getting bought by a larger company that does have patents. Then that company could be vulnerable, perhaps even if they are trying to use their patents defensively.

olavgg• 12 years ago

Has anyone tried ZFS together with Hadoop/HDFS? I find it to be a really interesting combination because of compression and self-healing of bad data-blocks.

erikb• 12 years ago

Not being an expert in this kind of area, I would really like to read a short summary. Is there one and I just have missed it, somewhere in the article? I didn't see a headline like summary or conclusions.

ryao18• 12 years ago

Nearly all software works on ZoL. A small number of applications will likely need configuration of the application(s) to turn off AIO (until the next release), configuration of the application(s) to turn off DirectIO, or setting acltype=posix on the dataset to provide POSIX ACLs.

Is ZFS a suitable replacement for other Linux filesystems?

Related Stories