A ZFS developer’s analysis of Apple’s new APFS file system

http://arstechnica.com/apple/2016/06/a-zfs-developers-analysis-of-the-good-and-bad-in-apples-new-apfs-file-system/

by rayiner • 10 years ago

384 187 10 years ago

arstechnica.com

e1ven• 10 years ago

You might also find value in the comments from when this was published via Adam Leventhal's blog -

https://news.ycombinator.com/item?id=11934457

4ad• 10 years ago

A ZFS developer’s analysis of the good and bad in Apple’s new APFS file system

What about linking to the actual source rather than a 3rd party: http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/

cm318• 10 years ago

Well, the article on Ars seems to be authored by Adam, so it's probably a cleaned up single piece of his blog posts.

ahl7• 10 years ago

Yep! They paid me the usual fractional pittance offered to authors.

edit: and to be clear I'm delighted and flattered to see my work featured in Ars!

lee_ars3• 10 years ago

Correct—I reached out to Adam to ask if Ars could syndicate the piece, and then I did some minor cleanup and clarification editing on it (mostly style conformance, but also some minor grammar tweaks and a few sentence re-writes). After checking with him to make sure my changes didn't change anything substantive, we ran the piece this AM.

Link wherever you'd like, of course, but the more traffic this pulls in, the more ammo I have to be able to get Adam contributing to Ars as a regular freelancer!

(edit - hi, adam!)

(edit^2 - corrections corrected. Apologies for the errors. I am just a simple caveman. Your mathematics confuse and annoy me!)

rayiner• 10 years ago

Fun fact: Dominic Giampalo (who wrote the BeOS file system) is on the APFS team. His book "Practical File System Design" is an excellent description of a traditional UNIX file system design. May be out of print now but I think used copies turn up on Amazon.

gherkin0• 10 years ago

Fun fact: Dominic Giampalo (who wrote the BeOS file system) is on the APFS team. His book "Practical File System Design" is an excellent description of a traditional UNIX file system design. May be out of print now but I think used copies turn up on Amazon.

It looks like he has a PDF up on his website:

http://www.nobius.org/~dbg/practical-file-system-design.pdf

greglindahl4• 10 years ago

He's had it up at that url since 2004, with a major update in 2008... says the Wayback Machine.

masklinn• 10 years ago

Dominic Giampalo (who wrote the BeOS file system) is on the APFS team.

I understand he's the APFS lead dev, he gave half the "Introducing Apple File System" session alongside Eric Tamura (the local FS manager)

jonhohle• 10 years ago

I've admired Dominic's work since BeOS, but this was the first time I got to see him present. What a treat!

kbhn• 10 years ago

Fun fact when it's in the article? Third paragraph:

"APFS, the Apple File System, was itself started in 2014 with Giampaolo as its lead engineer...(he built the file system in BeOS, unfairly relegated to obscurity when Apple opted to purchase NeXTSTEP instead)"

frik• 10 years ago

Fun fact: An early developer preview build of an older OSX version (now called macOS) had inbuilt ZFS support.

http://arstechnica.com/apple/2009/10/apple-abandons-zfs-on-m...

agumonkey5• 10 years ago

More than a fun fact. BeOS FS is said to be extremely fast and capable (by the view of its time). The cpp API was clean enough so that a newb me could appreciate the possibilities.

cm318• 10 years ago

The metadata use across applications was great and exactly what others tried to mainstream (WinFS et al). You could search quickly for media files, emails, etc. and save the queries for fast access later on. mbox vs maildir, etc.

Edit: And this wasn't slow and of considerable overhead like what we have right now with indexers for OS-wide search.

louwrentius• 10 years ago

What I found most interesting about the review is that Apple chose not to implement file data checksumming for the reason that the underlying hardware is very 'safe' and already employs ECC anyway.

AceJohnny2• 10 years ago

Similar design decision as IPv6, which doesn't have CRC at the IP level, unlike IPv4, for the same reason.

legulere• 10 years ago

If I remember correctly it's actually the opposite and it was reasoned that checksumming should be done higher up.

ebbv3• 10 years ago

That's not the only reason, though. There's other factors going into that decision that make it totally rational:

APFS isn't designed as a server file system. It's meant for laptops, desktop and most importantly (to Apple) mobile devices. Note that most of the devices are battery powered. That means "redundant" error checking by the FS is a meaningful waste.

That's not to say they might not add error checking capability in the future, but it makes total sense to prioritize other things when this file system is mostly going to be used on battery powered clients basically never on servers.

agumonkey5• 10 years ago

So the idea is that the cloud handle the reliable, APFS is average end-user experience optimized ?

brigade• 10 years ago

Serious question: what value does filesystem checksumming offer for the average user who has no redundancy?

I mean, it'll tell you that your only copy of a file got corrupted, but it'll still be corrupted...

liw• 10 years ago

It tells you that your file is corrupted. You can then restore from backups, re-download, or take some other corrective action, such as delete the file, reboot the machine, re-install the operating system, or play Quake 2 to test your RAM and graphics.

Never underestimate the value of a reason to play Quake 2.

radiowave• 10 years ago

The average user might have no redundancy, but they still ought to have a backup. Checksum failure tells them they need to restore.

At the very least, a checksum failure might tell them (or the tech they're consulting) that they have a data problem, rather than, say, an application compatibility problem.

comex• 10 years ago

Well, Apple is moving in the direction of syncing everything with iCloud - iCloud Drive has been around for a while, and Sierra adds the ability to sync the desktop and Documents folder; of course on top of long-existing things like photo sync. If the file was previously uploaded to iCloud, there is redundancy, and you definitely don't want to overwrite it with the corrupted version.

How big an issue this is in practice I don't know.

mahyarm• 10 years ago

Another way to look at it is checksumming is something that should be done at a generic lower level block layer, not the filesystem layer.

Do it in CoreStorage, not filesystem X.

wmf• 10 years ago

One of the key innovations in ZFS is storing checksums in block pointers which is something that cannot be done efficiently outside the file system. Storing checksums elsewhere is far more complex and expensive.

StillBored2• 10 years ago

Which is silly, and fails to isolate problems when/where they happen. Pretty much every significant level should have its own checksum, in many cases hopefully an ECC of some form. Hardware has bugs/failures as does software. What is particularly evil is when they both collide in a manner which causes silent/undetected corruption of important data for long periods of time.

kartickv3• 10 years ago

I'd prefer them in the software so that they work with external storage like USB hard discs.

chiph• 10 years ago

Because Apple systems are non-expandable, I usually store my large data on external drives (USB, NAS, etc). Any idea if APFS will work on them, either direct connect or iSCSI?

TazeTSchnitzel• 10 years ago

At WWDC it was demoed on a USB flash drive.

lee_ars3• 10 years ago

USB devices can definitely be formatted with APFS, since they did that on-stage during the demo.

I'm less sure about iSCSI targets, but iSCSI is a block-storage protocol. If you can format an iSCSI volume with HFS+ today, you'll probably be able to format it with APFS tomorrow.

raverbashing2• 10 years ago

File system is usually independent from storage media, with some exceptions

(and I would kind of avoid formatting external drives in "funny" FSs as I might want to read them in some other OSs. Unfortunately this usually means FAT32)

But I do have external media as HFS to work with Time Machine

laumars4• 10 years ago

For what it's worth, there's NTFS drivers for Linux and OS X (ntfs-3g). NTFS often gets a mixed reception, but regardless of your opinion of it, it's still massively better than FAT32.

Another option is ext3. The only caveat there is the Windows ext drivers don't fully support ext3 (unless I've missed an announcement). However they do fully support ext2 and ext3 is backwards compatible so you can effective get ext3 support in Windows.

Sadly though, all the good stuff requires 3rd party libraries. It's a real pity everyone can't agree on a standard to replace FAT32. :(

pohl2• 10 years ago

Great article!

Apple contrasts [space sharing] with the static allocation of disk space to support multiple HFS+ instances, which seems both specious and an uncommon use case.

Really? I depend on this use case every year to safely test drive pre-release OS versions.

lisnake• 10 years ago

I'm curios, how do you use this feature to install different OS versions?

pohl2• 10 years ago

Maybe I'm misreading the author, but I take "static allocation of disk space to support multiple HFS+ instances" as a fancy way to refer to multiple partitions on the same physical volume.

(So, Yosemite on one, and a pre-release of El Capitan on the other.)

Space sharing just promises to allow the same thing but without wasting space.

This seems too obvious and important a use case to call "specious and an uncommon".

ahl7• 10 years ago

That use of multiple partitions I think qualifies as uncommon which is not to suggest that it's not useful.

stephencanon• 10 years ago

Extremely commonly used by engineers within Apple, however =)

mietek• 10 years ago

The author is apparently proud of the fact that they have “literally never seen or heard of [the OS X document revision system] until researching this post”. The dismissive tone of the entire article is hard to stomach. Ars has really gone downhill since the departure of John Siracusa.

ebbv3• 10 years ago

John Siracusa's departure was after Ars already had gone downhill significantly. Also he was not an editor or regular contributor outside of his OS X reviews, so his departure doesn't have much effect on the day to day. It's more of a symptom rather than a cause.

The cause is the change in editorial standards for the site which has been decreasing for years, and so every year is at a new all time low. Not to mention the really invasive ads and even sponsored content.

acdha7• 10 years ago

People have been complaining about the editorial standards continuously since at least 2000, but that doesn't magically lend more weight than any other subjective assessment unsupported by evidence.

As for advertising, how do you expect them to hire writers and editors or run a high-traffic website without advertising? They've offered paid subscriptions for years and subscribers don't see ads at all but not enough people have taken them up on it.

symfoniq• 10 years ago

I would trace the beginning of the decline of Ars Technica back to when Jon Stokes sold it to Conde Nast. But Siracusa's departure was another nail in the coffin.

teacup50• 10 years ago

Or, Apple has really gone downhill since the departure of John Siracusa.

ahl7• 10 years ago

The author (me) isn't dismissive; he's skeptical. Siracusa did a great job of upbeat, critical, accurate reviews. Conversely, the coverage of APFS has been inane. A video, a PDF, and some high level docs were shuffled and reworded; that's not a review. Tech journalists live in a tough world where the details are obscure and hard to evaluate with deadlines that are becoming ever shorter and a reward system that values views over depth. My skepticism was meant to counterbalance the fluffy, fawning tone of what I had seen.

I stood around with Giampaolo, Tamura, and other Apple folks. I had a significant leg up on journalists; I know the subject well and the Apple folks know and respect me based on technology I've worked on. (And they didn't kick me out when I revealed that I didn't actually have a conference pass). I knew (some of) the questions to ask, and they were remarkably open with their answers.

Also: https://twitter.com/ahl/status/747074195192832001

Sanddancer2• 10 years ago

One of the (many) things that struck me in the article was his dismissive remarks regarding copy on write within a filesystem. That seems like a fantastically useful feature for development, etc, where you are almost always building, then copying the artifacts into the deploy/test directory. Avoiding the disk IO in those situations seems like a pretty sound win to me in terms of giving a performance boost for free.

erichocean• 10 years ago

was his dismissive remarks regarding copy on write within a filesystem

? ZFS is copy-on-write.

Sanddancer2• 10 years ago

Yeah, I know ZFS is CoW, I use snapshots all the time when building jails/containers precisely for that reason. I was just struck by a few paragraphs from the article that really stuck out:

With APFS, if you copy a file within the same file system (or possibly the same container; more on this later), no data is actually duplicated. Instead, a constant amount of metadata is updated and the on-disk data is shared. Changes to either copy cause new space to be allocated (this is called "copy on write," or COW). btrfs also supports this and calls the feature "reflinks"—link by reference.
I haven't seen this offered in other file systems (btrfs excepted), and it clearly makes for a good demo, but it got me wondering about the use case. Copying files between devices (e.g. to a USB stick for sharing) still takes time proportional to the amount of data copied of course. Why would I want to copy a file locally? The common case I could think of is the layman's version control: "thesis," "thesis-backup," "thesis-old," "thesis-saving because I'm making edits while drunk."

CoW is one of those features that is superficially questionable until you start noticing the bits and pieces of workflow it really makes faster and easier. Given the keynote was towards an audience of developers, I'm really surprised that there wasn't a demo showing how much faster deploys, etc are with such tech.

ahl7• 10 years ago

COW is great. It has applicability around on-disk consisteny, and it makes snapshots much more coherent. It's also not without downsides such as fragmentation--though that's far far less important on SSDs than HDDs. Apple's fast file copy mechanism also uses COW. I would suggest that it's an instance of listening to what your customers say rather than providing them what they need. They say file copies are too slow, but what they really want is a way to track versions, retain old ones, etc. It would be cool if Apple built on their existing approach with regard to file versioning.

tnorgaard• 10 years ago

I would recommend anybody interested in filesystems to watch Jeff Bonwick (ZFS inventor) explain the design of ZFS: https://www.youtube.com/watch?v=NRoUC9P1PmA. They have a few very nice war stories explaining why they found it useful to have the user data checksummed as well.

espadrine3• 10 years ago

I might be wrong, but I read APFL's no-checksum decision as founded upon Apple's hardware-and-software strategy. Given their situation, they can decide that checksumming is a hardware problem. They can build their latest devices with ECC, and require customers to only use Apple-developed USB keys. (I would not be shocked to learn that, in late 2017, Apple starts selling USB-C storage keys preformatted as APFS.)

TillE• 10 years ago

Checksumming at that level is a bit pointless, because then you can't repair the data. Instead of being able to recover from a mirror or parity data, all you get is "it's corrupt, oh well".

ysleepy2• 10 years ago

I think it is a decision based on Apples image. They can only lose when enabling data checksums.

The user would have to be informed, since Apples hardware has no storage redundancy. That will be perceived as an admission of failure on Apples part; your $2k device just murdered your data.

There is not really a UX flow apart from telling the user to recover the file from backup.

Technically it is the dominant strategy to have data checksums. - Even if you assume the Hardware is very awesomely perfect.

fs111• 10 years ago

so refreshing to see a humble presentation by an engineer and not a meme laden ego show, like you mostly see these days.

adiabatty• 10 years ago

You may be interested in http://daringfireball.net/thetalkshow/2016/06/17/ep-158 where Craig Federighi discusses how Apple implemented wear-your-Apple-watch-to-unlock functionality for their Macs; as it turns out, they make calculations based on how fast the Bluetooth responses are to judge the distance between the laptop and the watch — that way, a snoop can't unlock your laptop if you're merely in the room. It's nice to know that Federighi, at least, is involved in the nitty-gritty even though these aren't reflected in the keynotes.

rdtsc• 10 years ago

Thanks. Just watched. What a great presentation.

I really like the hack of a block pointers being a data structure and containing the birth time (tx id I guess) and how that avoids needing managing bitmaps.

The advantage of doing send/receive vs rsync was also a nice explanation.

It is not often that you see a technology and think "Oh, this is great stuff", this is like that with ZFS. Haven't played much with it. But get that idea just from learning about it so far.

Both Jeff and Bill are great at communicating and explaining the technology. Like how they tag team, with minor funny bits here and there.

Regarding the main issue here, checksums -- yeah I don't see how Apple engineers could have watched this and said "Meh, don't need data checksums". Maybe they do have a secret vault with magic new holographic storage, immune to cosmic rays and other vagaries of physics, who knows.

Etheryte• 10 years ago

The original article was on HN's front page when it was originally written.

throwaway2626• 10 years ago

APFS is slated for a 2017 release, yet development started as recently as 2014. By comparison, development on Btrfs started all the way back in 2007, yet many still consider it to be unsuitable for widespread deployment, particularly in mission-critical settings.

If Apple can actually pull off this turnover so quickly, does that suggest the complaints about Apple's declining software engineering quality were overblown?

Edit: Ted T'so in this talk(1) (at the 8 min mark) discusses the taskforce that birthed Ext4 and Btrfs and its estimate (based on Sun's experience with ZFS, Digital's with AdvFS) of 5-7 years at a minimum for a new file system to be enterprise ready--an estimate which definitely proved optimistic with regard to Btrfs. Will APFS be different?

1: https://www.youtube.com/watch?v=2mYDFr5T4tY

milcron• 10 years ago

Maybe Apple's software quality declined due to engineering prowess being siphoned away towards skunkworks projects like APFS

espadrine3• 10 years ago

While APFS' design mirrors that of current generation file systems such as ZFS and Btrfs, its scope is much reduced. It does not target large amounts of storage or disk arrays (such as RAIDZ and the normal RAID family) with automatic error correction, it is not required to perform competitively in server scenarios (especially for traditional databases, where COW file systems tend to perform poorly), and it can completely disregard rotating hard drive performance.

Snapshots are read-only, unlike Btrfs (but like Linux' LVM, which has been around for quite some time), and we won't even delve into online rebalancing and device addition.

APFS is a very welcome addition given that it brings Apple file systems into the XXIst century, but the reason for its speedy implementation is that they purposefully constrained its goals — a good call, in my opinion.

If you have the same workloads as Apple's customers and large amounts of SSD storage, you'll find Btrfs and ZFS both fit your needs on Linux.

ahl7• 10 years ago

I'm inclined to agree that 3 years to squeeze out a perfect file system is ambitious to say the best. That said, consider the robustness of HFS+; APFS is blessed with desperate, captive users.

coldtea5• 10 years ago

That said, consider the robustness of HFS+

Besides being old, there's nothing much to the "robustness of HFS+". It's not like users are losing data left and right, as its being painted. In fact it's an FS running just fine on about a billion devices...

coldtea5• 10 years ago

If Apple can actually pull off this turnover so quickly, does that suggest the complaints about Apple's declining software engineering quality were overblown?

The complaints are usually about minor parts of the OSX/iOS stack -- parts Apple might not even particularly prioritize.

A faulty filesystem on the other hand is something entirely else altogether and something they can't ship unless it's good.

TheCondor• 10 years ago

They toyed with zfs quite a few years back, when they opted out don't you expect that they started s project?

cyphar9• 10 years ago

[Disclosure: I work for SUSE]

an estimate which definitely proved optimistic with regard to Btrfs.

SUSE has had Btrfs support for enterprises since SLE11SP2[1,2] (2012). And it's been the default for the root partition since SLE12[3] (2014). So it wasn't overly optimistic at all, it was actually a very accurate estimate. The same support apples for openSUSE, but I think they had it for longer.

[1] https://www.linux.com/news/snapper-suses-ultimate-btrfs-snap... [2] https://www.suse.com/communities/blog/introduction-system-ro... [3] https://www.suse.com/documentation/sles-12/stor_admin/index....

viraptor• 10 years ago

Do you have any good source to point people at when they say btrfs is unstable and corrupts data? (and I mean, either positive or negative data, I don't want to be biased either way) I'm kind of tired of people posting comments like that, which are based on "google is full of people saying this", or "I know one person who lost data" (but no idea which kernel was it - could well be from 2014).

If I look at btrfs patches on lkml, on one hand side I can see some fixes for data loss, but on the other they're usually close to "if you change the superblock while log is zeroed and new superblock is already committed and there's a power loss exactly at the point, you'll get corruption" - which are just really obscure edge cases people are unlikely to ever hit.

So what can I look at to get a realistic picture of what's going on? (what would SUSE point me at)

(for the negative results, I know of the recent filesystem fuzzing presentation where btrfs comes out worst, but honestly I don't consider it interesting for real world usage - car analogy, I'm interested how the car behaves on a typical road, not which fizzy drinks added to the gas tank will break it)

developer2• 10 years ago

> does that suggest the complaints about Apple's declining software engineering quality were overblown

The type of developer you would have working on a filesystem are likely going to be from a different world than those who work on UI applications. Speaking about their apps, the quality problems Apple faces are usually design-based rather than functionality. When working on a filesystem, you're not going to be forced into developing a crappy application by some designer who ruins the entire application.

shmerl5• 10 years ago

Do they explain the reason why they needed a new filesystem instead of using some existing one (same OpenZFS)?

cm318• 10 years ago

We can speculate on the management decision, but from an engineering point of view Dominic Giampaolo said that he didn't dive into btrfs, ZFS, or HAMMER in order to not get potentially bad influence. At least that's how I read his answer as cited by Adam. It's interesting that Larry (Oracle CEO) being a good friend of the late Steve Jobs didn't result in finding an acceptable ZFS licensing agreement for Apple. I mean, they've incorporated DTrace (into the kernel of all things), so why not ZFS as well?

shmerl5• 10 years ago

from an engineering point of view Dominic Giampaolo said that he didn't dive into btrfs, ZFS, or HAMMER in order to not get potentially bad influence.

I'm not sure I quite understand this approach. It's sounds like a pure NIH kind of method (which is admittedly common for Apple). I.e. of course one can always try to reinvent the wheel, but why is it bad first to analyze what already exists and to evaluate good / bad sides of that? Or his approach is simply always to make everything from scratch and not to look at anything else?

btbuilder• 10 years ago

Another possible interpretation of "to avoid becoming tainted" is that it refers to avoiding "residual knowledge" that in some circumstances can be used to make a copyright claim. E.g. if someone reads the ZFS source code to see how they solved a particular problem and then goes and solves it the same way that residual knowledge might cause legal problems.

That being said IANAL and have no idea whether residual knowledge has been explored w.r.t. the GPL or CDDL.

rincebrain2• 10 years ago

A tracing framework is much easier to rip out if it presents legal difficulty than a filesystem.

tjl3• 10 years ago

For one, you're not going to put ZFS on a watch. They want a file system that can scale to all their devices.

shmerl5• 10 years ago

What about BTRFS? It's more lightweight. Or it's very Linux specific?

And I'm not sure you need snapshots and many of those industrial features on a watch anyway. Sometimes simplicity is actually a plus. So who said they need "one shoe fits all" to begin with? It's never a good approach.

forgotpwtomain• 10 years ago

APFS addresses this with I/O QoS (quality of service) to prioritize accesses that are immediately visible to the user over background activity that doesn't have the same time-constraints. This is inarguably a benefit to users and a sophisticated file system capability.

Could someone clear up how this is can be determined on the filesystem rather than scheduler level (I suspect it cannot be, or the article is making bogus claims)?

astrange3• 10 years ago

Why couldn't they be? Those are in the same process and can talk to each other.

HFS+ implements scheduling of background QoS threads, especially on HDDs - you can see it working with 'spindump'.

A ZFS developer’s analysis of Apple’s new APFS file system

Related Stories