josefbacik

I’m framing this comment and putting it on my wall in my office.

Keep in mind most of us don’t use our @fb.com addresses. LWN keeps track of companies by keeping track of who works where so our numbers are much higher than this list shows.

Agreed, drive managed is far better from a usability standpoint.

Host managed also means messing with the file system, and the patch set for btrfs is not small. Plus you have to disable a bunch of features (like preallocation since you can’t move the write pointer backwards) which is going to surprise people in unfun ways. These drives are useless for general purpose. If you are going to use them as expensive tape then by all means, but otherwise I have serious doubts about their usefulnes.

Edit: I’m talking specifically about the SMR side. The general zoned stuff is interesting, but when you start putting restrictions on how you can write to certain zones you wind up with a lot of weirdness that application developers are going to be surprised by.

Not sure what you are talking about, it’s definitely endian neutral. You can’t take different page sized fs and put it on another page size arch, but that’s a separate issue.

Cause it’s a summit to talk about the whole ecosystem, not one file system. We btrfs developers talk enough to each other, we don’t need to bore the rest of the attendees with our own topics. That’s what the hallway tracks are for. I put together the schedule for the file system side of this years conference, I specifically don’t put single fs issues on the general schedule.

It would be nice, but for complicated systems in the kernel you sometimes have to provide evidence that the old way sucks and the new way is better. With this work we can easily point at our production workloads and say the in kernel oom killer sucks and this new way is obviously better and hopefully work towards a better in kernel solution. That being said the user is always going to know what they care about more than the kernel, so being able to say “kill chef first, alway” can be a nice thing to have.

I spent an unreasonable amount of time working on grub2. The stuff I looked at the most where the uefi spec

http://wiki.phoenix.com/wiki/index.php/Category:UEFI

And then the uefi stub code in the Linux kernel, as well as the grub2 source code. It’s not too hard to figure out the flow in grub2, just figure out where it does EXIT_BOOT_SERVICES and work from there.

Lol what? We don't ship code that isn't upstream. It sucks to maintain and rebase if you have a shitton of proprietary patches. Everything we do with the kernel is open source, you wouldn't be able to hire kernel developers if you didn't do open source. Nice comment tho, I legitimately lol'ed.

Facebook is using it for lots of shit. Chroots for containers because we can easily snapshot them and use them. Use it for build testing, so snapshot base repo, checkout commit, build, throw away. Gluster which takes huge backups and then other random workloads, it's used in a few places.

Just because Facebook is fault tolerant doesn't mean we don't care about failures. We actively run down any issues we hit, so while it doesn't have much of impact in the short term, we don't just ignore issues.

And also Red Hat hasn't contributed much to btrfs in years, and Oracle has one developer working on it. We are constantly working on it, and now that my priorities have shifted back to btrfs I hope that we will start to close out some of these long term projects.

A Linux file system developer in theory, a "this is broken and is in/related to the kernel" developer in practice.

Sendfile isn't async, it just avoids the user space copy, and in some cases the copy from the fd to the socket. Aio doesn't require xfs, just O_DIRECT, which can be tricky to get right and more work than userspace guys want to deal with.

All this talk about Oracle is just plain stupid. Oracle doesn't control anything, the community does. One core developer still works on Btrfs from Oracle, the vast majority of the contributions come from outside Oracle.

Now as to

"Why Red Hat does not have engineers to support btrfs?"

You have to understand how most kernel teams work across all companies. Kernel engineers work on what they want to work on, and companies hire the people working on the thing the company cares about to make sure they get their changes in.

This means that the engineers have 95% of the power. Sure you can tell your kernel developer to go work on something else, but if they don't want to do that they'll just go to a different company that will let them work on what they care about.

This gives Red Hat 2 options. One is they hire existing Btrfs developers to come help do the work. That's unlikely to happen unless they get one of the new contributors, as all of the seasoned developers are not likely to move. The second is to develop the talent in-house. But again we're back at that "it's hard to tell kernel engineers what to do" problem. If nobody wants to work on it then there's not going to be anybody that will do it.

And then there's the fact that Red Hat really does rely on the community to do the bulk of the heavy lifting for a lot of areas. BPF is a great example of this, cgroups is another good example.

Btrfs isn't ready for Red Hat's customer base, nobody who works on Btrfs will deny that fact. Does it make sense for Red Hat to pay a bunch of people to make things go faster when the community is doing the work at no cost to Red Hat?

Nope I love Red Hat and loved working for Red Hat and still interact with most of my colleagues there on a day to day basis. I shouldn't be speaking for everybody, but from what I can tell we're all pretty happy where we are, so no real reason to switch companies.

They brought on Zach right before I left specifically to help with the effort, but he left as well. I can't really speak to Red Hat's overall strategic decisions, but really they have a large local file system team, and a lot of them are xfs developers. You aren't going to convince Dave Chinner he should go work on Btrfs instead of XFS. Unless there's somebody internally that actually wants to work on Btrfs the work simply isn't going to get done. All of the other Btrfs developers work for other companies, and none of them are interested in working for Red Hat.

People are making a bigger deal of this than it is. Since I left Red Hat in 2012 there hasn't been another engineer to pick up the work, and it is _a lot_ of work.

For RHEL you are stuck on one kernel for an entire release. Every fix has to be backported from upstream, and the further from upstream you get the harder it is to do that work.

Btrfs has to be rebased _every_ release. If moves too fast and there is so much work being done that you can't just cherry pick individual fixes. This makes it a huge pain in the ass.

Then you have RHEL's "if we ship it we support it" mantra. Every release you have something that is more Frankenstein-y than it was before, and you run more of a risk of shit going horribly wrong. That's a huge liability for an engineering team that has 0 upstream btrfs contributors.

The entire local file system group are xfs developers. Nobody has done serious btrfs work at Red Hat since I left (with a slight exception with Zach Brown for a little while.)

Suse uses it as their default and has a lot of inhouse expertise. We use it in a variety of ways inside Facebook. It's getting faster and more stable, admittedly slower than I'd like, but we are getting there. This announcement from Red Hat is purely a reflection of Red Hat's engineering expertise and the way they ship kernels, and not an indictment of Btrfs itself.

Or I just mistyped and you are reading in too much to it?

Systemtap was fantastic, but super finnicky. You had to have debuginfo for the kernel installed plus the source. This was fine for RHEL but kind of a pain elsewhere. Then you run into other problems like sometimes the distro compiler wasn't used to build the kernel you are running and suddenly you couldn't load any stap script. Also the only way to get output was through stdout, there was no easy way to programmatically pull results from an stap script.

BPF/BCC solves most of this. You still have to have the sources so you can get the right targets for kprobes, but you don't have the compiler mismatch problem. You can now access the hash maps directly from user space so you can do things like build system monitoring tools that run in production and record really specific information easily. I used systemtap for years, but bcc/bpf is a whole new world.

You are confusing hardlinks with symlinks. They are similar but not the same. Symlinks in Linux most certainly get their own inode and can contain data blocks.

Symlinks are implemented as files in Linux, we write the path into the file that you are pointing to, so there most certainly is data that must be fsync()'ed if you want it to be persistent.

Edit: If you want to fsync the symlink you can do an open with nofollow iirc (on my phone so I can't check) so you get the actual symlink and not its target.

The rename is atomic, the data being in the file is the problem, there is no garuntee unless you fsync.

This is just an accident of how some file systems are implemented and isn't actually garunteed. If you did this on xfs you could still end up with the a 0 length symlink if you crashed at just the right time.

We have to look up the physical extent in the extent reference tree, so the cost is independent of the number of snapshots and more a function of the fragmentation of the extent tree. The metadata is all cached of course, but fragmentation means you are likely to not find the entries in cache.

The other aspect that I haven't talked about is our fsync performance is kind of shit compared to other fs'es. Now this does get better in the nocow case but it's still pretty heavy and needs optimization.

Yeah nocow only works if there are no snapshots, which is why we have to check, so if you use snapshots it falls back to COW.

No it doesn't.

Not sure how that tracks. These file systems are huge code bases with lots of problems and areas for improvement. When you only have 2-4 people who understand most of it at a given time things move slowly. If I could tell somebody to go fix balance my life would be a whole lot easier, but in reality we end up being interrupt driven and have to prioritize differently.

ZFS has a 7 year head start on BTRFS and has traditionally had about 5 times as many core contributors at any given time so I imagine they've solved this in some novel way by now.

NOCOW is horribly expensive because we still have to go check and make sure that there are no snapshots pointing at the changing extents. It only solves the fragmentation issue, and if you don't prealloc your image it doesn't even do that.

There are no docs. I'm not a qcow2 expert, what I know is very basic so anything I say about qcow2 can be very wrong.

So qcow has a read only base image that gets updated when we change things. The image format just had the changes from the original image. So you update a package, it adds some metadata to point at the new stuff and adds the data in and you are done.

So with btrfs you have this image on top of btrfs, so you update a file and its metadata inside the image. Say you start with a pristine image that's in nice big extents. You update a package which changes small chunks all over the file. Let's say you update 12 4K extents. So now instead of one extent you now have 36 extents. This affects everything, fsyncs take longer because there's more extents we have to write out, the space is more fragmented so cold cache reads are more expensive, the csums are no longer contiguous so they also take up a larger more fragmented area. It has this really terrible cascading effect.

It's a general purpose file system, it does great with metadata heavy workloads and normal streaming writes. The overwrite case is special because of COW. Eventually we'll be comparable to everybody else but we aren't now.