Bcachefs Status Update

https://lore.kernel.org/linux-fsdevel/Y1wsQVSQ6ipWyFlX@moria.home.lan/

by pantalaimon • 4 years ago

94 37 4 years ago

lore.kernel.org

pdimitar• 4 years ago

I didn't understand even half of that but am still excited. I have no less than 6 external HDDs and SSDs lying around and it's been a pain to make a good stable ZFS pool out of them. `bcachefs` seems like a perfect match.

soopurman• 4 years ago

I eagerly await this work stabilizing to the point of merging to mainline. I'm impressed with how much progress Kent has made, but I'm frustrated by how often he seems to say some feature is basically ready except for this bug and except for that problem. I understand great designs often come from the mind of a single inspired individual, but I hope that he's ready to accept contributions from others and that more developers see the value in helping to get this across the finish line.

koverstreet7• 4 years ago

I'm _always_ happy for other people to jump in :)

But the reality is that filesystem development is hard, and there aren't that many people clamoring to work on this stuff - but that's ok. Slow and steady wins the race, in the end.

I'm a big believer in steady incremental development, and being up front about what works and what doesn't - and also making sure the core is a solid foundation for everything we want to do.

For anyone who does want to get involved - come join us! irc.oftc.net#bcache

slavapestov2• 4 years ago

Unfortunately, file system development is a pretty niche skill set these days, and the majority of the experts in the field are employed maintaining existing file systems (ext4, xfs, apfs, etc).

One thing I’ve been bugging Kent to do is to write documentation about the design and internal workings on bcachefs; very little about modern file system design is actually written down anywhere, and a detailed reference manual would attract more people to work in this area.

koverstreet7• 4 years ago

That exists! I just always forget to link to it (and I do need to do more work on it): https://bcachefs.org/bcachefs-principles-of-operation.pdf

pedrocr2• 4 years ago

Does anyone know if any of these next generation filesystems can do raid6 with more flexibility than normal linux software raid?

Say you do a 6x4TB raid 6 array in Linux and get a 16TB virtual drive. Now if you update 4 of the drives to 8TB will still get 16TB with 4x4TB of fully unused space. But 4x4TB is itself enough to do 8TB of raid6 that would get you a total of 24TB usable space where any two drives can fail. Does any of these filesystems just do that automatically and continuously upgrade the capacity as you phase in new disks?

RealStickman_• 4 years ago

BTRFS can do that, but RAID 5/6 use is discouraged to say the least.

kzrdude• 4 years ago

Synology seems to use it, doesn't it?

mastax2• 4 years ago

ZFS doesn't do it as you phase in disks, but once you've replaced all of them you get the extra capacity.

pedrocr2• 4 years ago

Linux raid does that already. You can also add disks to grow an array or change it from raid5 to raid6. The gradual phase-in is what's missing. It makes sense as having what is effectively a raid0 of a 4 drive raid6 and a 6 drive raid6 in the example is a more complex configuration with uneven performance across the array. But for a home NAS meeting the reliability with maximum space ends up being all that counts.

insanitybit• 4 years ago

Great to see this progress. This is one of the projects I'm happy to sponsor.

mastax2• 4 years ago

The talk linked at the end is really interesting, even as someone who doesn't know anything about filesystems outside of the classic Unix they teach in school and reading a lot of deep discussions about tuning ZFS. (None of my ZFS users are strenuous enough to require any tuning but I can't help but learn and optimize when there are levers to pull).

It's interesting that it's implementing all these modern features on top of extent-based non-COW transactional btrees (though it looks like APFS is the same way). With the very database-like architecture at the low level, it would be fun to stick a SQL interface on top. You should be able to do advanced queries very efficiently without having to fall back to scanning every file in the directory like you would normally have to.

I'm salivating looking at the caching/tiering features. People often try to implement a tiered filesystem with ZFS L2ARC, SLOG, and now the special vdev. But none of those features were intended for that and it shows.

justinlloyd3• 4 years ago

I was able to solo-developer build a reasonably complex CI/CD system for a company that relied heavily on bcachefs in a tiering system: RAM --> SSD --> HDD, to run hundreds of builds per day across dozens of SKUs measuring in the tens to hundreds of gigabytes per build directory. Later was able to port bcache (not the fs) to a vmx driver for VMWare esxi because of that understanding I had gained earlier in the project, again, RAM to SSD to HDD. Which then lead to making a down'n'dirty FusionDrive script (it's on github) for macOS that can tier RAM and SSD for doing fast CI/CD agent builds on macOS.

koverstreet7• 4 years ago

I'd love to hear more about that

metadat• 4 years ago

More info and highlights about bcachefs:

https://bcachefs.org/

Bcachefs is a new filesystem for Linux.

Copy on write (COW, like zfs or btrfs), Full data and metadata checksumming, Multiple device support, Replication, Compression, Encryption, Snapshots, Caching

Erasure coding (not quite stable)

Scalable - tested to 50+ TB

Already working and available today

brnt• 4 years ago

It sounds great, espcially erasure coding: bit rot resistance up to a point? Yes please!

Bit hesitant to give it a go though, a stable filesystem that is calamity resistant is such a core thing to have.

Is there anyone with larger scale experience with bcachefs?

tmulcahy3• 4 years ago

The lock ordering rules had become too complicated and this was getting us too many transaction restarts, so I stole the standard technique from databases

What is the standard technique from databases?

koverstreet7• 4 years ago

You detect the deadlocks when they happen and abort

tmulcahy3• 4 years ago

Any details/links that explain this? The article seems to suggest that they used to detect the deadlock as it was about to happen, and then abort everything and retry. This doesn't seem too different from "when they happen". What is the optimization?

slavapestov2• 4 years ago

Bcachefs is one of the most interesting projects going on in Linux kernel land these days, all the more remarkable because it is mostly the work of one person over the last 12 years. I hope it goes upstream soon; with a bit of polish it should be ready for widespread adoption.

mekster• 4 years ago

Questions is, if it's that interesting how is it only a single person project for that long? Any sponsors?

2pEXgD0fZ5cF• 4 years ago

Always excited to hear news about Bcachefs, can't wait for it to hit upstream!

Quekid52• 4 years ago

Just curious about the mention of persistent data structures. Is this about the FP notion of persistent data structures (which I suspect) or on-disk data structures? Or both? :)

I find that (FP) persistent data structures are like a super-power in lots of ways. (And it only costs a log-N slowdown in the worst case.)

koverstreet7• 4 years ago

On disk, transactionally-updated data structures

Quekid52• 4 years ago

Thanks for the clarification.

gautamcgoel• 4 years ago

Question from a filesystem n00b: what is the advantage of Bcachefs relative to standard filesystems like Ext4?

LinAGKar• 4 years ago

It's got about the same features as btrfs, and other new-gen filesystems like ZFS, APFS and ReFS, such as:

- Copy-on-write

- RAID 0/1/5/6 equivalents

- Subvolumes

- Snapshots

- Transparent compression

- Reflinks

- Metadata and data checksuming

- Replication

- Bitrot protection (when used with RAID and checksuming)

And it's supposed to be more though through and well designed than btrfs.

Shish2k• 4 years ago

I’m currently using a few different filesystems for different use-cases, and looking forward to bcachefs because (as I understand it) it covers some use-cases that I’ve been dreaming of for ages and not yet found any solution for:

- raid5/6 with odd-sized disks (btrfs does raid1 with odd-sized disks, mdadm & zfs do raid5/6 with equally-sized disks) - great for a homelab server that I upgrade ship-of-theseus style rather than the enterprise “buy a whole new server with whole new larger disks in one go”

- SSD as a cache integrated with HDD RAID (right now I’m using bcache + mdadm + ext4 for this usecase; having the filesystem coordinate all three of those layers feels more elegant and efficient)

- Different settings for different directories in the same filesystem (my super-important documents get RAID1 across 4 drives, my blu-ray rips that I can just re-rip if I lose them get RAID0, everything else gets RAID5)

aborsy• 4 years ago

It seems synology will use it next year in DSM 7.2?

Bcachefs Status Update

Related Stories