⁠Btrfs has been deprecated in RHEL

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.4_Release_Notes/chap-Red_Hat_Enterprise_Linux-7.4_Release_Notes-Deprecated_Functionality.html

by alrs • 9 years ago

368 338 9 years ago

access.redhat.com

drawnwren• 9 years ago

I know this says redhat.com, but title is still a bit misleading. Btrfs has been deprecated on RHEL, but it is still under active development on its own.

em3rgent0rdr2• 9 years ago

HN has a problem of users posting misleading titles.

josteink• 9 years ago

Deprecated? In favour of what?

Will Redhat too (like Ubuntu) start shipping ZFS?

tbrock• 9 years ago

XFS is the default FS on RHEL now, so likely that.

vacri2• 9 years ago

Do you know if XFS can shrink volumes yet? As far as I'm aware, that's the only limitation it has compared to other filesystems of that era.

mjevans• 9 years ago

I supposed their upgrade path is...

* Make fresh backups

* Verify the backups

* Re-install and use the backups

paulie_a2• 9 years ago

Basically. Those steps should be done regardless of FS. I heard this on a podcast recently but if you are not doing that you end up with "schrodinger backups"

paulie_a2• 9 years ago

The following will sound snarky but I would personally prefer ext2 if the other choice is btrfs. "ButterFs because your data melts away"

riffraff2• 9 years ago

I am curious as to the same. The document says

Red Hat will continue to invest in future technologies to address the use cases of our customers, specifically those related to snapshots, compression, NVRAM, and ease of use.

but it's unclear what this means exactly.

pinewurst2• 9 years ago

Like their Permabit acquisition: https://www.redhat.com/en/about/press-releases/red-hat-acqui...

dman• 9 years ago

XFS - the workhorse that keeps on running.

pinewurst2• 9 years ago

My best guess is XFS coupled with Permabit (which they just bought and will open source) compression and dedupe services. Probably layered on an enhanced mdraid too.

I'm guessing that the ZFS licensing hairball is a bridge too far for even Red Hat, so they'll cobble together equivalent-ish functionality - even if it's not anywhere near as elegant as ZFS's integral data production and reduction.

silly-silly• 9 years ago

Red Hat will never ship ZFS, because its an entity that exists in the US and a probable target for lawsuits / license violation if ZFS is included.

cookiecaper3• 9 years ago

What's the risk involved for Red Hat in shipping ZFS? OpenZFS and ZFS-on-Linux are under the CDDL, a legitimate open-source license that some feel may be GPL-incompatible. Red Hat distributes non-GPL programs as a matter of routine, and I'm sure this includes other CDDL programs, especially considering Red Hat's enthusiastic involvement in the Java ecosystem.

The only potential risk is that the GPL is so virulently infectious that any driver is automatically GPL'd by virtue of its own existence as a compiled kernel module, but that possibility seems fairly remote, and it hasn't seemed to affect the distribution of other purportedly-non-GPL kernel modules.

I'm not a lawyer so maybe I'm missing something.

JoshTriplett• 9 years ago

Has anyone here tried bcachefs (http://bcachefs.org/) for some of the same use cases as btrfs? What do people think of its current state?

lathiat12• 9 years ago

You'd need to read up on it but the TL;DR i got from previous comments was that previously it was being funded by a company that wasn't 100% aware of the entire situation around it (they used it in their commercial product, management didn't fully understand it was being released as open source, though I also understand they didn't own the rights to the entire code base so basically that was necessary effectively). The person developing it left that company and is seeking his own funding for it but that generally the development was significantly stunted as a result.

It seems a bit here-say-ish though, so please don't assume I'm entirely correct on all fronts there and I'd encourage you to research it further!

riffraff2• 9 years ago

OT, but I think you meant "hearsay-ish"

joecool10292• 9 years ago

I'd encourage you to research it further!

On that note, their Patreon is a better primer than the website is: https://www.patreon.com/bcachefs

koverstreet• 9 years ago

Oh, they knew it was being released as open source. There was a clear boundary between the open source and the proprietary code - I worked on both. The open source thing was just a convenient excuse for some political bullshit - either that or they thought they could buy out a GPL project and take it proprietary, which is just insane. I mean, half the copyright was Google.

joecool10292• 9 years ago

Looks neat. Nobody uses it though and LKML has zero discussion on the announcements.

However, HN has a previous thread on it here: https://news.ycombinator.com/item?id=12410798

geota• 9 years ago

I actually use bcachefs on my main data drive and so far it has been running solid for ~3months. That said, looks like the author has gone camping/moving/re-evaluating life. Hoping the development picks back up or that it gets rebased off of 4.12 (or maybe made into a set of kernel patches?). Wanted a COW fs - tried btrfs and kept running into issues where the drive would fall into ro mode so far not regretting bcachefs.

https://aur.archlinux.org/packages/linux-ck-bcachefs/

sargun3• 9 years ago

I've looked into BCacheFS extensively. Two problems I have -- First, it's not there in features yet. It doesn't have quotas, nor snapshots.

Second, it lacks a formally published design for these features. Because the author is the only person with knowledge of how these things might work, it makes it really difficult to mitigate the bus factor while the project is still in heavy development.

lathiat12• 9 years ago

This is specific to RHEL7, notably that they won't backport any further kernel updates and won't move it from Technology Preview to release. Red Hat wasn't really driving btrfs development at all from what I am aware of.

⁠Btrfs has been deprecated

The Btrfs file system has been in Technology Preview state since the initial release of Red Hat Enterprise Linux 6. Red Hat will not be moving Btrfs to a fully supported feature and it will be removed in a future major release of Red Hat Enterprise Linux.

The Btrfs file system did receive numerous updates from the upstream in Red Hat Enterprise Linux 7.4 and will remain available in the Red Hat Enterprise Linux 7 series. However, this is the last planned update to this feature.

Red Hat will continue to invest in future technologies to address the use cases of our customers, specifically those related to snapshots, compression, NVRAM, and ease of use. We encourage feedback through your Red Hat representative on features and requirements you have for file systems and storage technology.

jmspring2• 9 years ago

It's OT but how many people actually use RHEL? Or CentOS (more likely).

I'm not a fan of systemd, but have really loathed CentOS/RHEL compared to Debian/* for years.

esaym• 9 years ago

It's OT but how many people actually use RHEL?

Hopefully a lot because, like it or not, RHEL puts a ton of work into the linux kernel and other userland linux tools. We need them to stick around.

jmspring2• 9 years ago

As do others and they were one of the main systemd backers. Still doesn't answer the question.

If I'd left out my preference, probably would have had fewer downvotes.

myrandomcomment• 9 years ago

As a general rule if the end customer was finance/bank their Linux flavor was RedHat (someone to yell at when it broke). Other tech companies / web types it was Ubuntu. This was my experience in over 4 startups where these were are end customers. Your mileage may vary.

jsmthrowaway• 9 years ago

CentOS is extremely popular among startups. It lets you directly use RPMs, which many software publishers distribute on their own (not the case for debs, often), and gives you an upgrade path to buy RHEL when you want someone to yell at and not throw away all your automation.

I’m not trying to start a flame war here (honest), but among most “usual web” ops folks I know, the opposite of what you’re suggesting seems to hold and Debian/Ubuntu are looked at as an odd choice. It’s probably hire #1 going with what they know, more than anything, and it could very well be selection bias regarding the people I know. I’d love to see stats.

I’ve been working with CentOS or OEL in my own roles for several years now, and I’ve never chosen it. (Not saying I wouldn’t, just that it’s been there when I get there.)

wvh2• 9 years ago

Pretty much my exact experiences. Banking and large corporations: RedHat, because they buy support contracts to have guaranteed security and stability updates, and because they want some assurance that whatever (open/closed) corporate software they use will run on their Linux systems. Debian and especially Ubuntu the last years for smaller "dev" companies because the devs like to be close to the open-source ecosystem.

I've never met an individual (not corporate server) or end-user (desktop) running RHEL though.

Back to topic, I've been following Btrfs... At one point it seemed destined to become the default for most Linux systems, but I'm not sure where it is headed now.

jeltz5• 9 years ago

In my experience Debian seems to be more common in Europe. Here there is a mix of RHEL, Debian, Ubuntu and some CentOS. With banks and finance preferring RHEL.

WillyOnWheels• 9 years ago

Redhat is worth billions of dollars. Someone must be using Redhat.

madez3• 9 years ago

Tulips were sold for ridiculous prices, yet nobody used them.

krylon2• 9 years ago

We run one CentOS 6 server at work (which uses upstart, not systemd). The initial reason was that it needed to run on Hyper-V, and the Debian VM I had set up first, kept losing its network connection every couple of minutes, which was especially annoying since I wanted to run Nagios on that system. ;-)

So I did a bit of digging and found out that CentOS ran on Hyper-V just fine. I installed it and had no real reason to complain since. The selection of packages is rather spartan compared to Debian, but there is a third-party repository called EPEL that makes up for that.

Some things are different from Debian, of course, but nothing big, and the system has given me no problems whatsoever on the performance and reliability fronts.

fulafel3• 9 years ago

EPEL doesn't get security updates.

Security support for a wide range of packages is also a reason to prefer Debian over Ubuntu, since most of the Debian-inherited packages ("universe") are excluded in Ubuntu.

coredog64• 9 years ago

Amazon Linux is CentOS 6.something with new kernels and security fixes for the associated packages.

abrookewood2• 9 years ago

Which is fine if you are only running on one cloud provider and don't have any local infrastructure. But I want my one Linux flavour to run anywhere ...

fulafel3• 9 years ago

The E stands for "Enterprise", so there's a hint about the target audience. You can get support for a 10 year old RHEL version after you entangle it to your stuck in time IT infra :)

fleetfox• 9 years ago

It seems like "Enterprise" is synonym with "Technical debt"

rdtsc• 9 years ago

It's OT but how many people actually use RHEL? Or CentOS (more likely).

I'll ask the opposite question how many people in Enterprise don't use RHEL/CentOS?

I'm not a fan of systemd, but have really loathed CentOS/RHEL compared to Debian/* for years.

Those two are orthogonal. A lot of people loathe Windows and Oracle yet they are raking in billions every year in licensing.

Then there is Government as well, try selling them a production running in Ubuntu or Arch. It can be done but it won't be very easy. There RHEL is king as well.

abrookewood2• 9 years ago

I used to run CentOS in Dev/Test and then Red Hat in Prod. However, they have a habit of removing packages that compete with their commercial offerings, which forces you to either compile from source or to install from other repositories - for example, they removed Apache ActiveMQ because they offer JBoss AMQ. This starts to become annoying becuase all of a sudden you have multiple update processes to keep everything up-to-date, whereas previously, you had one. After being bitten by this at least twice, I shifted everything to Ubuntu. It's a shame - both CentOS & Red Hat are slow to move, but very stable. I was happy paying them support too ... but not when they kept making things more difficult.

lucaspiller• 9 years ago

I was working at a place a few years ago where I had to move a whole bunch of applications, running on Ubuntu Hardy circa 2008, to new servers (well VMs), as the servers were being decommissioned. A lot of the applications were running on Ruby 1.8 which newer versions of Ubuntu don't provide packages for, and it's not very easy to compile it yourself due to dependencies on old versions of OpenSSL. We upgraded some applications to newer versions of Ruby, however some weren't worth the effort or risk (before Ruby 2, even a patch level change likely had breaking changes).

Although other systems were running Ubuntu, we decided to go with CentOS 6 for these systems - mainly because it was preferred by the current era of sysadmins. It also had packages for Ruby 1.8 or at least the correct dependencies (I can't remember exactly). And even better, although it was released in 2011, it is still supported until 2020.

If I was going to do the same today I'd just use Docker, but it was fairly new back then.

nul_byte2• 9 years ago

It's OT but how many people actually use RHEL? Or CentOS (more likely).

A majority of Telcos, Banking, Military, Stock Exchanges, Medical and large commercial enterprises

Why?

Because when a system starts having issues at 4am dealing with x amount of transactions per second and the shit is going to hit the fan, they want top class support on hand and not to be awaiting on someone replying on a irc channel / mailing list.

I personally am an Arch user on my home machine and work laptop, yet servers that run commercial workloads and have SLAs tied to them will always run RHEL for the reasons above.

erikb2• 9 years ago

In enterprise installations people only install software that they can receive support for. So the biggest server farms probably all run something like RHEL or SLES.

Of course for private usage it's a little ove rthe top. But RH also offers solutions there. There's an upstream OS with all the cool stuff: Fedora. And a downstream OS that is just as stable as RHEL but free: CentOS (not sure if I get the name right. There are so many CabcOS out there nowadays).

louwrentius• 9 years ago

Sounds to me that Btrfs is done for long-term. So now what?

cyphar12• 9 years ago

Facebook, Fujitsu and SUSE all contribute to btrfs and it is still being actively developed (it's the default filesystem for SUSE Linux Enterprise). Red Hat stopped contributing a while ago since most of their btrfs developers moved to Facebook.

"Done for the long-term" is a major overstatement IMO.

tjoff2• 9 years ago

Well, for those of us that have been waiting on btrfs for a frigging decade I get the feeling that it is never going to be ready.

If I wait another decade, will btrfs have matured? Will there be ANY half-modern filesystem for linux? I'm not convinced.

Currently bcachefs seems more appealing but well, long way to go there as well.

0xFFC• 9 years ago

Wow, I thought their long term plan was to switch to Btrfs. Now it is deprecated? What happened there?

lathiat12• 9 years ago

I think you might be confusing with SuSE?

Generally though sadly btrfs just isn't getting the man hours from any commercial sponsor it seems :(

cyphar12• 9 years ago

Facebook, Fujitsu and SUSE are all actively contributing to btrfs. The idea it isn't getting "the man hours from any commercial sponsor" is ludicrous.

justin662• 9 years ago

I am as happy as anyone that XFS is finally getting the position of honor it deserves on enterprise Linux (something like 15 years later than it should have, grumble grumble) but it doesn't really take the place of what btrfs was trying to do. Only ZFS is in a position to do that. I wonder if there are any plans for supporting the native port on RHEL.

mrweasel• 9 years ago

Redhat hasn't been on the best of terms with Oracle, so I suspect that they want to stay clear of ZFS. It does however leave Redhat without a more modern feature rich filesystem.

Perhaps Redhat could help to develop snapshots on XFS. It's not the only feature XFS is missing, but it's a start.

cyphar12• 9 years ago

Perhaps Redhat could help to develop snapshots on XFS. It's not the only feature XFS is missing, but it's a start.

Upstream has been working on adding more btrfs-like features to XFS, but I believe that RHEL encourages using devicemapper snapshots (which you then format with XFS).

boomboomsubban3• 9 years ago

Redhat hasn't been on the best of terms with Oracle, so I suspect that they want to stay clear of ZFS

The only connection Oracle has to ZFS on Linux is ownership of some patents that the license allows you to use, their reluctance is based on distribution issues between the GPL and CDDL.

lunchables4• 9 years ago

What about just using LVM for snapshots? Considering that there default partition schemes include LVM maybe that's what they bank on in most use cases?

jacquesm4• 9 years ago

I have one machine running XFS but if that one is representative then I won't be installing XFS anywhere else and would happily discourage others from using it. It is terribly slow when doing some fairly common operations when you have a large number of small files.

dijit• 9 years ago

You're joking, surely.

XFS has outperformed EXT4 in almost all "high" use-cases in my experience and testing: Large files (500GB~) or many small files (128k files * 2,400,000 or so). EXT4 under those loads is comically bad.

BTRFS is also terrible at this, only XFS and ZFS are good at handling it.

baldfat• 9 years ago

Here is a link to benchmarks for the file systems. To bash XFS which is about as solid as one can get I would love to know why you would say that? http://www.phoronix.com/scan.php?page=article&item=linux-44-...

feld3• 9 years ago

You have to tune the parameters at filesystem creation time if you care about small file performance. It was designed for large files.

__david__• 9 years ago

Anecdote time: Last month I had an XFS volume fail on me. It got some sort of internal inconsistency and refused to work (all fs calls returned errors until I unmounted). This is where I discovered that XFS still has extremely poor recovery tools.

xfs_repair will complain if there is a journal present and tell you to mount the fs to replay the journal. But mount would refuse, saying the fs was inconsistent. So the only option was to xfs_repair -L to just throw out the journal.

Then, xfs_repair sucked up something like 30GB or more of RAM, so I had to make a huge swapfile so that the kernel would OOM the repair.

Then, after roughly 20 to 30 hours of repair it would exit with an error. At that point it would actually mount, but hitting certain areas of the filesystem would trigger the inconsistency again and start the entire process over.

In the end I couldn't fix it and sadly had to reformat. I chose ext4 when I did—I've had lots of experience with ext3 and 4 and I've never had a filesystem that I couldn't at least make consistent again (even if it loses some data).

rob-olmos3• 9 years ago

Yikes. That's unnerving to read. Were you able to create a bug report? Was there any other related issue like an underlying storage controller messing up?

sparky_• 9 years ago

That's too bad. The subvolume [0] features were an interesting paradigm. Kind of let you have a virtual filesystem-within-a-filesystem.

[0] https://en.wikipedia.org/wiki/Btrfs#Subvolumes_and_snapshots

rleigh7• 9 years ago

It's certainly interesting, but if you look at the ZFS design they were inspired by, they got a lot wrong. Some points to consider:

With ZFS, you have a hierarchy of datasets. These inherit properties from their parents, and while the mountpoints can also mimic this hierarchy, the mountpoint property can be set independently. Btrfs couples the two concepts, forcing subvolumes to be in a specific place in the actual filesystem; zfs datasets in comparison are purely metadata and are for organisation and administration, not direct use in the filesystem hierarchy.

ZFS snapshots are read-only, and clones of these snapshots are datasets in the hierarchy. Btrfs snapshots are read-write by default, which in some ways defeats the point of a point-in-time snapshot. You can also make changes to a ZFS clone and later promote it to replace the original dataset. Likewise rollbacks. Btrfs makes no provision for doing either; you have to delete the original and then rename the snapshot, which isn't atomic. ZFS' metadata preserves all relations between datasets, snapshots and clones.

The ZFS way of doing things makes things safe and accessible for system administration. There's no way to confuse the origin of a snapshot because it's tied to a parent dataset. Likewise clones of snapshots, unless you deliberately choose to break the link. The Btrfs way looks superficially nicer, but in practice is much less flexible, and potentially more dangerous since you don't have the ability to audit what came from where and when. Btrfs snapshot performance is also abysmal. ZFS handles snapshots simply by recording the transaction ID, which makes them really lightweight (and it also provides "bookmarks" which are even lighter weight). ZFS keeps the referenced blocks in deadlists, and its performance is excellent (compare how fast snapshot deletion is between the two). ZFS also allows delegating permissions to perform snapshot, clone, rollback etc. to normal users; I'm unware of Btrfs allowing such delegation--some operations can be performed like snapshotting, but not deletion, while ZFS permits this all to be configured transparently.

tylerjd2• 9 years ago

Are you trying to say that BTRFS is supposed to compete feature-to-feature with ZFS? It's not. https://lwn.net/Articles/342892/

I had a unique opportunity to take a detailed look at the features missing from Linux, and felt that Btrfs was the best way to solve them.
From other points of view, they are wildly different: file system architecture, development model, maturity, license, and host operating system, among other things

-------------------------------

Btrfs snapshots are read-write by default, which in some ways defeats the point of a point-in-time snapshot.

Yes, and have the option of being read only for your temporal "in place" snapshots. But if I want to clone a container for instant use (as LXC or Docker does), then the RW snapshots make sense. Btrfs doesn't make a distinction between a Clone and Snapshot, they are one and the same with a flag.

but in practice is much less flexible

Tell me more how I can mix disks of differing size in RAID on ZFS

There's no way to confuse the origin of a snapshot because it's tied to a parent dataset

There's no confusing to the origin of my sanpshots. `btrfs subvolume list -q` shows the ancestral parent as well as the subvolume it's located in, example:

  ID 6442 gen 50527 top level 751 parent_uuid 0f4442f8-6363-6944-be8d-e2b45d809352 path .snapshots/321/snapshot

> some operations can be performed like snapshotting, but not deletion

See user_subvol_rm_allowed mount option, available since Kernel 3.0

It's like comparing a car and a truck, they both have four wheels, transport passengers and cargo, and have an engine. Just because a truck runs on diesel does not make the fact that the car running on gas "wrong". Due to its fundamentally different implementation, the way the filesystem works is also different.

Yes ZFS has many more features, has been in development longer, and probably more "production ready" than BTRFS. But ZFS is not GPL compatible. And BTRFS doesn't require it's own separate cache that is apart from the normal filesystem cache.

kodfodrasz• 9 years ago

I wonder if this is a response to Oracle's Java9/Jigsaw push, where RedHat was forced to give up its position.

bedros2• 9 years ago

I think this is a political move disguised as technical move

oracle pays the developers of btrfs [0]

redhat hates the guts of oracle, since oracle released oracle linux, which is a clone of redhat enterprise (based on centos)

so, redhat wants to cripple btrfs and hurt oracle.

However, btrfs is my favorite FS, been using it on my home computer and backup drives for at least 6 years, before it was included in the kernel, love the subvolumes, snapshots, and compression; never had issues with it .

[0] https://oss.oracle.com/~mason/

[Update] Chris mason no longer at Oracle since 2012

cleech• 9 years ago

Chris Mason left Oracle for Fusion-io back in 2012 and then from there to Facebook in 2013

skrause• 9 years ago

Chris Mason, the principal Btrfs author, left Oracle over 5 years ago:

https://en.wikipedia.org/wiki/Btrfs#History

"In June 2012, Chris Mason left Oracle for Fusion-io, which he left a year later with Josef Bacik to join Facebook; while at both companies, Mason continued his work on Btrfs."

sargun3• 9 years ago

Oracle still has a couple developers on their payroll that do BtrFS development:

* Liu Bo * Anand Jain

newscracker• 9 years ago

However, btrfs is my favorite FS, been using it on my home computer and backup drives for at least 6 years, before it was included in the kernel, love the subvolumes, snapshots, and compression; never had issues with it .

Slightly off topic. I chose btrfs as my main filesystem recently on a system running Ubuntu/Xubuntu. I have done some research on backing up (with the advantage of snapshots) but it looks like there aren't (m)any graphical tools (this gets a little more confusing with /@ and /@home subvolumes on the same partition being treated separately for snapshots, AFAIK).

Do you manage it all from the command line and/or do you have any suggestions for graphical tools to do "as-is clones of entire partitions" (and also incremental backups) to local external drives (not over the network)? Or if you could point to any great documentation or blog posts on this topic, that'd be helpful too (I have read some bits of the btrfs wiki and the btrfs parts in the Arch wiki).

Currently I'm doing a plain rsync using Grsync, and not really taking advantage of btrfs features like snapshots.

The main reason I'm looking at avoiding the command line is to make it easier for others around me to use it.

bedros2• 9 years ago

my line for snapshots

btrfs subvolume snapshot /source/drive/folder/ /source/drive/folder/.snapshots/snapshot-`date +"%Y-%m-%d-at-%I-%M%P"`

this will create a snapshot with date and time attached to the snapshot name

you can more info here

https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Managi...

make sure to remember to delete old snapshots, otherwise you'll run out of disk space and not know where it went

lathiat12• 9 years ago

performance during snapshot creation/deletes (presumably mostly the deletes) is one of the reasons I personally stopped using btrfs on my desktop. Now using ZFS root (with Ubuntu devel).

I had auto hourly snapshots and sometimes when it deleted one my entire system would hang for a few seconds and occasionally 10s of seconds.

Having said that I do suspect that might be partially related to also using ecryptfs on top, but still.

ge0rg• 9 years ago

I think this is a political move disguised as technical move

I think there are solid technical reasons to discourage Btrfs use, just to quote from the official wiki [0]:

The parity RAID code has multiple serious data-loss bugs in it. It should not be used for anything other than testing purposes.

Now I don't know if this issue has been addressed already, or which kernels are affected, but the fact that there is a prominent warning on the wiki speaks for itself.

Personally, I'm a happy btrfs user deploying a mixed-disk-size array without parity, with the hope to add redundancy some time in the future. Currently, btrfs is the only FS allowing to mix disks of any size and to run an optimal configuration on top of them [1].

[0] https://btrfs.wiki.kernel.org/index.php/RAID56

[1] http://carfax.org.uk/btrfs-usage/

cyphar12• 9 years ago

The particular bug that sparked that warning was fixed a while ago, but as a precaution against "btrfs ate my data" stories they've removed the ability to create btrfs-raid from the CLI tools (you can still use md RAID with btrfs but you lose most of the benefits of btrfs that way).

rleigh7• 9 years ago

Even when it gets resolved, it's far from being the only major problem with Btrfs. It's merely the current high profile one.

cmurf4• 9 years ago

Fixed in 4.12.

rodgerd5• 9 years ago

I think this is a political move disguised as technical move

You literally have no idea what you're talking about, and I doubt you've used btrfs seriously, or you wouldn't talk this shit. The fact it's been upvoted so heavily just shows what absolute technically-false nonsense will draw support at HN.

madez3• 9 years ago

There was no reason given for the deprecation of BTRFS.

cookiecaper3• 9 years ago

If anything, the political move would've been Oracle's sponsorship of btrfs in the first place. They want to push people to enterprise OS/storage systems, so they've told everyone "Oh yeah, btrfs is coming soon, it'll be great" ... and it isn't great. It sucks.

I've finally broken and installed ZOL (ZFS on Linux) after trying btrfs repeatedly over the last three years. ZFS is already a breath of fresh air and I've only been using it a couple of months. For whatever reason, btrfs came together as a messy hodge-podge, and it shows in bad performance for many use cases (e.g. "omg I forgot nodatacow"), buggy implementations, difficult user interfaces, kernel bugs, etc.

btrfs needs a reboot (I hear bcache? is trying). Meanwhile, everyone should stop getting hung up on the arcane licensing details and just use ZFS directly. It can't be distributed as part of the kernel, but that's why we have distributions, isn't it? They bundle all that crap together for us. There shouldn't even be the normal OSS infighting because this isn't a proprietary blob or something, it's just using a license that's GPL-incompatible.

The best thing Linus could do for the community at large would be to fork and start committing to ZOL, giving it a tacit endorsement.

pmontra3• 9 years ago

Offtopic, but if anybody from RedHat and especially Mozilla read this, go to that page with an Android phone, possibly with a small screen:

To RedHat: do something for that menu at the left. It stays in the way when scrolling and zooming. The X to close it is not immediately visible on a small screen. Expected behavior: the menu scrolls away with the page and doesn't stay fixed in the way of the reader.

To Mozilla: open the page with Opera and copy what you see there. Tldr: autofit the text in the screen width. Maybe Chrome does the same. Btw, reader mode doesn't kick in.

tyfon• 9 years ago

Worse, I browse without javascript and their page has no text.

It's utterly terribly designed to be an open source information page.

creshal• 9 years ago

Even as paying customer I wouldn't want to put up with that shit.

jwilk• 9 years ago

The text is there, just invisible. Disable CSS and it'll show up.

icebraining3• 9 years ago

Re: Mozilla, they are aware, and there are a couple of about:config flags to enable text reflow on page load and on tap-to-zoom, but they're disabled by default "due to a number of reasons (mainly performance)".

pmontra3• 9 years ago

I looked for them but maybe they don't have obvious names. I found that I set browser.ui.zoom.force-user-scalable to true, it defaults to false.

Do you remember their names? Thanks.

jhasse2• 9 years ago

Maybe Chrome does the same.

Nope

noisem4ker• 9 years ago

As I recall, the whole idea of text reflow in Firefox for Android has been abandoned because nobody was willing to fix broken code for a feature deemed no longer important for the modern web. (I'm sure following Google's decision to remove it from their browsers since KitKat also played a part, knowing modern Mozilla and their attitude.)

A real shame, especially for those coming from Presto-based Opera Mobile, which used to do reflow seamlessly and in a better layout than modern Blink-based Opera.

Fortunately, Firefox supports addons and you'll find at least a couple ones claiming to do the trick: https://addons.mozilla.org/en-US/android/addon/fit-text-to-w... https://addons.mozilla.org/en-US/android/addon/text-reflow/

headlands• 9 years ago

The default file system for the root partition of SUSE Enterprise Linux is Btrfs. So RHEL have stopped supporting it. It is the only filesystem that implements many of the features found in ZFS.

fundabulousrIII4• 9 years ago

It's also buggy as hell and though I've used SuSE for 17 years I almost stopped this year due to btrfs. Terrible decision from SuSE: but they aren't the company they used to be.

vbezhenar4• 9 years ago

I don't think this matters much. If you want btrfs, you'll likely be able to get kernel module from EPEL even if they'll stop shipping it. It's like deprecated and unsupported LXC which could be used just fine and which doesn't have usable replacements.

tylerjd2• 9 years ago

deprecated and unsupported LXC

Where did you hear this? This is news to me. https://linuxcontainers.org/ and https://github.com/lxc/lxc both seem active and supported by Canonical to me. The only deprecated project by them is listed to be CGManager.

vbezhenar4• 9 years ago

LXC is unsupported by RHEL. They chosen Docker as a primary container solution. LXC as project is fine, of course.

kureikain• 9 years ago

I have used Btrfs in production and I would say it's great. It's super easy to just add an extra EBS volume and attach to a Btrfs volume and now you have more disk space. Performance is good enough for me as well, I used it as storage for InfluxDB and Docker.

Luckily this is only Redhat, not Btrfs itself.

beedogs• 9 years ago

Unfortunately, RedHat tends to be the trendsetter in the Linux world. Once it's gone from RHEL, it'll be gone from most other RPM based distros soon enough. It's kind of a shame that one company has grown to dominate the Linux software ecosystem, often to the detriment of all involved.

gcp4• 9 years ago

Red Hat has a lot of core Linux contributors on their payroll. Their status is, AFAIK, not undeserved.

smarnach• 9 years ago

RPM-based distros make up maybe 10 % of the Linux installations these days, so we should not overstate RedHat's influence.

bonzini5• 9 years ago

On the contrary, RHEL ships a relatively small subset of packages, and is especially stingy on kernel features in order to make the distribution supportable.

Btrfs is definitely not gone from Fedora, for example.

(Disclaimer: I'm on the virtualization team at Red Hat).

Dylan168072• 9 years ago

How much did you deal with snapshots? Because I've had crippling performance problems stemming from them. Even something as simple as having 90 daily snapshots and deleting the oldest few can cause trouble, where the filesystem does not respond to any requests for multiple seconds. And that's on an SSD. I don't remember if it was deleting snapshots or running balance, but I've had Btrfs on a hard drive not respond to I/O requests for two minutes. A light-use server that had snapper running for a few months fragmented so badly that it regularly hitched up even after snapshots were paused. I had to migrate the entire filesystem.

I'm still using Btrfs on my backup system, but that's only because I like the dedup enough to overlook the brief hangs.

fundabulousrIII4• 9 years ago

Not only that but in conjunction with journald you can achieve amazing disk space leaks that cannot be repaired easily without losing data.

X-Istence5• 9 years ago

More importantly Red Hat has deprecated FCoE in RHEL, which is big news, because at a previous $JOB they went all in on FCoE because it was supposed to be the future.

B1FF_PSUVM• 9 years ago

https://en.wikipedia.org/wiki/Fibre_Channel_over_Ethernet

KaiserPro• 9 years ago

FCoE died a death <2013.

Expensive hardware, with little gain sadly. It was a nice idea, however at its very core is a fairly large problem: converged network adaptors are problematic.

Unless you have lots of bandwidth in said adaptor (ie 56gig inifiband) you are going to get contention between network and disk IO.

X-Istence5• 9 years ago

I agree, not only that, but at least with Cisco gear FCoE was a real pain in the behind to manage and configure. So much duplication in configuration and settings across a lot of gear, and it never was as smooth as they made it out to be.

jabl• 9 years ago

Ouch?

What's the replacement then? ISCSI, or going back to FC? Or is everything cloud something these days? :)

lunchables4• 9 years ago

Yeah iSCSI and NFS. Also affecting this is the huge growth in "Hyperconverged Infrastructure" (HCI).

X-Istence5• 9 years ago

iSCSI off-load that is available on a variety of network cards, along with network policies that are similar to FCoE (never drop a packet) allow you to get the same speed/reliability as FCoE for a lot less.

FCoE requires that the networking gear drops only one packet in 10 million or something like it, if you can make the same guarantees for iSCSI, it is for all intents and purposes the same thing. With iSCSI off-load it is even better.

iSCSI also runs across your existing network stack, and doesn't require purchasing special equipment and is better supporter across a variety of different vendors, thereby making it easier to find the gear with the features you need, rather than settling for something that supports FCoE.

StillBored• 9 years ago

NVME over fabrics.. I guess, the general idea being to run NVME over FC, RCoE, or infiniband.

Given the existing install-base of FC, I'm guessing as people upgrade to the 32/128Gbit adapters they will start to purchase disk's that can support FC-NVME as well. Which will bootstrap the market there.

Although it could go to infiniband as well, if people buy into the converged infiniband/ethernet adapter route.

To soon to tell, but a lot of it will be dependent on which technology does a better job avoiding the "forklift" upgrade problem that FCoE required.

lunchables4• 9 years ago

We've been big Cisco customers for years and ever since I saw FCoE I thought it was a disaster. All of the DCB extensions that had to go into ethernet to get it to work was such an ugly mess. It was just too complicated compared to alternatives, and iSCSI got a free ride on all that work (ethernet pause, flow control, etc) and was far simpler to implement. And of course with lots of 10Gb options with iSCSI offload, it was getting harder and harder to find any advantage LARGE ENOUGH in FCoE to justify it's cost and complexity.

X-Istence5• 9 years ago

Completely agreed, but it is something that Cisco was still pushing fairly heavily alongside their partners. Various different vendors bought into it, and it was deployed heavily in various different MSP's.

The complexity of FCoE is staggering, and the configuration required across all the different moving pieces to make it a success made things even more difficult!

cmurf4• 9 years ago

Unsurprising. Red Hat has not hired upstream Btrfs developers for years, where SUSE has hired bunches. Meanwhile Red Hat has upstream ext4, XFS and LVM developers.

If you're going to support a code base for ~10 years, you're going to need upstream people to support it. And realistically Red Hat's comfortable putting their eggs all in the device-mapper, LVM, and XFS basket.

But, there's more: https://github.com/stratis-storage/stratisd

Btrfs has no licensing issues, but after many years of work it still has significant technical issues that may never be resolved. page 4

Stratis version 3.0 Rough ZFS feature parity. New DM features needed. Page 22 https://stratis-storage.github.io/StratisSoftwareDesign.pdf

Both of those are unqualified statements, so fair or unfair my inclination is to take the project with a grain of salt.

gcp4• 9 years ago

As for the significant technical issues, one thing is the core decision to make it a CoW system, which has fundamental performance issues with many workloads that are exactly those used in the server space. You can disable CoW, but you lose many reasons to use btrfs in the first place if you do.

When I gave up on it there were also fundamental issues with metadata vs data balancing, not-really-working RAID support, and so on...

pgaddict6• 9 years ago

I find the suggestion that the technical issues are caused by the CoW design a bit strange.

Sure, making the filesystem CoW-based means there are some inherent costs, but it allows the filesystem to implement some interesting features (e.g. snapshots) in a more efficient way. For example if you want to do snapshots with ext4/xfs, you'll probably do that using LVM (which you can see as turning the stack into a CoW). In my experience the performance impact of creating a snapshot on ext4/LVM is about 50%, so you cut the performance in half. While on ZFS the impact is mostly negligible, due to the filesystem is designed as CoW in the first place.

And thanks to ZFS we know that it's possible to implement a CoW filesystem that provides extremely stable and balanced performance. I've done a number of database-related tests (which is the workload that I do care about) and it did ~70-80% TPS compared to ext4/xfs (without snapshots). And once you create a snapshot on ext4/xfs, the performance tanks, while ZFS works just like before, thanks to the CoW design.

Unfortunately, BTRFS so far hasn't reached this level of maturity and stable performance (at least not in the workloads that I personally care about). But that has nothing to do with the filesystem being CoW, except perhaps that CoW maybe makes the design more complicated.

cmurf4• 9 years ago

Unconvincing. LVM's snapshots are CoW whether thick or thinly provisioned. And while not yet merged in mainline, XFS devs are working on CoW as well which is used when modifying reflinked files (shared extents).

Btrfs behaves basically like that with 'nodatacow' today. It will overwrite extents if there's no reflink/snapshot. If there is, CoW happens for new writes and any subsequent modifications are overwrites until there's a reflink/snapshot in which case CoW happens.

The 'nodatacow' flag can be used as either a mount option, or selectively with an xattr per subvolume, directory, or file. And in all cases, metadata writes (the file system itself) are still CoW.

denji• 9 years ago

https://github.com/redox-os/tfs

buserror• 9 years ago

Well I used btrfs as a root filesystem for quite a while, until I realized it was pig slow for sync() -- I mean, it would take AGES to do and apt-get upgrade for example. I ended up having to do some tasks using 'eatmydata' [0] to make it all better, risking filesystem corruption in trade for speed. Also, at the time, there was no functioning fsck.

So I moved back safely to ext4 and never looked back!

[0] https://www.flamingspork.com/projects/libeatmydata/

buster5• 9 years ago

Over the recent years on every new laptop install i switched between filesystems, so i had ext3/4, btrfs and (currently) xfs on my system. I have to say, btrfs had the most glitches (a few years back), although it worked ok'ish (no data loss).

Nowadays, i must say that i very much prefer a stable filesystem with as little complicated logic as possible. I actually never use snapshots or subtrees! I never put another disk in my laptop (where would that go?!) so i don't need to do dynamic resizing (while online of course!). All this makes the filesystems a lot more complex then it has to be. I've also run into problems using ZFS on Solaris some years back which took ~2 weeks in dtracing what the hell is going on. Of course it was related to CoW.

My lessons learned: Check your requirements. Will you really need and use subtrees/snapshots/XYZ on your system? Will you really need to do online-resizing? If not, just use a stable, simple filesystem. There are perfect usecases for ZFS or btrfs. But not everyone needs the advanced features.

viraptor3• 9 years ago

Will you really need and use subtrees/snapshots/XYZ on your system?

It's a valid question, but not the best one. Almost nobody needs snapshots. But they make things easier. You most likely don't need a journaled fs in your laptop either (battery level notification should take care of the issues). But it does make life better.

"Need" is not the threshold I'm interested in. Most features, I'd like. One feature I think I do need most is scrubbing, which is still absent from most filesystems :(

eternauta3k• 9 years ago

It's ok to play with the OS in order to learn.

lorenzfx• 9 years ago

I'm using ZFS on my FreeBSD laptop. Snapshots not only make backups safer (by making sure the complete backup is taken at the same time, and by zfs sending and receiving them), the boot environments feature also make upgrading safer.

I also really like that I don't need to partition my disk, if it turns out that /tmp needs > 10% of the disk for whatever reason: no problem!

And as I like my data, I appreciate checksumming and copy-on-write.

I haven't noticed any bad slowdowns compared to ext4 on my Debian laptop I used before.

Maakuth• 9 years ago

It's a good practice to use LVM there between disk partitions and volumes. It has negligible performance implications but makes things very flexible when you need to resize volumes or add space. You also gain reliable snapshotting from device mapper, although that does have some performance effects.

pmlnr2• 9 years ago

My requirements: transparent compression. I'm left with: btrfs or, if possible, ZFS.

StreamBright• 9 years ago

We are using XFS for most of our production workloads, it turned out to be an excellent choice for most data heavy use cases. Brtfs was never an option, it is a bad idea to gamble with beta technology for data storage that a production system relies on. ext4 vs xfs is a much interesting argument, I haven't had time to follow up on this.

jerven• 9 years ago

We use XFS for sparql.uniprot.org (basically columnar database with semantic graph) there we recently retested it by accident. We use 2*4 TB consumer SSD ,ok rich consumer ;) With XFS we have 10-13% faster linear write of one big file (1.3TB) and about 20% more reads serviced per minute than EXT4. EXT4 was selected by accident instead of XFS for 1 of the 2 otherwise equal machines when upgrading these with the new 4TB SSDs instead of 1TB ones which they had before.

In general I feel that the more you go into "enterprise" storage levels, the more XFS pulls ahead from the EXT family. i.e. laptops and small servers are not where the difference lies.

joosters2• 9 years ago

apt-get upgrade syncs so often that 'eatmydata' gives a noticeable speedup pretty much everywhere (I got into the habit of using it for ext3/4)

I don't get why apt syncs so often - isn't the main point of log-structured file systems their ability to recover after a crash or powerloss? If so, why should you need to sync more than one every ten seconds or so?

dsr_2• 9 years ago

apt doesn't assume that you have a reliable filesystem. It assumes that you might crash at any moment, and it would be really important for you to have a consistent view of what packages are installed when you reboot.

lathiat12• 9 years ago

Though true I have literally never had my system get corrupt or inconsistent during failed dpkg/apt from power loss, hang, filesystem going ro, etc. It's very reliable.

I've had older rpm & yum/dnf failures multiple times leaving me in weird inconsistent states from crashes or power losses etc. not conclusive but anecdotal experience - It's also possible it's been improved.

Meanwhile you can disable the file syncing with the apt preference dpkg::unsafe-io (google will be required for the exact syntax and file in /etc/apt - fairly sure you can cmdline it also)

shmerl• 9 years ago

That's weird. What do they have against btrfs?

nul_byte2• 9 years ago

They don't see it as stable enough to move beyond tech preview.

rurban• 9 years ago

It is also extremely slow. And the known RAID problems didn't get better.

seesomesense• 9 years ago

Not surprised. I have been using BTRFS for years. Unfortunately, one gets the feeling that not too much development has been going into it.

horusthecat• 9 years ago

Having been using BTRFS at home for years, i get the exact opposite impression... I've had bugs (kernel panics) that just disappear on the next update. I'm happy with the RAID1 ease of use. But even there, like some have said, manual scrub/balance--which is still nice to have--indicates a design flaw.

If I was in ops I would not use btrfs on anything but lab experiments... if i can get hardware RAID, then I can blame the vendor and the equipment

alexdowad• 9 years ago

I tried btrfs and got bitten by bugs; not doing that again. Judging from this move by RH, it looks like I wasn't the only one.

lucb1e2• 9 years ago

What bugs?

I recently setup a software mirroring raid with btrfs and I'm loving features like checksumming. It makes me feel my data is quite safe and can't bit rot anymore. So far it is working fine.

rodgerd5• 9 years ago

Hi.

Did you notice that the official description of RAID-1 is "Mostly working"? Are you aware if one of your drive fails, you have one chance to re-mirror it, before the remaining drive can no longer be mounted read-write and you need to dump the filesystem and re-create from scratch?

mseidl• 9 years ago

I had tried xfs recently and I hated it. I have a thin laptop which would just lose power if I grabbed it wrong and very quickly my xfs partition got corrupted and I was unable to fix it. So, I had to re-install my os, and went back to ext4. Which even with the same power offs, never an issue since several months. This is largely anecdotal. But have had a coworker with the same issue too...

I also had issues with a vms using xfs too.

But, I do use xfs on ssd raids(in our servers, being used for testing) and never had an issue there.

rleigh7• 9 years ago

We lost an entire openstack host due to an xfs glitch last year. It needed rebuilding from scratch.

erikb2• 9 years ago

Why is there no explanation for this? It seems weird to deprecate something that most people haven't even started using yet.

snakeanus• 9 years ago

Still waiting for bcachefs. It's like btrfs except it does not suck. It also has proper encryption.

betaby• 9 years ago

How many developers involved in bcachefs full time? What I've heard it's a one man show.

pmlnr2• 9 years ago

Great... So can I please have something that can do transparent compression? That is my sole reason for using btrfs. (My personal stack is on ZFS with Debian and I'm never going to look back from that, even if I sometimes have speed issues due to SATA instead of SAS system underneath it; ZFS compression and snapshots are incredibly powerful.)

parasubvert• 9 years ago

Concourse (which is a CI/CD system that orchestrates Docker containers) recently switched from btrfs to overlay to fix performance and stability issues.

For those with morbid curiosity on the many stability issues with btrfs as a container file system, this is chronicled in Github: https://github.com/concourse/concourse/issues/1045

stargrazer2• 9 years ago

Considering the size of disks now-a-days, the chance of bit rot is high. And (I don't have the original source) on SSD, bit rots probability is higher still. So... ZFS and BTRF have meta-data as well as data checksumming. From what I've read, XFS may have metadata checksumming, but not on the data side of things.

I consider checksumming important. Do others? What is the solution? What other file systems offer that sort of capability?

Snapshotting is a second go-to function. Particularly when it is integrated into the LXC container creation process. (There was a comment elsewhere here which said LXC is on it's way out.... huh? what?)

legulere2• 9 years ago

on SSD, bit rots probability is higher still

Do you have a source for this? So far I believed that bit-rot rates are pretty similar.

stargrazer2• 9 years ago

A google search turns up a number of sources. Another concept which could use justification is whether SSDs bit rot more over long term than do spinning disks. I heard that somewhere as well.

ars• 9 years ago

Use mirror raid and have mdadm do a full disk compare/check every month (this is the default on Debian).

Additionally use smartmontools and configure it to do a short self test each night, and a long self test (i.e. full disk read) each week.

This will catch/flag errors early, which mdadm will then detect.

Wicher• 9 years ago

Disk A and disk B both contain file SomeFile.

On disk B this file has rotted.

When reading the file SomeFile into memory, the read will be distributed among the disks (for performance reasons) (and it will probby need to span a multiple of the stripe size).

Ok, file is read into memory, including the bitrotted part from disk B. Now we write the file blocks back - as one does.

Voila! Both disks now contain the bitrot. And mdadm will not complain - disk A and B are identical for the area of file SomeFile.

binaryphile• 9 years ago

"Never go to sea with two chronometers; take one or three."

- adage cited in the Mythical Man Month

gigatexal5• 9 years ago

Or just do raidz6 in ZFS and call it a day.

usefulcat• 9 years ago

Yes it can detect errors, but it can't continue to function correctly (read: return the correct data) because it doesn't know which copy of the differing data is damaged because it doesn't have checksums.

Moreover, if it doesn't always read both copies of the data (which it may well not, for performance reasons), then you have the possibility of silently propagating damaged data to all mirrors in the case that damaged data is returned to an application and the application then rewrites said data.

Compare that to a filesystem with checksums, which, in addition to being able to detect such a problem, could also continue to function completely correctly in the face of it.

tytso3• 9 years ago

There are many different ways that storage can be layered, and depending on your use case, you can put various advanced features (snapshots, checksum/data integrity, encryption, etc.) functionality in different places in the storage stack. You can put functionality the block device layer (e.g., lvm, dm-thin, dm-verity), you can put functionality into the file system, you can put functionality into the cluster filesystem (if you have such a thing), or you can put iti in the application level.

Depending on the requirements of your use case different choices will make more sense. It's important to remember that RHEL is used for enterprise customers, and what might be common in the enterprise world might not be common for yours, and vice versa. Certainly, if you are using a cluster file system, it makes no sense to do checksum protections at the disk file system level, because you will be using some kind of erasure coding (e.g., Reed Solomon error correcting codes) to protect against node failure. This will also take care of bit flips.

If you are using cloud VM's, or if you are using Docker / Kubernetes, then LXC won't make sense. It all depends on your technology choices, and so it's important to look at the big picture, not just at the individual file system's features.

rob-olmos3• 9 years ago

Given a stock (or additional packages?) RHEL 7.4 install on non-clustered storage, what would be the best combination to detect & correct bitrot at the filesystem and lower level?

mrob• 9 years ago

Linux 4.12 introduced dm-integrity, which adds integrity checking at the block device level, so it will work with any file system:

https://gitlab.com/cryptsetup/cryptsetup/wikis/DMIntegrity

zerd• 9 years ago

One good thing about ZFS integrity checking is that when it finds an error it can repair the bit rot from another disk if you have parity or mirroring. Can dm-integrity do that?

RX14• 9 years ago

I've done many bad things to BTRFS, used it on multiple drives of differing sises, used it on drives connected over the cheapest USB to SATA adapters I could find, used it on disks with consistent corruption for over a year, and it's handled it gracefully.

I've also been using btrfs as the backend to docker for a long time on my desktop PC and never noticed any problems. BTRFS has been rock solid for me. I don't doubt it is more unstable than other filesystems, however it seems i haven't been unlucky enough to experience any issues.

When using BTRFS, i've always stuck to the latest kernel releases, and run a scrub + balance every month. This is the advice I heard from people who used btrfs, and I wonder how many of the people who complain about data corruption do these steps. Perhaps their corruption bugs are solved in a newer kernel version. I've had multiple scrubs pick up data corruption, which other filesystems wouldn't have found.

The only time btrfs corrupted my data was when I used the ext4 to btrfs conversion tool, it created an unmountable FS and then I just migrated my data manually.

rleigh7• 9 years ago

You shouldn't have to do these steps.

Manual balancing is a workaround for a critical flaw in the implementation.

In my last major use of Btrfs, whole archive rebuilds of Debian, it would take less than 48 hours to completely unbalance a brand new Btrfs filesystem. ~25k snapshots continuously created and deleted over the period in 20 parallel jobs absolutely toasted the filesystem, even though it was 1% utilised for the most part, 10% at peak usage.

The point I want to make is that a Btrfs filesystem can become unbalanced at some indeterminate point in the future, which makes it impossible to rely on if you want to guarantee continued service.

I've also suffered from a number of dataloss incidents which likely are fixed now, but despite lots of bugfixing, there are still major flaws to address.

cookiecaper3• 9 years ago

btrfs requires far too much maintenance to be a good general-purpose FS, and it absolutely crumbles under certain not-too-uncommon use cases (specifically, databases and VMs). I really wanted to like btrfs and used it extensively over the last couple of years, but I've finally given up and moved on to ZFS, and I'm grateful that I did. Give it a try.

jorrizza• 9 years ago

Strange decision, as systemd-nspawn[1] specifically mentions and supports btrfs as a CoW filesystem for its containers. And as far as I understand, systemd is primarily developed by Red Hat employees. So either they'll add support for CoW alternatives, or they'll remove btrfs support from systemd-nspawn all together.

[1] https://www.freedesktop.org/software/systemd/man/systemd-nsp...

amorphid• 9 years ago

There's nothing which prevents you from creating an empty block device file, putting btrfs on that, and mounting it.

Torgo• 9 years ago

nspawn uses btrfs snapshotting natively for its templating and no other FS (ZFS was explicitly rejected by Mr. Poettering because "it's not in the kernel") so yeah either they are going to have to do something about this or there will be a significant step down in functionality for nspawn. I can't see how this isn't really bad news for nspawn.

I guess one option is to pull btrfs tree into Systemd :-)

josefbacik4• 9 years ago

People are making a bigger deal of this than it is. Since I left Red Hat in 2012 there hasn't been another engineer to pick up the work, and it is _a lot_ of work.

For RHEL you are stuck on one kernel for an entire release. Every fix has to be backported from upstream, and the further from upstream you get the harder it is to do that work.

Btrfs has to be rebased _every_ release. If moves too fast and there is so much work being done that you can't just cherry pick individual fixes. This makes it a huge pain in the ass.

Then you have RHEL's "if we ship it we support it" mantra. Every release you have something that is more Frankenstein-y than it was before, and you run more of a risk of shit going horribly wrong. That's a huge liability for an engineering team that has 0 upstream btrfs contributors.

The entire local file system group are xfs developers. Nobody has done serious btrfs work at Red Hat since I left (with a slight exception with Zach Brown for a little while.)

Suse uses it as their default and has a lot of inhouse expertise. We use it in a variety of ways inside Facebook. It's getting faster and more stable, admittedly slower than I'd like, but we are getting there. This announcement from Red Hat is purely a reflection of Red Hat's engineering expertise and the way they ship kernels, and not an indictment of Btrfs itself.

appleflaxen• 9 years ago

thanks for this context; i read this thread previously and had no idea of the "why" behind the news item. great comment to understand better.

nailer3• 9 years ago

Thanks. Any indication why RH didn't hire btrfs devs? It looks like a decision was made that it wasn't strategic (obviously xfs on Linux has a much longer history).

josefbacik4• 9 years ago

They brought on Zach right before I left specifically to help with the effort, but he left as well. I can't really speak to Red Hat's overall strategic decisions, but really they have a large local file system team, and a lot of them are xfs developers. You aren't going to convince Dave Chinner he should go work on Btrfs instead of XFS. Unless there's somebody internally that actually wants to work on Btrfs the work simply isn't going to get done. All of the other Btrfs developers work for other companies, and none of them are interested in working for Red Hat.

feld3• 9 years ago

The problem is that Redhat and others are refusing to challenge the norm and break away from the "freeze the release; backport fixes" mantra.

Stop backporting fixes. You're forking the codebase.

Ship exactly what upstream provides.

Teach upstream projects how to do better release engineering if they're abandoning major releases to early or breaking API/ABI in a minor release.

Stop backporting fixes. You're forking the codebase.

edit: also stop incorrectly backporting security fixes and creating new CVEs. Seriously. Stop it.

admax88q• 9 years ago

I think you're underestimating the stability that such practices provide for enterprise. This is what people pay Redhat for.

Not all upstreams are interested in doing release engineering. There are non zero costs to doing it. It can eat up time that can be spent on bug fixes and features, or even make it too costly to change direction if a certain approach to implementation is proving more difficult than it should be.

Look at the Linux kernel. The only reason there is a stable kernel series is because Greg K-H decided it was important enough. He was unable to convince any other developers to go along with it, and eventually the decision was "if you want to support it, then you can do it."

Do you consider the stable kernel series a fork of the codebase? Should everyone be running the newest kernel every release despite the plenty of regressions that appear?

Kernel developers are not interested in making every change in such a slow and controlled manner as to avoid any regressions. And it works for them. They get a lot of stuff done, and come back and fix the regressions later.

ajross• 9 years ago

If upstream releases were doing better release engineering in the way you mean then there would be no money to be made shipping RHEL as a product.

thehardsphere2• 9 years ago

Taking this to its most ludicrous extreme, everyone should use Arch, and anyone who can't should... what? Not use Linux?

SwellJoe• 9 years ago

I don't get why this would be a positive thing for Red Hat's customers (or Red Hat, since stability/predictability is what Red Hat customers are paying for). There is a Red Hat-maintained Linux that is very close to upstream (Fedora). But the people who pay for RHEL don't want upstream and surprises, they want predictable for seven years and they're willing to pay a lot of money for that. Why would that be a negative for you or me? RHEL isn't breaking upstream with this practice, even if they are making mistakes in their own backports.

"edit: also stop incorrectly backporting security fixes and creating new CVEs. Seriously. Stop it."

Can you give some examples of cases where Red Hat introduced bugs in their backported patches? I follow RHEL CVEs relatively closely (because some of my packages are derived from their packages), and I can't think of an example of that happening. Debian has done so, but very rarely, that I can recall. (And, Ubuntu, too, since they just copy Debian for huge swaths of the OS.)

tytso3• 9 years ago

There are real tradeoffs between development velocity, stability, scope (wide/narrow applicability), and headcount.

If you don't care that much about development velocity, it's really easy to make something that is super stable.

If you only care about making things work on a very narrow use cases (to support the back end of a particular company's web servers, or just to support a single embedded device), life also gets much easier.

If you want to "move fast and break things", that's also viable.

Finally, if you have unlimited amounts of head count, life also becomes simpler.

Different parts of the Linux ecosystem have different weights on all of these issues. Some environments care about stability, but they really don't care about advanced features, at least if stability/security might be threatened. Others are interested in adding new features into the kernel because that's how they add differentiators against their competitors. Still others care about making a kernel that only works on a particular ARM SOC, and to hell if the kernel even builds for any other architecture. And Red Hat does not have infinite amounts of cash, so they have to prioritize what they support.

So a statement such as "Teach upstream projects how to do better release engineering", is positively Trumpian in its naivete. Who do you think is going to staff all of this release engineering effort? Who is going to pay for it? Upstream projects consists of some number of hobbists, and some number of engineers from companies that have their own agendas. Some of those engineers might only care about making things better for Qualcomm SOC's, and to hell with everyone else. Others might primarily interested in how Linux works on IBM Mainframes. If there are no tradeoffs, then people might mind work that doesn't hurt their interests, but helps someone else. They might even contribute a bit to helping others, in the hopes that they will help their use case. That's the whole basis of the open source methology.

But at the same time you can't assume that someone will spend vast amounts of release engineering effort if it doesn't benefit them or their company. Things just don't work that way. And an API/ABI that must be stable might get in the way of adding some new feature which is critically important to some startup which is funding another kernel engineer.

There is a reason why the Linux ecosystem is the way it is. Saying "stop it" is about as intelligent as saying that someone who is working two 25 hour part-time jobs should be given a "choice" about her healthcare plan, when none of the "choices" are affordable.

cyphar12• 9 years ago

I get the feeling you've never had to provide support for a distribution before. There are many guarantees that Red Hat or SUSE provide that are not provided by upstream projects. Freezing the release is the only sane way of doing it, and backporting fixes is necessary. There are exceptions to this, such as stable kernels (which was started by GregKH out of frustration of the backporting problem while at SUSE).

Upstreams don't have the resources to do proper release engineering, they're busy working on new features. The fact that SUSE and Red Hat spawned from a requirement for release engineering that upstreams were not able to provide should show that it takes a lot more work than you might think.

Also, can we please all agree as a community that writing patches and forking of codebases is literally the whole point of free software? If nobody should ever fork a codebase then why do we even have freedom #1 and #2? The trend of free software projects to have an anti-backport stance is getting ridiculous. If you don't want us to backport stuff, stop forcing us to do your release engineering for you.

krylon2• 9 years ago

the "freeze the release; backport fixes" mantra.

For many customers of Red Hat, that mantra is the very reason they use RHEL in the first place.

noselasd2• 9 years ago

I for one am very grateful Red Hat does not do that. We have a kernel driver for custom hardware, there's around 80 of these devices in the world, split roughly 50/50 between Windows and Red Hat users. While these devices are not cheap, we could not recoup the cost of maintaining it if we had to track the upstream kernel all the time - we tried, and could not justify the cost.

The number of times the APIs changes from under your feet is astounding - even with just keeping up with Red Hat, we spend around 4-6 times the engineering time on the driver compared to what we do with the Windows version of the driver, tracking upstream gave us almost an order of magnitude more work (And keep in mind that /only/ supporting the most recent upstream kernel is rarely an option - several versions need to be supported concurrently)

gkya• 9 years ago

You cant teach anything to anybody. An unrelated proof: I'm having to write a bot with phantomjs to scrape my uni's announcements page and turn it into an RSS feed so that I won't have to periodically control it; because they decided wordpress wouldnt cut it and they needed some blumming angular.js stuff, breaking all the urls and removing any sort of rss feeds on the way. And all it is is a blog, basically, nothing more. Mailed them telling that I used to use that stuff, no replies in weeks. At least I'm learning phantomjs which seems to be very useful of a tool.

chasil9• 9 years ago

We use both RHEL and Oracle Linux as a peer to VAX VMS and Unisys(Univac) OS2200 (a COBOL mainframe).

From the perspective of legacy systems, Red Hat's approach is more comfortable.

user59944612• 9 years ago

>> This announcement from Red Hat is purely a reflection of Red Hat's engineering expertise and the way they ship kernels, and not an indictment of Btrfs itself.

It's a clear indicator that RedHat doesn't want or can't support btrfs.

Which is a reflection of btrfs AND RedHat: the effort required to maintain it, the lack of usage in RHEL paying customers, the immaturity/fast development of the filesystem.

pgaddict6• 9 years ago

I think a natural follow-up question is "Why Red Hat does not have engineers to support btrfs?" That is, if the lack of engineers is a symptom, what is the cause?

I'm pretty sure, had RH wanted they could either hire or assign engineers to maintain the btrfs code, take care of patches from upstream, etc. So why didn't that happen? I wonder what is your opinion on that.

I see a bunch of possibilities (not necessarily independent ones):

1) Politics. Perhaps RH wants to kill btrfs for some reason?

I see this as rather unlikely, as RH does not have a competing solution (unlike in the Jigsaw controversy, where they have incentives to kill it in favor of the JBoss module system).

2) Inability to hire enough engineers familiar with btrfs, or assign existing engineers.

Perhaps the number of engineers would be too high, increasing costs. Especially if not only to maintain the RHEL kernels, but to contribute to btrfs and move it forward.

Or maybe there's a pushback from the current filesystems team, where most people are xfs developers?

3) Incompatible development models.

If each release requires a rebase, perhaps supporting btrfs would require too much work / too many engineers, increasing costs? I wonder what Suse and others are doing differently, except for having in-house btrfs developers.

4) Lack of trust btrfs will get mature enough for RHEL soon.

It may work for certain deployments, but for RHEL customers that may not be sufficient. That probably requires a filesystem performing well for a wider range of workloads.

5) Lack of interest from paying RHEL customers.

Many of our customers have RHEL systems (or CentOS / Scientific Linux), and I don't remember a single one of them using btrfs or planning to do so. We only deal with database servers, which is a very narrow segment of the market, and fairly conservative one when it comes to filesystems.

But overall, if customers are not interested in a feature, it's merely a pragmatic business decision not to spend money on it.

6) Better alternatives available.

I'm not aware of one, although "ZFS on Linux" is getting much better.

So I tend to see this as a pragmatic business decision, based on customer interest in btrfs on RHEL vs. costs of supporting it.

chasil9• 9 years ago

Oracle has essential control of both "nextgen" filesystems that should be used in Linux - as Sun, they developed and licensed ZFS, and they are the chief contributors of BtrFS. Their refusal to release ZFS under a license that is compatible with the GPL is keeping it out of Red Hat's distribution.

This move by Red Hat must be seen as a provocation of Oracle, to force either greater cooperation and compliance in producing a stable BtrFS for RHEL, or the release of ZFS under a compatible license. Red Hat has put an end to BtrFS for now, and Oracle will have to go to greater lengths to use it in their clone. Customers also will not want it if it does not run equally well between RHEL and Oracle Linux.

It is obvious that Oracle will have to assume higher costs and support if they want BtrFS in RHEL. Red Hat is certainly justified in bringing Oracle to heel.

Oracle recently committed preliminary dedup support for XFS, so they must be intimately aware of the technical and legal issues behind Red Hat's move.

https://blogs.oracle.com/linuxkernel/upcoming-xfs-work-in-li...

josefbacik4• 9 years ago

All this talk about Oracle is just plain stupid. Oracle doesn't control anything, the community does. One core developer still works on Btrfs from Oracle, the vast majority of the contributions come from outside Oracle.

Now as to

"Why Red Hat does not have engineers to support btrfs?"

You have to understand how most kernel teams work across all companies. Kernel engineers work on what they want to work on, and companies hire the people working on the thing the company cares about to make sure they get their changes in.

This means that the engineers have 95% of the power. Sure you can tell your kernel developer to go work on something else, but if they don't want to do that they'll just go to a different company that will let them work on what they care about.

This gives Red Hat 2 options. One is they hire existing Btrfs developers to come help do the work. That's unlikely to happen unless they get one of the new contributors, as all of the seasoned developers are not likely to move. The second is to develop the talent in-house. But again we're back at that "it's hard to tell kernel engineers what to do" problem. If nobody wants to work on it then there's not going to be anybody that will do it.

And then there's the fact that Red Hat really does rely on the community to do the bulk of the heavy lifting for a lot of areas. BPF is a great example of this, cgroups is another good example.

Btrfs isn't ready for Red Hat's customer base, nobody who works on Btrfs will deny that fact. Does it make sense for Red Hat to pay a bunch of people to make things go faster when the community is doing the work at no cost to Red Hat?

ericfrederich• 9 years ago

I ran Fedora some years ago when btrfs support was added. I thought the idea to be able to roll back after a yum update messes things up was cool... in practice, it's a lot more complicated than hitting some "rollback" button in the system update gui.

snvzz• 9 years ago

On a non-totally unrelated note, Matt Dillon recently updated the HAMMER2 design document.

https://www.dragonflybsd.org/hammer/

kzrdude• 9 years ago

Any notable changes, or what do you take note of?

mhd• 9 years ago

Is JFS still a thing in the Linux world? I remember using it for some partitions not because it was particularly high performant, but because it was supposed to have very few worst case scenarios…

sargun3• 9 years ago

This is disheartening to hear. We recently (<3 months ago), introduced Btrfs into part of our fleet. This probably made us one of the larger Btrfs users. If Btrfs falls out of general favour, I'm afraid it may impede our ability to keep using it.

Given our use-case, multi-tenant containers, there weren't many choices which had sane snapshotting support, as well as quotas and some level of subtrees. ZFS on Linux has its own share of issues. I won't say that Btrfs was without issues -- there is still a lot of performance work that needs to be done, especially in a multi-tenant workload, but it looked like there were solutions available.

XFS is an excellent filesystem, and it may work well for our usecase in the near future. It's exciting to see new XFS features landing, like reflinks, and collapsing ranges. Hopefully, folks like Redhat continue trying to bring XFS to the future.

beagle3• 9 years ago

Btrfs brings IMO too little to the table, considering that ZoL is quite mature; It always seemed to me like it's the "we can't have ZFS so..." solution, but unlike Gnome (being the "can't have KDE" solution), it does not yet deliver, after a long long time.

The next fs to make a difference, post ZFS, is likely HAMMER2[0] - it's supposedly already stable for single node use (the ZFS / XFS / ext4 use case), and is advancing towards the multinode-at-the-underlying-fs-level, a first.

[0] https://gitweb.dragonflybsd.org/dragonfly.git/blob_plain/HEA...

⁠Btrfs has been deprecated in RHEL

Related Stories