ZFS is mysteriously eating my CPU

https://www.brendangregg.com/blog/2021-09-06/zfs-is-mysteriously-eating-my-cpu.html

by mfrw • 5 years ago

332 87 5 years ago

brendangregg.com

waynesonfire• 5 years ago

Nice job!

jeffbee• 5 years ago

Memory reclaim is where all the kernel bugs are.

amelius• 5 years ago

Is this an argument for using Rust?

marcan_422• 5 years ago

No. Memory reclaim != memory leaks != memory safety. This is about filesystem caches.

mhh__• 5 years ago

No amount of avoiding memory safety bugs will save you from writing fundamentally incorrect code. Rust stops you making dumb mistakes, and (as a high-level language, this part isn't Unique to rust in any way) lets you create abstractions to stop other people making less dumb mistakes, but will do absolutely nothing if your model of the process is wrong.

The only way to really be sure of that is extensive testing at multiple levels, and ideally some form of verification like a TLA+ model.

b112• 5 years ago

For the love of 3$&$>=*=$, not rust again!

marcan_422• 5 years ago

Reminds me of how, after years of massive systemwide latency spikes, I finally discovered that XFS was to blame. It was blocking on I/O writes in the reclaim path... so even random processes that wanted to allocate some RAM but don't do any disk I/O ended up blocked. This was on a machine with tons of free RAM (the reclaim was for clean cache).

ComputerGuru2• 5 years ago

Actual issue here: https://github.com/openzfs/zfs/issues/6531

It seems the whole thing took place in 2017 and was fixed then (first by not calling the reclaim if used ARC memory was zero, then the root cause also fixed by using a PRNG instead of the default CSPRNG in 0.7).

seg_lol• 5 years ago

Ahhh, sweet sweet reuse [1]. If code is poetry, most code is noetry, software reuse should be renamed to software refuse. Refuse it straight to the /bin

No code.

https://ocw.mit.edu/courses/aeronautics-and-astronautics/16-...

mlk• 5 years ago

stale bot closed this on 23 Nov 2020

as is tradition

alberth• 5 years ago

Where do you see the actual issue was fixed in that ticket?

ComputerGuru2• 5 years ago

You’re right, the hack/workaround to not reclaim ARC if it’s below a size was written but not merged [0]. But the PRNG thing (actual CPU usage bug) is real and in 0.7+ so far as I can tell. Ideally both would have been committed.

[0] https://github.com/openzfs/zfs/pull/6544

yjftsjthsd-h2• 5 years ago

Well that was a wild ride - I personally lost it at "We aren't using ZFS." :) I'm surprised that the kernel module was even loaded, though; if they weren't using it, and the only team using ZFS was a minor experimental group, why was it even installed, let alone loaded, let alone burning CPU on a cryptographically randomized noop?

pizza234• 5 years ago

I'm surprised that the kernel module was even loaded, though

On Ubuntu, some kernel packages install also the zfs modules package. I think (but not 100% sure) that if they're on that O/S, with those packages installed, then the zfs module will be loaded on startup.

chasil• 5 years ago

Netflix leans towards FreeBSD.

Is ZFS removable there without recompiling?

https://papers.freebsd.org/2019/fosdem/looney-netflix_and_fr...

fencepost• 5 years ago

From the top of the article (added?) "I summarized this case study at Kernel Recipes in 2017."

The bottom of the article notes that his book Systems Performance 2nd Edition is 45% off until 9/13.

Edit: looks like the comments about "didn't I see this a few years ago?" have been deleted.

brendangregg3• 5 years ago

Not added, but maybe I'll add something extra to make it more clear. I have a backlog of interesting case studies I'm trying to find time to share.

bostonsre2• 5 years ago

Not sure if you're comfortable sharing it, but if so, how many copies have you been able to sell?

Also, any idea how something like safari online impacts those numbers?

Thanks again for the book, it's an amazing read that I recommend to anyone that is interested in the area.

nicolas_t• 5 years ago

Where is it 45% off? I somehow can:t find that?

boomboomsubban2• 5 years ago

informit, the link is at the bottom of the page.

elktea• 5 years ago

Love the look of all the tools netflix have at their disposal

H8crilA• 5 years ago

A quick to use remote profiler does wonders to an org. By fast to use I mean it takes me 10 seconds to kick off a remote profiling scrape. Someone entrepreneurial please make a purchase-able service out of that, I'll be feeling lost once I quit my corp (not Netflix). Stuff is 100x more useful than remote debugging which ultimately almost nobody uses in practice.

tdullien• 5 years ago

Shameless plug: We just launched Prodfiler last week -- it's continuous in-production profiling based on eBPF, handles C/C++/Go/Rust/JVM/Python/Perl/Ruby (with NodeJS and .net in the pipeline), and it does not require symbols on the machine, nor recompiles with framepointer, nor special JVM flags.

https://prodfiler.com

:-)

bostonsre2• 5 years ago

Datadog has continuous profiling that is pretty useful. I'm pretty sure they follow Brendan religiously.. I hope they release something like flamescope next.

Cthulhu_• 5 years ago

And that he described "Normally I'd SSH into a machine", but you shouldn't even be able to do something like that if you've got infrastructure-as-code; the machines are ephemeral, they probably don't have any of the tools that you need to do an analysis, they shouldn't even have SSH or open ports unless there's some tool interacting with it, but in practice, they should be disposable on a whim if there is an update of sorts.

Had a colleague who asked for SSH access to production machines to debug an issue. Ops team asked what he wanted to do, guy just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration - instead of give him access, because they had a mandate to ensure five or seven nines uptime. Can't risk it.

I had a lot more respect for the ops people there than I had for my fellow SWE's.

raverbashing2• 5 years ago

I disagree.

There are always bugs that will happen only on the prod machines. Sure, they are rare, but they exist.

just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration

Well, that's not risk free neither. That needs a code change and you risk exposing secrets, or logging more than you should. (Though I agree with the general idea of you shouldn't be SSHing if you can do it another way)

Dylan16807• 5 years ago

Ops team told him how to do his job - log the configuration - instead of give him access

"do your job" is a really rude way to refer to debugging by logging instead of debugging interactively.

foldr• 5 years ago

Side node: Why do we have to talk about these things in such highly charged terms in our profession? It sounds like all that happened here is that one person made a reasonable request, and another person gave a reasonable explanation of why the request couldn't be granted.

closeparen• 5 years ago

You had a mandate for 5-7 nines uptime and you couldn't tolerate a problem on a single node?

xorcist• 5 years ago

log the configuration

That's a change, inherently much more risky than the "cat /.../env" the colleague wanted to do.

Also the change might well cause the problem to go away, and now you know nothing instead.

The principle is good, but it sounds like it has taken on a life on its own. The ops guy could also have executed the cat command on the spot with the right privileges. Sure, it's gatekeeping, but so is the four eyes principle, and a little of that gatekeeping can be necessary to keep those nines rolling.

throwaway81523• 5 years ago

If messing up a single instance is going to take your site down, you are nowhere near seven 9's.

teitoklien• 5 years ago

Sounds weird , logging env vars would be even more risk prone to leaking credentials, than just letting an employee ssh into the machine and see the env vars.

Also like others noted here , If one guy sshing into one machine can screw your seven nine uptimes.

Then you never had a seven nine uptime (you’re company is probably just gambling on that uptime metric until some component fails)

StreamBright2• 5 years ago

This is the sort of fanboyism that I usually avoid when working. It is perfectly reasonable to SSH into a node for debugging. In fact I do it very often in a heavily automated infrastructure as code environment.

brendangregg3• 5 years ago

Most issues we hit are solved by the monitoring GUIs (Atlas, FlameCommander, PerfDash, S3 log collection, etc.), but some aren't, and there's a ton more information available from the CLI tools. Imagine losing your five nines because no one can SSH on and try a few ad hoc tools.

But yes, CLI can be dangerous, even for observability. E.g., people running strace(1) on production apps and causing outages due to the strace overhead (I wrote a prior post about that). You need to understand the risks and overheads of all tools. It's why I have a "pull no punches" policy when writing eBPF tool man pages [0]: If the overhead can be bad, it should say so clearly.

[0] https://github.com/iovisor/bcc/blob/master/CONTRIBUTING-SCRI...

vidarh• 5 years ago

I know at least one place where they do allow ssh as a last resort, but the VMs that gets ssh'd to get flagged to be recycled automatically.

So you can log in and debug, but once you're done, the VM is replaced with a clean one.

raverbashing2• 5 years ago

I usually find people running ZFS to run into way too many more problems than what other people run using Ext4. (Edit: I'm surprised to see how people run into issues with Btrfs)

It might be good for some specific purposes but I'd reckon the majority of people using it do not fall into those. Using it "just because" is probably more trouble than worth.

ZFS really wasn't in use, ever! But at the same time, it was eating over 30% of CPU capacity! Whaaat??

Interesting bug, sounds like the best option is to not even have the module loaded then. (Not blaming the developers here, necessarily, but yeah, it's a bug)

pseudalopex• 5 years ago

Btrfs gave me the most trouble by far. And most people say the same in my experience. Replicating half of ZFS's features with ext4 is a whole lot of trouble too.

usefulcat• 5 years ago

We have a machine that has a large ZFS storage pool, and one smaller btrfs volume on a pair of SSDs. If I delete a large directory hierarchy on the btrfs volume and then run sync, it locks up the entire machine for 5-10 minutes. WTF. Nothing like that ever happens with ZFS or ext4.

azalemeth• 5 years ago

I've had a lot to learn with zfs, but it's never just flat out lost my data, unlike btrfs (in about 2017)

teekert3• 5 years ago

Raid5?

daddylongstroke• 5 years ago

Consider yourself lucky. There are data-corruption bugs all over the now-diverged ZFS codebase(s). Check the Solaris 11 SRU notes from the last 24 months for many examples.

isatty2• 5 years ago

Usually in my experience whenever someone has a problem with zfs it’s solvable in one way or another and the devs are very responsive on irc and GitHub.

On the other hand a lot of problems caused by btrfs is unrecoverable and a mess. Ext4 is stable but lacks the feature set that makes customers go to zfs instead (and that is fine, software should do one thing well and right).

sirius873• 5 years ago

For a homelab use-case, I backed out of creating a ZFS mirrored storage pool for network storage. It wasn't that the steps were too complicated or that ZFS let me down as such.

Just that RAID in general and ZFS put my files in a blackbox, and that I didn't really need a significant boost in network reads. I settled for plain old ext4 and periodic rsync mirror files to another disk (I don't mind some interim data loss + writes are fast). Use SFTP or sshfs for accessing the drive.

isatty2• 5 years ago

Any RAID solution that does block/bit level striping will be a “blackbox” as you describe it. That’s not a bad thing (nor is your approach) but pros and cons.

sennight7• 5 years ago

I settled for plain old ext4 and periodic rsync mirror files to another disk

This isn't a mirror, not in the sense that most would understand. No performance boost, no protection from bit rot, unnecessarily wasteful in every potential metric. If you wanted to intentionally design a system to propagate errors and render backups useless - you would start with this kind of setup. It makes sense to avoid all the RAID related problems associated with hardware solutions... but I'm drawing a blank on rational reasons to do what you've described. Super weird boot manager + physical space constraints?

ZFS is mysteriously eating my CPU

Related Stories