OOM relation to vm.swappiness=0 in new kernel

http://www.mysqlperformanceblog.com/2014/04/28/oom-relation-vm-swappiness0-new-kernel/

by tinplate • 12 years ago

53 25 12 years ago

Important news, thank you. I don't expect to see things that directly impact my servers here, but it's a welcome surprise.

peterwwillis3• 12 years ago

IMHO, RHEL kernels should never be considered stable or production-ready. Sadly, support contracts sometimes require their use.

otterley5• 12 years ago

On what basis is your opinion founded? Most of the top websites in the world use RHEL or one of its brethren (aside from Google).

peterwwillis3• 12 years ago

Yeah, and they use it mainly due to support contracts and industry certifications. Doesn't make it a good idea.

They used to release RHEL kernels with hundreds of patches that could be applied in line, or removed/fixed individually. Now they keep one tree that they manage privately and release only the fully-patched source, without any documentation about what's changed or why. Your only hope of figuring out what has been changed or getting help in fixing it is to pay for support, or browse the commits at Oracle's RedPatch (https://oss.oracle.com/git/?p=redpatch.git). Basically, a RHEL kernel is an obscure product designed to make them money.

The primary reason RHEL kernels aren't stable is the bug this thread is about. There is no valid reason to backport a change to swappiness behavior into a 'stable' kernel. The whole idea behind 'stable' is it works, so don't change it. But RHEL has other ideas.

RHEL kernels are more likely to contain security holes than a vanilla kernel of the same version, partially due to the backporting of new or changing functionality. They also don't get as much testing and scrutiny as the vanilla tree. Besides internal testing, changes that aren't included in stock linux kernels are introduced into Fedora for their users to beta-test. Over time we've seen plenty of security holes make it into otherwise 'stable' RHEL kernels and tools by RHEL backports. It's usually a feature they're importing that brings along with it additional attack surface, or introduces a hole in pre-existing functionality.

It's also a big pain in the ass to get a RHEL kernel to support something a normal kernel would support, due to the major changes. You also have to modify whatever patches you want to apply on top of RHEL since most people create patches targeting the vanilla tree and not RHEL. In general if you have a problem with the RHEL kernel you have a lot better chance of success by throwing it out and downloading a vanilla kernel.

At this point RHEL's kernel is a nightmare hybrid of changes going into a tree that nobody else uses because they've moved to 3.x trees for new enterprise releases. Between vanilla 2.6.32 and RHEL's 2.6.32-431 there are 9144 files changed, 3,563,927 insertions, 699,721 deletions. The diff is 45MB compressed. The RHEL sources are 102 megabytes larger than the vanilla source. And in case anyone asks, no, this swappiness change was not introduced into the vanilla kernel's long-term support tree (which is still maintained for this same kernel version). (http://www.kroah.com/log/linux/2.6.32-stable.html)

bri3d• 12 years ago

I don't think this issue has much to do with Red Hat kernel specifically - that commit has been in the mainline kernel since 3.5.

mmmooo2• 12 years ago

well redhat back ported it to their 2.6.x kernels, which, one could consider, can cause a giant regression due to a minor kernel patch (assuming you are using swappiness=0). As said by others, swappiness=0 isn't as great an idea as some people claim in the first place.

otterley5• 12 years ago

Two observations:

First, setting vm.swappiness to 0 does not have the beneficial impact that people say it does, and yet it's religious dogma to set it that way on MySQL servers. To my knowledge there has never been a single article published that clearly demonstrates its benefit under a controlled test. The evidence of any benefit is anecdotal at best.

Second, you can easily exclude any process from being killed by the kernel's OOM killer: "echo -17 > /proc/PID/oom_adj"

falcolas• 12 years ago

Second, you can easily exclude any process from being killed by the kernel's OOM killer: "echo -17 > /proc/PID/oom_adj"

Doing such with a large memory process is a sure way to cause a server to shut down and reboot when it runs out of memory. OOM killer is bad to large memory processes, but the alternative is worse.

Ideally you don't want to swap, you definitely don't want to preemptively swap, but even swapping is better than rebooting a server.

otterley5• 12 years ago

I don't know about "definitely." Swap death is often worse than a reboot because (a) response latency goes through the roof, effectively making the service down anyway, and (b) you probably can't log in anyway because of I/O saturation (ssh, shell etc. could well be swapped out).

sitkack2• 12 years ago

Agreed, too many people conflate any kind of paging with server death. Yeah servers get slow when they start paging too much, most of the time I can still login and checkout what is going on.

So much time is wasted by not enough swap space resulting in rebooted or hung machines. The way linux memory allocation works, it is really not a good idea to constrain swap. It is really easy to trigger the OOM killer by using up too much virtual address space. Doing a `fork/exec` from a process near your physical memory limit can easily get you over virtual memory limit with not enough swap, even though under this condition no paging would occur.

swap on and have great week.

ambrop73• 12 years ago

This looks like a bug. The kernel shouldn't raise OOM if there's swap available, the swappiness=0 should just make it not swap unless absolutely necessary. From my interpretation of the commit, it's supposed to improve the implementation in this respect. Quote:

"avoid swapping out pages of important process or process groups while there is a reasonable amount of pagecache on RAM so that we can satisfy our customers' requirements."

But apparently it's deciding to not swap even when there isn't RAM available, failing to allocate, and not falling back to swap.

It's also possible the bug is only in the redhat kernel, due to the correctness of this commit depending on some other commit not backported.

stingraycharles• 12 years ago

I think thats a matter of taste. Not saying i'm disagreeing with you, but I can see the reasoning behind not swapping when you set swappiness to 0.

Looking at the patch, it looks like swapiness=1 now does what the old 0 did. Would that not cover your use cass?

ambrop73• 12 years ago

"Looking at the patch, it looks like swapiness=1 now does what the old 0 did."

Yeah, it seems, but the commit message, in my incomplete understanding, contradicts what you're saying. Also, I wouldn't feel so confident making claims about what the changes do, not without looking at the context and at least some understanding of that code.

ambrop73• 12 years ago

What is the page doing with that code display? As soon as you click into it or select text inside (just like that, because you tend to do that while reading), it unexpectedly changes into some kind of plain text view and it seems you need to reload to get it back.

molecule• 12 years ago

looks like it's switching to copy-friendly presentation, so that code can be copied without line numbers (?).

I'm able to toggle between plain text and syntax highlighting via the '<>' icon @ the top of the code blocks.

zimbatm• 12 years ago

Talking about OOM: why is it that the only negotiation protocol between the kernel and userland is to hard-kill the userland processes when out of memory. I wish there was some sort of soft-OOM where the application would get notice and be given a bit of advance to free memory.

OOM relation to vm.swappiness=0 in new kernel

Related Stories