Improved default settings for Linux machines

http://tobert.github.io/post/2014-06-24-linux-defaults.html

by dctrwatson • 12 years ago

140 95 12 years ago

tobert.github.io

lazyant2• 12 years ago

I'm very much against changing kernel settings in production servers without really understanding the implications. Take for example the "swappiness = 0", most likely what you think it does it's not what it does.

voidlogic• 12 years ago

vm.swapiness is not magic: http://en.wikipedia.org/wiki/Swappiness

People should of course learn about these setting as they change them, but many of the suggested values really are improvements for lots of use cases.

lazyant2• 12 years ago

swapiness=0 in some Linux versions disables swapping, also even with swapiness=0 and plenty of RAM, kernel sometimes will swap; people put too much faith in these values intead of doing proper capacity planning/monitoring imhho

mackal• 12 years ago

Yeah, my swappiness is set to default (60) I still never use my swap space :/

jcurbo• 12 years ago

Indeed, I wish this post had his rationale behind the changes and descriptions of what each change does.

phunge2• 12 years ago

Looking through his list though, most of the settings are less controversial than swappiness.

I do wish people were more rigorous about changing swappiness. Maybe if swappiness=0 was documented as "pageout every byte of /usr/sbin/sshd and /lib/libc.so before you pageout a single byte of java's bloated heap" then people would be less eager to apply it.

sp332• 12 years ago

Since kernel 3.5-rc1, vm.swappiness=0 will disable swapping entirely. https://news.ycombinator.com/item?id=7660385

bcoates2• 12 years ago

Are there people running servers that actually swap without it being a disaster? For regular 'cattle not pets' servers I just decline to create a swap partition and vm.panic_on_oom = 1.

Java's bloated heap, for example, has easy to use startup options that cause the Java process to use a deterministic amount of RAM, as does any other sane program that uses nontrivial amounts of memory.

OOMing (or swapping) a server almost always indicates an error that needs human intervention, so cleanly killing/rebooting the faulting system and raising the alarm in your monitoring system is the right thing to do.

jebblue2• 12 years ago

Well I can attest that having user's file limits set lower than 4096 will cause some web servers to not handle a multi-user client performance test of any decent size.

guidedlight• 12 years ago

Isn't "swappiness = 0" recommended for SSD-based swap partitions to reduce SSD wear?!?

emeraldd• 12 years ago

I've seen swappiness = 1 as a recommendation rather than 0.

khc• 12 years ago

if you want to reduce SSD wear, just don't have a swap partition on SSD (or by not having a swap partition at all if you only have SSD)

Rapzid2• 12 years ago

It's like a new pair of shoes; give it a few weeks and you won't mind the wear so much.

simoncion• 12 years ago

Relatively modern SSDs will write at least 500TB before dying:

http://techreport.com/review/26523/the-ssd-endurance-experim...

I've an SSD that's been running in my development Linux laptop for very, very close to three years. The drive houses a couple of encrypted swap partitions along with the rest of the system. According to the SMART attributes, I've written 18TB to it in that time.

Don't worry about SSD wear. Really, don't. Either you'll get a drive that succumbs to crib death or super-shitty v1.0 firmware, or you'll get a drive that will last until long after you outgrow it.

AlTobey5• 12 years ago

My recommendation for any server with enough RAM is to disable swap. This works on just about every system whether swap is configured or not.

That said, on my Linode I have swappiness=1 and a few GB of swap on the SSD since it is memory-constrained.

In any case, I've been running vm.swappiness=0 or 1 on nearly every machine I touch for many years now and have yet to see any problems. Swap is almost always a bad idea on modern machines.

Thanks for reading!

aidenn02• 12 years ago

I strongly disagree, and prefer the swappiness at the default level with a modestly sized swap file (1-2GB). There are lots of poorly behaved applications that will allocate RAM that they never touch again. If you turn down swappiness you end up wasting that RAM which could be better used for disk cache.

I currently have 1.2GB of swap in use on a machine that is not doing any active swapping at all; that's 1.2GB more space for caching.

[edit for spelling]

usefulcat2• 12 years ago

I do a lot of dev work on a machine that has 12 cores and 96 GB of RAM and I currently have swappiness set to 85.

96 GB would be plenty of memory for what I need to do--mostly I benefit from lots of filesystem caching. BUT--there's another user of this system and he does most of his work using Matlab. For the stuff he's doing, Matlab routinely sucks down 10s of gigabytes of memory. And it may stay that way for days at a time.

Without swapping, Matlab will happily sit on all that memory indefinitely, whether it's doing anything or not. Meanwhile, everything I do takes ages because of the paltry amount of memory left for FS caching.

With swapping, I can get some of that memory back for FS cache when Matlab isn't being used, and it makes a huge difference.

babuskov• 12 years ago

I'd agree with you only if you'd agree with me that:

There can never be enough RAM. ;)

kev009• 12 years ago

Sorry but this is really bad 'default' advice. Shocking, but defaults often are default for a reason. Cranking everything up to 11 is a sign of ignorance in which case you need to step back and understand what you are doing first.

The mmap, file-max, and SHM advice is application dependent. Understand what your system is doing, and only increase if necessary. i.e. PostgreSQL < 9.3 is the only large user of SHM I can think of off hand.

The limits.conf advice is also bad. You should have a safety net here and increase these as needed and per user in /etc/security/limits.d

A less harmful guide would be something like "these are the knobs you may need to turn for certain apps, and here is the documentation on what they affect" - this looks a bit better https://wiki.archlinux.org/index.php/sysctl

mathnode• 12 years ago

In fact any RDBMS of a similar design to Ingres requires a larger shmax value including, postgres, sybase axe, and informix.

colanderman5• 12 years ago

Ya, this reminds me of car modders who suggest stupid things like drilling holes in your air intake. Any variable whose change (1) is cheap or free, (2) doesn't violate some emissions standard or whatnot, and (3) doesn't cut into profit of a pricier model, is practically guaranteed to be set by the manufacturer at its optimal value for the vehicle's intended use. They're not morons and they have an interest in maximizing the car's utility.

Sysctl variables in Linux meet all these same criteria. Linux is already tuned for general use (or is damn close to it). Any knob you tweak is likely to make things worse in a way you don't understand. Just leave things unless you have a specific use case that requires different tuning.

zorbo• 12 years ago

is practically guaranteed to be set by the manufacturer at its optimal value

The optimal value for what? Speed? Performance? Comfort? Economics? I don't drive a car, but wouldn't trade-offs apply to car tuning just like it does to everything else?

camperman• 12 years ago

mplayer and VLC also use SHM and are completely broken with these settings.

stevekemp• 12 years ago

I put together a simple page of some notes here:

http://tweaked.io/guide/kernel/

In practice the file-handles tweak is the one that many people seem to forget, and can make a big difference for heavy webservers.

lloeki• 12 years ago

instead of blindly suggesting noatime, investigate on nodiratime and relatime, which may or may not be what one needs.

Side note: noatime is possible on OSX too [0]

[0] https://github.com/tlvince/noatime-osx/blob/master/com.tlvin...

alex_duf• 12 years ago

Don't change max values unless it's really needed. Not every production machine needs billions of IPC handles.

My philosophy is keep it do default unless you have an issue. Guess what ? It works just fine.

AlTobey5• 12 years ago

Increasing the max setting does not consume additional resources. It merely makes it possible for applications to get the resources they ask for.

For example, just this morning Chromium started failing because I hand't disabled limits on one of my machines. I pulled down my standard settings, applied them, then the problem went away. It wont' be coming back either.

adobriyan2• 12 years ago

Increasing the max setting does not consume additional resources. It merely makes it possible for applications to get the resources they ask for.

From unswappable kernel memory, yeah.

kernel.pid_max = 999999
* - nproc unlimited

1.000.000 pids is 1 mil task_struct's.

On my quite stripped out kernel 14 task_struct's fit into order 3 slab -- 14 objects per 32KB or kernel memory.

1000000 / 14 * 32 * 1024 = 2.18 GB of kernel memory

and that's not even counting other kernel structures!

mediaserf• 12 years ago

Swappiness is not just about swapping. http://www.linuxjournal.com/article/10678 <- This is a great article on Linux swap and how it works. It will change your life.

KaiserPro• 12 years ago

It would be good if you were to explain your reasoning for your changes.

As alluded to before, defaults are default for a reason. Having someone explain why they change them is a good exercise for both reader and author.

for example fiddling with swappyness means that you'll end up with less RAM for important things, like file cache.

AlTobey5• 12 years ago

I've added some notes explaining my reasoning. I hope that helps. I'll dig in and explain some of the settings more thoroughly in the future.

jccooper2• 12 years ago

To apply the sysctl changes right away:

sudo sysctl -p /etc/sysctl.conf

My oldish kernel doesn't recognize the PID settings, which is unfortunate.

AlTobey5• 12 years ago

Wow, how old is that kernel? I've been using that since at least the 2.6.32 era.

jccooper2• 12 years ago

I think I was on 2.6.38. Seems like it ought to work, but didn't. Dunno.

hassy• 12 years ago

Increasing the number of file descriptors on machines running HTTP servers is one of the first recommendations I make to my consulting clients.

It's much easier to overlook than you'd probably imagine. I have seen apps serving hundreds of thousands of API requests per day that had the default settings. It's one of those quick changes that can have a big impact.

colanderman5• 12 years ago

"this disables swap entirely, which I think is virtuous" -- sigh.

Swap isn't some artifact from the days of 640k, used only because memory is expensive. Shit is always stored on disk; swap just allows that shit to be unused pages of active programs rather than actively used pages of files on disk.

Without swap, you force the kernel to prioritize cold code paths of rarely used daemons over, say, your web browser's cache. That's just dumb.

coryrc• 12 years ago

Unless you care about latency. Just because an indexer ran through my hard drive doesn't mean I should have to wait 60+ seconds next time I log in for everything to page back into memory.

colanderman5• 12 years ago

I haven't had this experience in 10 years of using a desktop system that updates the locate DB every night.

suprjami2• 12 years ago

If you think swap introduces latency, you need to learn a lot about memory management.

noise• 12 years ago

locate indexer cronjob should never be enabled on a production box. Problem solved.

Jam0864• 12 years ago

How much swap space should you allocate for e.g. 16GB RAM?

Every reference I've read suggests there's no need for swap space once you have more than ~2GB of RAM, but I find that extremely hard to believe.

colanderman5• 12 years ago

I'd allocate swap on the order of how much RAM your applications use. So if they need 2 GB of RAM to be comfortable, 1-4 GB of swap will give them room to be paged out when necessary. In most situations disk space is cheap, so erring on the side of too much won't hurt.

pizza234• 12 years ago

I find this an interesting, common, conceptual misunderstanding.

When somebody asks this question, he always thinks this way:

- I have a system "A" with x GB of ram, with y GB of swap. - I have a system "B" with x+y GB of ram, and no swap, because it has all the virtual memory A is using.

well, the problem is that one should not compare system B with system A, he should compare system B with system C:

- I have a system "C", with x+y GB of ram, and z of swap.

system C will perform potentially better than B.

The generic explanation is that the kernel may decide that it's better to swap out some data, and use the space for caching purposes. This is a concept, that within limits, it's not related to the amount of RAM.

Bill_Dimm2• 12 years ago

If you plan to hibernate, I believe you need at least 16GB of swap (i.e. the amount of memory you have).

dale-cooper3• 12 years ago

For a desktop with sufficient amount of memory swap is just bad from my experiences. When i was using swap on my desktop system, which i usually have running 24/7, i'd often come back in the morning to find that linux decided to swap out my X session or something stupid like that. How much of my 16GB memory will a "rarely used daemon" use? Insignificant compared to how annoying it is to have to wait for your interactive applications to be swapped in when that happens.

Bill_Dimm2• 12 years ago

It sounds like you had a process running over night that needed enough memory to force X (or whatever) to swap. Without the swap, that memory-hungry process would presumably have run less optimally (if it got memory allocation failures and adjusted its behavior to utilize less memory) or it would have simply died. It's not so clear that having processes die because they can't get memory is better than having something else run slowly, in general.

Edit: Possible caveat: Maybe whatever was running overnight did a lot of disk I/O and the OS decided to cache it at the expense of moving idle processes to swap (not sure if Linux does that or not).

edwintorok5• 12 years ago

I turned off swap on my desktop. Even if you have a lot of RAM swap just becomes an annoyance during development. Sooner or later you have some runaway process eating up all the memory. If the system starts swapping it will take a long time to recover (unless you're lucky enough to hit Ctrl-C within a few seconds and the app responds to it). Even with swap off there are long pauses when the system is low on memory, maybe it just waits too much before the OOM killer kicks in, but its more usable than with swap on.

Freaky• 12 years ago

# allow up to 999999 processes with corresponding pids

This is now the default in DragonFlyBSD: http://freshbsd.org/commit/dfbsd/3a877e444fff816b8a340d35fe3...

singlow• 12 years ago

Interesting, but I tried them out in my Ubuntu 13.10 desktop and applying the whole lot completely killed Chrome. Strangely, certain tabs would consistently load fine, while others would end up with white screens - seemingly consistently per-uri over several browser launches and reboot. Other thing seemed to be working fine. Oddly, the settings screen was one uri that did not work. I took his file handle limits and scrapped the rest and it went back to normal. Something in there did not agree with my video card settings is my best bet.

velodrome2• 12 years ago

I have the same issue.

If I use the --disable-gpu flag, I will hit the file handle limits. I have increased the limits and it works fine now.

I have an AMD GPU and chrome/chromium just does not work. It will constantly flicker.

suprjami2• 12 years ago

Thanks so much, random guy on the internet.

I can't wait to see these settings cargo-culted onto systems of customers who then complain Linux doesn't behave the way they expect it to.

Next time, keep your sysctls to yourself.

adobriyan2• 12 years ago

Note, vm.overcommit_memory is not even mentioned.

How disappointing.

jebblue2• 12 years ago

> Edit: I've run across a few comments complaining about these large max values. The reason I set them high is that the machines I work on are not multi-user in any way.

Then why is this posted at all, this isn't improved default Linux settings, it's settings some guy likes for some customized environment.

slyall• 12 years ago

Redhat and Centos have the command "tuned-adm" which has various machine profiles with settings like this. It is an official thing supported by the vendor.

https://access.redhat.com/site/documentation/en-US/Red_Hat_E...

eg when I run it on one of our KVM hosts.

  $ tuned-adm list
  Available profiles:
  - throughput-performance
  - laptop-ac-powersave
  - virtual-guest
  - latency-performance
  - enterprise-storage
  - default
  - spindown-disk
  - desktop-powersave
  - virtual-host
  - laptop-battery-powersave
  - server-powersave
  Current active profile: virtual-host

velodrome2• 12 years ago

This is a great feature. You also have the ability to create and use your own profiles.

I wonder if they made any new improvements in RHEL 7?

Firefishy• 12 years ago

The git repo is here: https://git.fedorahosted.org/cgit/tuned.git/tree/profiles

Easy enough to implement own scripts using the profiles as a basis.

kolev• 12 years ago

This is a very clean approach preventing people from reinventing the wheel. I just found this, although not recently updated, which is tuned ported to Ubuntu: https://github.com/edwardbadboy/tuned-ubuntu

PointerReaper• 12 years ago

OMG! I can improve the audio 200% by simply setting: pactl set-sink-volume alsa_output.pci-0000_00_1b.0.analog-stereo 200%

Teh sound is so much more sound-ier! Way much more cranked up than the lame defaults! http://goo.gl/TJLTMF

markhahn• 12 years ago

wow, wrong on almost all the settings. seriously: maximize bufferbloat, cause huge intrusive IO pauses, keep useless pages in memory, etc? the only good ones there are kernel.panic = 300 kernel.sysrq = 1

angry_octet• 12 years ago

An even better idea for developers is to reduce limits (memory, PIDs, file handles) and start triggering those rarely-used (or non-existent) error handling code paths.

Also, I think I would prefer process sbrk failure to OOM killer activation. So setting vm/overcommit_memory=2, overcommit ratio to 80%, a decent swap size, and code actually handling errors. IE consistency versus randomness.

Not that randomness is bad for testing, cf Chaos Monkey: https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

mje__• 12 years ago

Agreed. The last thing you want is to have to debug those weird "it worked on my desktop" issues that come from having a really weirdly configured system

nwmcsween• 12 years ago

Well here are my 'improved' settings for sysctl.conf http://sprunge.us/dhgM. Most of the TCP stuff is to guard against server resource exhaustion by syn flood, etc, the vm settings are optimized for hot cache (vs cold cache but programs) and spinning media (page cluster).

rlpb• 12 years ago

Doing one of these things may introduce a security vulnerability, depending on the rest of your environment.

Some programs that use select(2) are known to assume that FD_SETSIZE is at least the maximum number of file descriptors available (instead of checking FD_SETSIZE). This lack of bounds checking may lead to a stack or heap overflow and a security vulnerability.

More recently, if you build with fortified glibc options, then you'll get automatic bounds checking, but do you know that your own daemons are built this way?

This is an example of why it's not a good idea to arbitrarily change a list of default settings system-wide without understanding the implications. The defaults have not been changed for a reason; otherwise distributions would already ship with these changes.

References: https://lists.ubuntu.com/archives/ubuntu-devel/2010-Septembe... http://www.outflux.net/blog/archives/2014/06/13/5-year-old-g...

Improved default settings for Linux machines

Related Stories