In emergency cases a passenger was selected and thrown out of the plane (2004)

http://lwn.net/Articles/104185/

by nkurz • 13 years ago

329 115 13 years ago

lwn.net

MattJ100• 13 years ago

Depending on its use, the first thing I generally do on a new server is disable the OOM killer.

At runtime: sysctl vm.overcommit_memory=2

To make it persist, just add vm.overcommit_memory=2 to /etc/sysctl.conf

HerrMonnezza2• 13 years ago

Note however, that turning off overcommit might yield unexpected results upon fork()/exec(): http://www.quora.com/What-are-the-disadvantages-of-disabling...

For instance, if you run a Redis server and turn off memory overcommit, you might not be able to background save.

alextingle• 13 years ago

Warning - that site requires a login.

calinet62• 13 years ago

What are the true consequences of disabling it though? What happens in the OOM situation? Or does it just not allow allocation?

adobriyan2• 13 years ago

Allocation fails with ENOMEM.

derefr5• 13 years ago

I guess I'm one of the few people(?) who like the OOM killer. If all your deployed software is written to be crash-only[1], and every process is supervised by some other process which will restart it on failure, then OOM is basically the trigger for a rather harsh Garbage Collection pass, where software that was leaking memory has its clock wound back by being forcefully restarted.

Of course, this works better when you have many small processes rather than few monolithic ones. But now you're designing an Erlang system :)

---

[1] http://lwn.net/Articles/191059/

powertower• 13 years ago

So OOM killer as Chaos Monkey? Sounds like an intresting idea that should be implemented as a config setting.

HerrMonnezza2• 13 years ago

every process is supervised by some other process which will restart it on failure

I'm curious if this works in practice for you. The current OOM algorithm in Linux sums up the memory usage of a process and all its children. So there are good chances that the restarter process is killed first, and then the main software is killed too (when OOM killer realizes the last kill didn't free enough memory).

This is exactly the problem we're facing at work here: on a computational cluster, users sometimes start wild code that consumes all the memory, but the OOM decides to kill the batch-queue daemon first, because it's the root of all misbehaving processes. We have to explcitly set `oom_adj` on the important daemons to prevent the machines from becoming unresponsive because of a bad OOM decision.

derefr5• 13 years ago

My "restarter process" is upstart. It's convenient, since the OOM-killer tries to not kill init (for bad things happen when you kill init), so it's a somewhat-safe place to put supervisory logic. One of the better calls Canonical has made, I think. :)

Still, in your use-case, I'd definitely recommend only letting users run their "wild code" inside a memory cgroup+process namespace (e.g. an LXC container.)

Crash-only systems only work when a faulty component crashes itself before it crashes you. Processes modellable as mutually-untrustworthy agents should always have a failure boundary drawn between them. (User A shouldn't be able to bring down the cluster-agent; but they shouldn't be able to snipe user B's job by OOMing their job on the same cluster node, either.) And on a Unix box, the only true failure boundaries are jails/zones/containers; nothing else really stops a user from using up any number of not-oft-considered resources (file descriptors, PIDs, etc.)

adobriyan2• 13 years ago

The current OOM algorithm in Linux sums up the memory usage of a process and all its children.

It used to sum until Linux 2.6.36.

harrytuttle3• 13 years ago

That is until you get a process that triggers OOM due to a bug on startup (pissing all over RAM) that ends up getting stuck in a restart loop which cripples your machine so you cant get in and fix it. Either that or it fails and neutralises supervisord or something resulting in site down anyway.

ulimit is a better solution i.e. set reasonable constraints based on available resources.

derefr5• 13 years ago

I've never seen a process supervisor daemon that didn't have an "if process FOO exceeds X restarts in Y milliseconds, stop trying to start FOO" clause.

btilly• 13 years ago

I disbelieve that ulimit is a real fix. If you disagree, save this shell script, set a reasonable set of ulimits then run it:

  #! /bin/sh
  sh $0 &
  while true
  do
      perl -e 'push @big, 1 while 1' &
      sh $0
  done

lucian19002• 13 years ago

A particularly bad case of the OOM killer making a mistake is in-memory databases. If some other process is using lots of memory, but still less that the DB, it'll get killed for no good reason anyway.

derefr5• 13 years ago

One would hope that any machine hosting an in-memory database has gobloads of RAM and not much else to do. :) [Still, this is what replication is for! "In-memory database" is only a scary concept if you don't have any hot slaves.]

primitur• 13 years ago

I guess I'm one of the few people(?) who like the OOM killer.

Diff'rent strokes. I also like the OOM killer, its a dastardly wonderful thing to tie lots of safety-critical things to .. and in the SIL-4 OS business (my domain), it is indeed a crash imperative to understand how to use the OOM properly. Or: not.

So in this light .. I know Erlang is "the thing" right now, but I feel I must just mention that:

Of course, this works better when you have many small processes rather than few monolithic ones. But now you're designing an Erlang system :)

.. one could also be designing a Lua-based distribution, or JVM, or whatever you like, essentially, and integrating with oom_killer. There's nothing Erlang'y about it. Because if you're playing with the oom_killer, you're really making a distribution choice, in the topology.

If your app cares about oom, well kiddo .. you better not be doing anything less than excercising complete control over your distro, its launch policies, its use of the TextSegment as installed, and so on. Absolutely you're making Distribution decisions about the functionality of the combined system. oom_killer isn't useful by itself.

My point being, worrying about oom_killer isn't just something Erlang users need think about, nor are they the only ones who really 'get' why an oom_killer can be used nicely .. if you're building a distro, either for use as an embedded machine, a tight secure server image, or indeed even as a desktop user, well .. careful memory integration is, as you say, a harsh pass.

Incidentally, I use the oom_killer exactly as you mention, in a few embedded distro applications, specifically indeed, a kill of whatever 'lua' is hogging resources. Its an extraordinarily functional mechanism for recovery ..

kudu• 13 years ago

I know Erlang is "the thing" right now

I don't think Erlang is "the thing" right now. The "hot" languages for concurrent programming are Go and Node.js, for better or for worse.

anonymous• 13 years ago

The OOM killer has almost never worked for me. I think once or twice only, the rest of the time, high memory pressure combined with high storage pressure effectively locks up the system. Easiest way to trigger it is to start eclipse, virtual box and start a system update (just package download, not install) and I can't move my mouse any more.

IvyMike• 13 years ago

This reminds me of my one and only question on Stackoverflow: "Throwing the fattest people off of an overloaded airplane." http://stackoverflow.com/q/7746648/67591

arethuza• 13 years ago

That got me wondering as to how much stuff (hand luggage, trollies, food and drink, in-flight magazines...) an average flight carries!

AmVess• 13 years ago

Rats. I read this post, and my Windows 8 install disc vanished.

eliben• 13 years ago

Sincere congrats. Losing your Windows install disc is one of the best things that can happen to a person.

jameswilsterman• 13 years ago

At least offer parachutes?

twistedpair2• 13 years ago

$150 each, correct change only.

mistercow• 13 years ago

Also, they count as your carry on bag, and they only bring one parachute on board, so if multiple people have to be thrown off and they've both paid for a parachute, they have to draw straws to decide who gets it.

Edit: And remember, OOF goes off on less than .1% of flights, so they have hundreds of times as many parachutes per flight as they have people who need them. Rumors that parachutes are oversubscribed are therefore wildly inaccurate.

cbsmith4• 13 years ago

The real irony here is that airlines actually do something very much like overcommit & OOM killer when it comes to reservations, and for precisely the same reasons: they know that not all the reservations will be used at the same time, but sometimes they do end up double booked, so then someone has to be kicked off the flight.

tcoppi• 13 years ago

Precisely. The OOM killer has its uses, and while it's debatable if it is suited for this workload or that workload, it is easily tunable for specific use cases, and you can disable it altogether if you'd like the classic behavior.

lucaspiller• 13 years ago

sometimes they do end up double booked

Sometimes is pretty much all of the time. My SO works as cabin crew for one of the large airlines you will have heard of, and they typically always overcommit. It isn't just because they are greedy and want lots of money though (well I guess it kind of is). As you said, they know some people won't turn up, but also they may decide to change the class of the airline (economy and business -> economy, business and first) depending on how people book or what aircraft are available on the day.

ubernostrum2• 13 years ago

but also they may decide to change the class of the airline (economy and business -> economy, business and first) depending on how people book or what aircraft are available on the day.

United is infamous for this, prompting a sound piece of advice: if for some reason you're in first class on a flight scheduled for an A320, don't take a seat in row 3, since odds are high it'll be swapped to an A319 (which only has two rows in F) day of the flight. In theory there's a pecking order for who gets downgraded in that case (first, anyone who moved up to F on a status upgrade, then the paid F fares starting with whoever was latest to check in), but in practice the gate agent just bumps row 3 and calls it a day.

sliverstorm• 13 years ago

It isn't just because they are greedy and want lots of money

Certainly not in the current airline environment. Even with a perfectly loaded plane, they only make a little money, and you can't plan for a perfectly loaded plane with any kind of confidence without double-booking.

cbsmith4• 13 years ago

My SO works as cabin crew for one of the large airlines you will have heard of, and they typically always overcommit.

I phrased what I said poorly. They always allow overcommit, but they try to work it out so that on average they efficiently make use of their seats without constantly having to give people giveaways and bad customer experiences.

bruceboughton3• 13 years ago

Except they do this before the flight, not once you've taken off.

Though Ryanair could well be heading down this route [1]

[1] http://www.independent.ie/irish-news/three-ryanair-mayday-ca...

Kliment• 13 years ago

Very clever, that, once you think about it. "The on-time airline" running late, stuck in holding pattern, claim you're out of fuel, get landing priority, no more delay. About as classy as the other Ryanair tactics.

cuu508• 13 years ago

It's not the best analogy as when airplane is low on fuel, dropping a passenger wouldn't make a big difference (I'm assuming).

But think of a a hypothetical situation where low fuel situations somehow were common, unforeseeable and unavoidable. And if dropping a passenger could indeed make a difference between full crash or safe landing. It's a shitty situation, but it makes sense to give somebody a chute and kick them out, instead of letting all die.

furyg3• 13 years ago

Whoa..

At least two memorandums were sent to Ryanair pilots detailing the company's concern about what was described as "excess fuel explanations" -- a description of the reasons flight commanders have to give if they take on extra fuel over the recommended minimum fuel load.

Is "because it's only the minimum!" not a valid answer?

cbsmith4• 13 years ago

Right, so they do it at pretty much the precise time that everyone grabs the seats. They don't somehow get everyone on the plane, take off without everyone in their seats, and THEN kick out passengers.

mhb• 13 years ago

Also, Samoa Air charges by the pound. Maybe there could be some kind of auction system for "heavier" processes to "pay" more to keep running.

http://business.time.com/2013/04/08/air-travel-by-the-pound-...

mikeash• 13 years ago

That's more analogous to the strategy of pausing processes which require memory when there isn't any available, and then resuming them once memory becomes available. The airline still lets you complete your trip when they bump you from a flight, you just have to wait until capacity becomes available for you.

The problem with translating that approach to OSes is that you can easily deadlock the entire system.

cbsmith4• 13 years ago

It depends on circumstance. For a lot of people, having their flight canceled and having to switch to another is all but catastrophic. You basically have to restart and build a new travel plan, and much of what you set out to accomplish gets lost.

So... a lot like killing a process.

BrainInAJar2• 13 years ago

Yes, but that's more like fork or malloc failing in the first place than letting you run and then randomly killing something when the system gets overcomitted

kalleboo3• 13 years ago

Here's a novel way to deal with an out of memory situation caused by slow memory leaks in a long-running server process: start swapping memory that hasn't been touched in literally days or weeks to /dev/null, and pray the process doesn't ever need it again.

jbrechtel• 13 years ago

Is pray a syscall?

kalleboo3• 13 years ago

I'm pretty sure there's a linux kernel module. Obviously it'd be required for my method to work. Might not be doable in FreeBSD (incompatible with the branding).

bonzoesc• 13 years ago

That's so indescribably worse than just killing that process the mind boggles. Breaking in a simple, predictable, and detectable way vs. corrupting data and hoping (excuse me, "praying") nobody notices.

kalleboo3• 13 years ago

vs. corrupting data and hoping (excuse me, "praying") nobody notices.

Well it seems to work for MySQL

Anyway, you wouldn't ever return the nulled data. If the process tries to access the data, THEN you crash it.

networked• 13 years ago

Compressing the memory area used by the leaky process in question would be a gentler and more robust "solution" here. There already exists a Linux kernel module called "zram" [1] with which you can accomplish just that (though you might have to tune your swappiness a bit first).

[1] https://en.wikipedia.org/wiki/Zram

fmax30• 13 years ago

Better yet instead of sending it to /dev/null save it on disk and reload it whenever the process needs it again. Oh wait that is how virtual memory works.

lcampbell• 13 years ago

This works for an overcommitted load, but the parent talks about a leak. In this case, you'll eventually exhaust swap.

antocv• 13 years ago

The few cases when Ive seen OOM invoked, it took couple of minutes to kill chromium after flash (of course) messed up, during that time the system was unresponsive and it killed few random smaller processes until it hit the correct one, flash or chromium in some weird interdependent bug. Either way, I wasnt too happy.

After a while I noticed when the bug triggered/the system started becoming unresponsive, and I had a terminal with killall -9 chromium & killall -9 flash-plugin ready to go, so I could myself preempt it and OOM wouldnt get involved. There has to be better mechanism than OOM.

BrainInAJar2• 13 years ago

There has to be better mechanism than OOM.

Like not overcomitting memory in the first place

Piskvorrr3• 13 years ago

Too slow for the command to pass through the usual "X - WM - TE - shell - killall" chain? Try Alt+SysRq+f ; that, in my experience, is waaaaay faster for invoking oom_killer.

pritambaral• 13 years ago

His issue is that oom_killer doesn't get it right straightaway. Alt+SysRq+f would still need multiple invocations before it gets flash. Personally, I do use Alt+SysRq+f since it predictably targets GMail-on-Chrome everytime on my system. That is usually enough on my desktop OS for me to jump in manually kill the offender. I can then just F5 GMail.

jballanc• 13 years ago

I wonder why Linux hasn't adopted something like OS X's "Sudden Termination" mechanism: https://developer.apple.com/library/mac/documentation/Cocoa/...

calinet62• 13 years ago

Well FFS that makes a lot of sense.

OSX is just like, "Hey guys, if any of you happen to not need your memory sometimes, would you mind kindly letting me know and I'll go ahead and let you go at a convenient time?" Meanwhile Linux goes on a murderous rampage with unpredictable effects.

erichurkman• 13 years ago

... And Windows is sitting in a corner, fans kicked on high, trying valiantly to manage with swapping to disk until a sysadmin gives up trying to connect via RDC and yields to the age-old "have you tried turning it off and back on again?"

plorkyeran• 13 years ago

The memory alert stuff on iOS is similar; when the phone runs low on free memory it tells the app free up as much as possible, and then only starts killing processes if there still isn't enough available.

It can be a bit finnicky to work with, but it does seem to be pretty much strictly better than just killing random processes without giving them a chance to be better behaved.

ajross2• 13 years ago

That's a few layers up the stack. Sudden Termination is a message to the OS management layer that an app can recover robustly from an unexpected death. That's not a job for the kernel, and presumably Darwin has some handling analagous to the OOM killer for the situation where memory truly is exhausted.

FWIW: Android baked that idea into the framework from the start. All apps are essentially required by the application lifecycle to be robust against sudden failure, and the system is designed to give apps fair warning to save themselves at various times (e.g. on backgrounding). If you think about it, on a battery-powered embedded system that's pretty much a firm requirement anyway.

tmzt• 13 years ago

Android accomplishes this using a slightly modified OOM killer, which it can control directly through /proc. In addition to "out of memory condition" which traditional OOM killer supports, it has the "low memory killer" (LMK). Processes on Android that are not supposed to be randomly terminated are protected by adjusting the OOM/LMK settings, which the Android system does when launching Davlik for that activity/service.

This may have changed in recent version of Android, as I haven't kept up with internals that much.

Systemic33• 13 years ago

Has anyone forwarded this to Ryanair yet?

Argorak• 13 years ago

If you want to seperate your passengers in first and economy class, this is the relevant guide:

http://lwn.net/Articles/317814/

Usually, I recommend that database and queue servers run the database/queue with a priority that makes it unlikely for them to be killed.

I had a case where a colleague running a script on a server with high pressure killed the queue, which is unadvisable, even if is crash-safe. Before that, the queue was running for 1.5 years straight.

ajdecon• 13 years ago

It is, in fact, possible to make a process immune to the OOM killer:

echo -17 > /proc/$PID/oom_adj

where $PID is the process ID you want to protect.

oom_adj can be tuned with other values to make a process more or less likely to be killed.

http://www.oracle.com/technetwork/articles/servers-storage-d...

axylone• 13 years ago

oom_adj is depreciated, use oom_score_adj instead.

https://www.kernel.org/doc/Documentation/filesystems/proc.tx...

zw123456• 13 years ago

Some years back I was flying a small commuter who used small prop type airplanes (I call them pterodactyl air). Part way through the flight, I noticed one prop seemed like it was not working, so I leaned forward to alert the co-pilot (the plane was that small). He told me that they would turn off one engine and "feather the prop" to save fuel. I told him that I would be happy to take a up a collection back in the cabin from the other passengers to pay for the extra fuel to power both engines. He chuckled, but I was serious. I never flew with them again.

Maybe there is a way to suspend a process (feather the prop) rather than completely kill processes.

yannyu• 13 years ago

Just FYI, pretty much every plane with multiple engines is able to safely fly and land with only one engine working, and all commercial pilots are trained to do so. If you were halfway through the flight already, it's entirely possible you were already in descent and didn't need the other prop at all to complete the flight.

yardie• 13 years ago

Wow, that was ridiculous. So, because you don't understand how planes work this is the pilots' fault. Even if you were serious about the fuel exactly where do you propose they put it? In-flight refuelling of commercial planes doesn't exist.

cmbaus2• 13 years ago

My memory is a bit hazy in this area, but I think by default memory is over committed in Linux. What that means is malloc() can return an address that doesn't have physical memory assigned in the page table. Memory isn't committed until it is written to.

This isn't the case with the default MSVC implementation of malloc() in Windows. In Windows address space is reserved and committed with VirtualAlloc(), and typically that is done in one step.

I think memory is over committed because Linus wanted to keep the memory footprint lower than NT early on in the development of the kernel. The drawback is applications may segfault when writing to memory that was successfully returned by malloc().

kyberias2• 13 years ago

You seem to have some terminology problems here. Windows VirtualAlloc may "commit" memory but that does not mean it actually reserves physical pages [1]. That always happens only when the memory is accessed. On the other hand, MSVC's malloc() probably uses HeapAlloc(), which in turn uses VirtualAlloc(). I don't think there are any fundamental differences between Linux and Windows here.

[1] "Actual physical pages are not allocated unless/until the virtual addresses are actually accessed." http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...

MichaelGG• 13 years ago

That link says: "Allocates memory charges (from the overall size of memory and the paging files on disk) for the specified reserved memory pages."

It does count against the total memory allowed. My laptop has 8GB of RAM and 1GB page, giving me a 9GB overall commit limit. If I spawn a process that eats up 1GB at a time, even Task Manager can clearly show me going up and hitting 8/9GB, then I'll get OOM in my process.

Windows won't commit memory that a process can't use. You can't overcommit, although you might end up in the pagefile. Without the odd concept of fork, you don't end up with processes having huge "committed" address spaces that aren't ever going to be used.

The note about physical pages is just saying it's not mapping it, not that it's not guaranteeing it.

cmbaus2• 13 years ago

VirtualAlloc can reserve address space and commit pages for it depending on the flag provided. Address space is a resource of the process, and pages are a resource for the entire system.

fusiongyro• 13 years ago

Or, here's a crazy idea: how about we actually allocate the memory when you call malloc(), and if there isn't any, give you an error instead? Programs could check the return code and decide what to do when they run out of memory themselves. Crazy, I know.

ars• 13 years ago

You can do that if you like. Use vm.overcommit_memory - setting it to 2 will still allow overcommit if there is room in swap for it. (So a process doesn't have to use the memory, it's just assured of a place in swap should it need it.)

wmf• 13 years ago

If you disable overcommit then large processes can't fork()/exec(). In theory we should all switch to spawn() but that's not going to happen.

mikegagnon2• 13 years ago

That post is a great, poetic allegory. But ultimately, I think the analogy presents a bad idea. The allegory makes the point that we could entirely avoid OOM errors by engineering a system such that resources are never overcommitted. This is true; we could do that.

However it would be bad.

Under-committing resources (thus removing the need for an OOM killer) will NOT lead to a net gain compared to over-committing resources (and thus requiring an OOM killer of some sort).

If we are unwilling to overcommit resources then it would be woefully uneconomical to run algorithms that have bad worst-case performance (because to avoid over committing you would necessarily need to assume the worst case is encountered every time).

It's just not feasible to avoid algorithms that have bad worst-case performance. Rather, we need to develop better abstractions for dealing with components (e.g. computations, programs, processes, threads, actors, functions etc.) that go over budget. Here's my attempt at developing a better abstraction for web servers: mikegagnon.com/beergarden

Ultimately, we need to treat every system like a soft real-time system, because at the end of the day every program has timeliness requirements and has resource constraints. The current POSIX model does not provide such abstractions and I think that's why we have these debates about OOM killers.

AsymetricCom• 13 years ago

I love this blog's UI. Did you make it yourself?

I like the idea of the doorman but what if you could somehow pass back useful math to the client? Of course, then you'd have to disregard that useful work yourself or double check it negating the energy savings. Or, perhaps return a map of traveling sales man type problem (maybe a map of metadata and their traversal cost), and they could navigate that map depending on exactly what kind of data they really want, thus reducing your load for valid, heavy requests and if they return a path with lots of nodes, you know to de-prioritize it or drop the request.

mikegagnon2• 13 years ago

I love this blog's UI. Did you make it yourself?

Thank you! Yes, I made the UI. It's open source: sidenote.io

I like the idea of the doorman but what if you could somehow pass back useful math to the client?

I think it would be great to have clients perform useful computations instead of just burning cycles. But that's not MVP, even for a research project.

joelthelion• 13 years ago

I wonder if someone is suddenly going to come up with a magical solution for the OOM problem and put and end to all these pointless discussions.

In emergency cases a passenger was selected and thrown out of the plane (2004)

Related Stories