Dealing with Out-of-Memory Conditions in Rust

https://www.crowdstrike.com/blog/dealing-with-out-of-memory-conditions-in-rust/

by zbentley • 5 years ago

118 96 5 years ago

monocasa• 5 years ago

FWIW, this is a little outdated now. If you're willing to bite off nightly, there's cfg(no_global_oom_handling) which will just remove access to the aborting allocation API. It came out of the Linux kernel work and their very similar concerns IIRC.

tialaramex4• 5 years ago

If you cfg(no_global_oom_handling) lots of nice things go away. Which is both appropriate - those nice things did in fact depend on global OOM handling - and hopefully likely to keep more people from erroneously believing they can't afford global OOM handling.

As an example, ordinarily on Rust this does what it looks like it does:

  let mut lyrics = "Never was".to_owned();
  lyrics += " a cornflake girl";

With cfg(no_global_oom_handling) this can't work, most importantly because the second line is an arbitrary concatenation and therefore allocates, but there is no possible way to signal if it fails.

AlexanderDhoore• 5 years ago

As an embedded C guy this makes me so happy. Yes, that second line allocates and the fact that it is hidden makes me nervous. I want a programming environment that exposes all of the complexity and makes me deal with the consequences.

Waterluvian2• 5 years ago

Newbie question:

How hard would it be for a text editor to highlight all lines that do or might allocate? Can they be known statically?

umanwizard4• 5 years ago

I’ve been doing Rust full-time for two years and I had no idea you could do this. I’d just have used `push_str` in any case.

Giving this up doesn’t seem very impactful and I’d be happy to do so to enable actually useful features.

superjan2• 5 years ago

Why not budget how much memory your rust component needs, and allocate a pool ahead of time? This is how it’s sometimes done in embedded systems. It depends on what your rust code does of course, but if it’s runtime memory use depends on untrusted data it is handling that is a concern in itself.

develatio• 5 years ago

I'm not sure if this can apply to the most commonly used software. A browser can't know in advance how much memory it would need. Same thing applies for audio, image or video editing software. Probably office suites too.

This might be feasible with some limited-in-features apps, like an audio/video player, an IM client, etc...

1f60c2• 5 years ago

...and missiles: https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...

superjan2• 5 years ago

It won’t. I was thinking more about backend components where you allocate a fixed per request memory budget. It would work best if you could apply the budget per in-flight request. In case a request goes over budget, only the affected (presumably nefarious) request gets terminated, other requests and other processes will run uninterrupted.

SiebenHeaven• 5 years ago

Most commonly used software such as browsers can abort on OOM, this isn't about those cases at all. In conditions that you do not want to abort, usually you can and would want to allocate pool before hand.

tialaramex4• 5 years ago

"where the C++ pieces can all recover from OOM and the Rust component cannot"

I would like to read about their successful test of their C++ pieces using the same (random fail small allocations) strategy. My impression from Herb Sutter is that on the popular C++ Standard Library implementations (for MSVC, Clang, GCC) this actually doesn't work, but of course Herb has a reason to say that - so it'd be interesting to hear from somebody who has a reason to believe the opposite.

the_mitsuhiko• 5 years ago

Handling allocation failures on the STL is also only possible with exceptions.

oconnor6632• 5 years ago

I think I know the talk you're referring to. This is a great one: https://youtu.be/ARYP83yNAWk?t=56m56s

lmilcin8• 5 years ago

My experience with out of memory is that in every single language and environment I worked on in the past, once an application hits that condition, there is very little hope of continuing reliably.

So if your aim is to build a reliable system, it is much easier to plan to never get there in the first place.

Alternatively, make your application restart after OOM.

I would actually prefer the application to just stop immediately without unwinding anything. It makes it much clearer as to what possible states the application could have gotten itself to.

Hopefully you already have designed the application to move atomically between known states and have mechanism to handle any operation getting interrupted.

If you did it right, handling OOM by having the application drop dead should be viewed as just exercising these mechanisms.

SigmundA7• 5 years ago

I am not sure how you can make a user friendly application like that.

How about an image viewer that tries to open too large an image, should it just crash when it OOMs? I would much prefer an error dialog and the program continues to run.

lmilcin8• 5 years ago

Users are aware things stop working when memory is exhausted.

As a user I am ok with this. I don't let my memory end and would prefer that the application worked reliably while there is still memory.

What I don't like is applications trying to pretend nothing happened but then doing some strange shit.

I have seen, for example, applications using 100% cpu after they hit OOM. Yes they survived. No, it is not user friendly.

ectopod• 5 years ago

For big infrequent allocations (like your example of loading a huge image) it is easy to use a non-default allocator that returns an error rather than aborting.

As the article notes, this is all moot if you are running on Linux though. The allocation will always succeed but if it's too big the kernel will start killing processes, quite possibly including your image viewer.

tooltower2• 5 years ago

You can look at the SQLite codebase for good examples of this. Essentially, you will have to introduce a fixed upper bound for the file size you are willing to handle, given your minimum system requirements. This way, you can actually test that your program gracefully handles OOM conditions.

This page explains more on this: https://sqlite.org/limits.html

kragen• 5 years ago

That is what Linux's OOM killer does, yes. In Linux you can handle the problem by spawning off a subprocess and watching to see why it dies. Not an acceptable result for your antilock braking system firmware.

sharkbot• 5 years ago

In this case (cybersecurity application), there is a complicating factor: you cannot trust that the environment is trustworthy. The out of memory condition may have been caused by malware to disable/degrade the antivirus sensor. For example, the malware allocates just enough memory that further OS allocations fail, but it has enough memory to accomplish its task then terminates to allow normal system processing to resume.

Under that scenario, the antivirus sensor should be able to take some action (log, likely) that a malloc failed, and possibly even try to recover memory and identify the risk.

But beyond that narrow use case, you’re right.

amelius• 5 years ago

That's a design philosophy called "crash-only software".

https://en.wikipedia.org/wiki/Crash-only_software

lmilcin8• 5 years ago

I did it at least once for a credit card terminal.

The application saved state to internal database, and even if you cycled power it would just come back to the same screen and same state it was before power cycle.

This wasn't to deal with memory problems (in fact, it had no dynamic memory allocation at all) but rather to deal with some crappy external devices and users that would frequently power cycle things if it didn't progress for more than couple of seconds.

simcop2387• 5 years ago

Build perl with -DUSEMYMALLOC and -DPERL_EMERGENCY_SBRK, then you can preallocate a buffer by doing $^M="0" x 65536; then you can trap the out of memory condition with the normal facilities in language and handle it appropriately (mostly letting the big data get deallocated, or exiting). Then you can continue on just like normal. It's a weird setup and I don't think I've run into any other language with that built in.

zbentley5• 5 years ago

Useful, but on Linux it's highly likely that by the time you're comparing $@ to ENOMEM the OOM killer has already awoken and is heading your way.

jbverschoor3• 5 years ago

The next thing OSes will do is use memory compression to go against this behavior.

Behold, CS is as bad as governments with all their tax rules

zbentley5• 5 years ago

You can increase the probability of being able to survive an OOM condition long enough to emit diagnostic info in most memory-managed languages (java, python, etc.) fairly easily. Allocate (and write data to) some global fixed size chunk of ballast memory at program startup, and whenever an allocator failure is detected immediately free that block and emit whatever diagnostics you need. simcop2387 pointed out that Perl even supports this as an interpreter setting; in other languages I've done it manually.

This is more useful than OOM'd programs silently disappearing, but isn't foolproof (that's why I said "increase the probability" rather than "guarantee"): if the OOM killer gets invoked before your telemetry, if you've forked, or if whatever your language does in order to even detect an allocator error is itself memory-costly (looking at you, Python--why on earth would you allocate in order to construct a MemoryError object?!), you will still go down hard.

Other than going back in time and reversing Linux's original sin (allocation success is a lie), swap is the "solution" to these situations, but for many people that cure is worse than the disease.

I often wish there was a generally-available way to map memory to files (rather than mmaping files to memory) selectively in my programs. For cases like this--handling OOM conditions and doing cleanup/reporting before turning out the lights, when the OOM condition was caused either by code in my program outside of my control or other programs on the system outside of my control--having a "all allocations after this instant should occur in a file on disk, not memory" bit to flip would be nice. I don't need my cleanup to be fast; if this is happening I'm already on a sad enough path that I'm happy to trade shutdown/crash performance for fidelity of diagnostic data and increased probability of successful orderly cleanup.

Of course, not being an OS developer, I assume this is impossible for reasons outside of my understanding; the few complicating factors I can think of (copy-on-write pages, for example) are hard enough, and I'm sure there are others.

jbverschoor3• 5 years ago

Yeah please don’t.. memory isn’t cheap when everybody does this.

This behavior is the reason Java had such a bad time with sysops.

Just make use of proper libraries that will invalidate caches if needed, or simply refute to spawn new threads/actors/processes.

I’d rather have some application halt/pause or simply crash if it ran out of memory.

pornel4• 5 years ago

I have the same experience in C applications, but not in Rust. Rust's problems are IMHO purely self-imposed from unnecessarily nihilistic assumptions.

• There's no problem of "untested error paths". Rust has automatic Drop. Cleanup on function exit is written by the compiler, and thus quite dependable. Drop is regularly exercised on normal function exits too.

• Overwhelming majority of Drop implementations only free data, and don't need to allocate anything. Rust is explicit about allocations, so it's quite feasible to avoid them where necessary (e.g. if you use a fat error type that collects backtraces, that can bite on OOM. But you can use enums for errors, and they are plain integers).

• Probability of hitting OOM is linearly proportional to allocation size, so your program is most likely to hit OOM when allocating its largest buffers. This leaves a lot of RAM left to recover from that. It's basically impossible to exhaust memory up to the last byte.

Crashing may be fine for tiny embedded software or small single-threaded utilities. However, crashing of servers is expensive, and may not even work. When you crash, you kill all the requests that were in progress. This wastes work already done by other threads, and you're not making progress. When clients retry, you're likely to get the same set of requests that lead to OOM in the first place, so you may end up crashing and restarting in a loop forever. OTOH if you detect OOM and reject one offending request, you can keep making progress with other smaller requests.

lmilcin8• 5 years ago

Compiler can only do limited number of things and I am not talking about it.

Think in terms of trying to save the file you are working on before the application exits. Or notify cluster map that the node is not going to be available to process requests.

How does compiler factor in it?

grishka• 5 years ago

It depends on the application. On Android, for example, I don't want my app to crash with an OOM while trying to load an image. I want to instead clear my in-memory cache and try again with more memory available. So I wrapped the code that loads images into a try-catch that catches OutOfMemoryError and it worked wonderfully.

derefr• 5 years ago

The ability to "handle" OOM in userspace, exists for a particular class of software — software that is usually configured to 1. use unbounded and unpredictable amounts of memory, where 2. requests are entirely-sandboxed in their resource usage, but within the software itself, rather than at the OS level.

Basically there's only one kind of software in this class: DBMS software. DBMSes want to be able to try to process ridiculous queries if users ask them to; and then fail in a way that only affects the processing of that query, rather than the stability of the DBMS as a whole. And they also mostly can't afford the overhead of pre-calculating just how ridiculous a query will be, before attempting it; because that calculation often requires effectively 90% of the work involved in actually running the query.

For every other type of software, letting the OS handle the OOM (by killing your process) — and setting up your higher-level inter-process / inter-node architecture to be resilient to that — is the sensible approach.

mlindner• 5 years ago

Why are you ignoring the obvious examples of operating systems? An Operating System doesn't and shouldn't crash if you run out of memory.

Additionally I have personal experience with network appliances (firewalls/deep packet inspection/etc). If such a device runs out of memory, it degrades gracefully (unless there's a bug) and starts shedding connections rapidly to avoid running out of memory. Such devices can't just restart if they run out of memory as that would be a network outage and can often be exploited to cause a more serious denial of service attack.

Rust is a system's language. Handing out of memory conditions is par for the course for systems programming.

remram4• 5 years ago

All this is moot for me since I only use systems with overcommit enabled.

Is there a way to disable it for a single program or cgroup, to enable it to deal with out-of-memory conditions? Maybe changing/hooking the standard library?

mlockall(2) seems overkill, since it will also force all code and mapped files to be resident.

wyldfire2• 5 years ago

You can probably use a pool allocator that won't go back to the OS for more when it's exhausted.

I wouldn't be surprised if the popular allocators - jemalloc/tcmalloc/scudo or even the defaults in glibc/musl support a setting like this.

remram4• 5 years ago

I suppose I could use a custom allocator that calls mlock() on each page. I will look whether the ones you mention have Rust integrations and whether they expose a setting like that.

hyperman12• 5 years ago

Can't you use ulimit for this? Finding out the exact flag combination might be gnarly

remram4• 5 years ago

I don't think so. You can set a resident size limit, but you can't force all anonymous memory to be resident (or non-overcommitted).

eqvinox3• 5 years ago

Indeed it's pointless for desktop/server OSes.

But at the same time it's critical for kernel and embedded (µC, the "too small for Linux" kind) code.

Narishma• 5 years ago

Indeed it's pointless for desktop/server OSes.

Isn't it only Linux that does overcommit by default?

pornel4• 5 years ago

It's bonkers to allow programs to destabilize the whole machine to the point that the kernel has to start killing off processes just to survive.

I use https://lib.rs/cap which self-imposes memory limit on the process. Setting that limit below cgroup limit allows programs to actually handle OOM before they get killed by the OS (if only Rust's libstd wasn't so eager to self-abort anyway).

creata• 5 years ago

I use https://lib.rs/cap which self-imposes memory limit on the process.

How do you choose that limit?

vzaliva• 5 years ago

Is it the same issue Linus was pointing out when discussing possible adoption of Rust for Linux kernel?

loeg• 5 years ago

Yes. Kernel mode code needs to gracefully handle OOM conditions.

marcinjachymiak• 5 years ago

It's good to see that OOM errors can be handled in Rust now, at least in unstable.

Zig has a similar approach that is pretty cool. I don't know of any other language that let's you handle it like this https://ziglang.org/documentation/0.8.0/#Heap-Allocation-Fai...

dralley• 5 years ago

The zig approach seems really well suited to low-level libraries where you really need to handle these errors (and custom allocators, etc.) but it's kind of annoying for general purpose programs.

Dealing with Out-of-Memory Conditions in Rust

Related Stories