Is RAM wiped before use in another LXC container?

https://security.stackexchange.com/questions/269507/is-ram-wiped-before-use-in-another-lxc-container

by Aachen • 3 years ago

309 168 3 years ago

security.stackexchange.com

Aachen• 3 years ago

I found the currently accepted answer (by user10489) interesting, tying together a lot of concepts I had heard of but didn't properly understand. For example, looking in htop, chromium has six processes each using a literal terabyte of virtual memory, which is obviously greater than swap + physical RAM on my ordinary laptop. Or, why does an out-of-memory system hang instead of just telling the next process "nope, we don't have 300KB to allocate to you anymore" and either crash the program or let the program handle the failed allocation gracefully? This answer explains how this all works.

The TL;DR answer to the actual question is: processes generally don't get access to each other's memory unless there is some trust relation (like being the parent process, or being allowed to attach a debugger), and being in a container doesn't change that, the same restrictions apply and you always get zeroed-out memory from the kernel. It's when you use a different allocator that you might get nonzeroed memory from elsewhere in your own process (not a random other process).

dale_glass5• 3 years ago

Or, why does an out-of-memory system hang instead of just telling the next process "nope, we don't have 300KB to allocate to you anymore"

Blame UNIX for that, and the fork() system call.

It's a design quirk. fork() duplicates the process. So suppose your web browser consumes 10GB RAM out of the 16GB total on the system, and wants to run a process for anything. Like it just wants to exec something tiny, like `uname`.

1. 10GB process does fork().

2. Instantly, you have two 10GB processes

3. A microsecond later, the child calls exec(), completely destroying its own state and replacing it with a 36K binary, freeing 10GB RAM.

So there's two ways to go there:

1. You could require step 2 to be a full copy. Which means either you need more RAM, a huge chunk of which would always sit idle, or you need a lot of swap, for the same purpose.

2. We could overlook the memory usage increase and pretend that we have enough memory, and only really panic if the second process truly needs its own 10GB RAM that we don't have. That's what Linux does.

The problem with #2 is that dealing with this happens completely in the background, at times completely unpredictable to the code. The OS allocates memory when the child changes memory, like does "a=1" somewhere. A program can't handle memory allocations failures there because as far as it knows, it's not allocating anything.

So what you get is this fragile fiction that sometimes breaks and requires the kernel to kill something to maintain the system in some sort of working state.

Windows doesn't have this issue at all because it has no fork(). New processes aren't children and start from scratch, so firefox never gets another 10GB sized clone. It just starts a new, 36K sized process.

jacquesm6• 3 years ago

2. Instantly, you have two 10GB processes

No, that's not how it works. The process table gets duplicated and copy-on-write takes care of the pages. As long as they are identical they will be shared, there is no way that 10GB of RAM will be allocated to the forked process and that all of the data will be copied.

RcouF1uZ4gsC• 3 years ago

In retrospect, fork was big mistake. It complicates a bunch of other things and introduces suboptimal patterns.

reisse5• 3 years ago

Blame UNIX for that, and the fork() system call.

At least that design failure of UNIX has been fixed long ago. There are posix_spawn(3) and various clone(2) flavours which allow to spawn new process without copying the old one. And a lot of memory-intensive software actually use them, so modern Linux distros can be used without memory overprovisioning.

I'd rather blame people who are still using fork(2) for anything that can consume more than 100MB of memory.

eru2• 3 years ago

Keep in mind that copy-on-write makes analysing the situation a bit more complicated.

l33tman2• 3 years ago

The Windows example is a non-sequitur as in both cases, you end up with the 36K sized process both in Windows and Linux if you want to spawn a sub-process that exec's. The fork() => exec() path is not the issue (if there is an issue at all here), and if you use threading the memory is not forked like this to start with (on either of the OSes).

I guess the case you want to highlight is more if you for example mmap() 10 GB of RAM on that 16 GB machine that only has 5 GB unused swap space left and where all of the physical RAM is filled with dirty pages already. Should the mmap() succeed, and then the process is killed if it eventually tries to use more pages than will fit in RAM or the backing swap? This is the overcommit option which is selectable on Linux. I think the defaults seem pretty good and accept that a process can get killed long after the "explicit" memory mapping call is done.

josefx2• 3 years ago

blame UNIX for that, and the fork() system call.

Given that most code I have seen would not be able to handle an allocation failure gracefully I wouldn't call it "blame", if the OS just silently failed memory allocations on whatever program tried to allocate next you would basically end up with a system where random applications crash, which is similar to what the OOM killer does, just with no attempt to be smart about it. Even better, it is outright impossible to gracefully handle allocation failures in some languages, see for example variable length arrays in C.

0x0• 3 years ago

You don't HAVE to use fork() + exec() in Linux as far as I know. There is for example posix_swawn() which does the double whammy combo for you.

Also, if you DO use fork() without immediately doing an exec(), and start writing all over your existing allocations... just don't?

KMag4• 3 years ago

Option 3: vfork() has existed for a long time. The child process temporarily borrows all of the parent's address space. The calling process is frozen until the child exits or calls a flavor of exec. Granted, it's pretty brittle and any modification of non-stack address space other than changing a variable of type pid_t is undefined behavior before exec is called. However, it gets around the disadvantages of fork() while maintaining all of the flexibility of Unix's separation of process creation (fork/vfork) and process initialization (exec*).

vfork followed immediately by exec gives you Windows-like process creation, and last I checked, despite having the overhead of a second syscall, was still faster than process creation on Windows.

peheje• 3 years ago

I find your writing style really pleasant and understandable! Much more so than the StackExchange answer. I really like the breakdown into steps, then what could happen steps and the follow ups. Where can I read more (from you?) in this style about OS and memory management?

whartung• 3 years ago

We actually ran into this a long time ago with Solaris and Java.

Java has JSP, Java Server Pages. JSP processing translates a JSP file into Java source code, compiles it, then caches, loads, and executes the resulting class file.

Back then, the server would invoke the javac compiler through a standard fork and exec.

That’s all well and good, save when you have a large app server image sucking up the majority of the machine. As far as we could tell, it was a copy on write kind of process, it didn’t actually try to do the actual work when forking the app server. Rather it tried to do the allocation, found it didn’t have the space or swap, and just failed with a system OOM error (which differs from a Java out of memory/heap error).

As I recall adding swap was the short term fix (once we convinced the ops guy that, yes it was possible to “run out of memory”). Long term we made sure all of our JSPs were pre-compiled.

Later, this became a non issue for a variety of reasons, including being able to run the compiler natively within a running JVM.

progbits• 3 years ago

If you would like to go deeper down the rabbit hole, fasterthanlime recently made a few videos that explore this topic:

https://youtu.be/YB6LTaGRQJg https://youtu.be/c_5Jy_AVDaM https://youtu.be/DpnXaNkM9_M

The code is in Rust but that doesn't matter for the explanation.

skyeto• 3 years ago

What I find to be a rather interesting tidbit related to this is that some applications (e.g. certain garbage collectors) map multiple ranges of virtual memory that address the same physical memory.

yakubin• 3 years ago

That’s how you can create a gapless ring buffer: <https://learn.microsoft.com/en-us/windows/win32/api/memoryap...> (see scenario 1 in examples).

jeffbee• 3 years ago

In Linux, a task that tries to allocate more than the allowed amount will be immediately terminated and not notified about anything. The reason out-of-memory systems thrash is merely because people do not always bother setting the limits, and the default assumes users prefer a process that continues through hours of thrashing instead of being promptly terminated.

Sakos• 3 years ago

Who is "people" in this? If you mean your average user, it's unrealistic to expect them to know or care about details like this. This is the kind of thing that needs sane defaults.

saagarjha6• 3 years ago

For example, looking in htop, chromium has six processes each using a literal terabyte of virtual memory, which is obviously greater than swap + physical RAM on my ordinary laptop.

Generally browsers allocate large swaths of address space as guard pages (they're unmapped, but accesses into them will trap). Or they'll double map pages, so that the same physical page shows up multiple times in the virtual address space.

adamgordonbell4• 3 years ago

My shortcut for answering these questions is just replace 'container' with process and find the answer for that.

A running container is just a process that is using certain kernel features: cgroup, namespaces, pivotroot.

LXC has a different DX around containers than OCI containers do, but the same rules apply.

sgt2• 3 years ago

DX?

adamgordonbell4• 3 years ago

Developer Experience (Maybe there is a better term, but how you use them)

Docker/ OCI containers tend to be a single process and LXC containers have a collection of processes.

But in either case they are just processes running on the host in a different namespace.

So they feel different to use, but use the same building blocks (to my understanding).

commandersaki• 3 years ago

One exercise on pwn.college in their kernel security module is they have a program that forks, opens the flag file (/flag owned by root), reads the content, and then child process exits.

The program continues to run to allow you to load in shell code that'll be run by the kernel.

Your task is basically to write some shellcode to scan the memory for the flag. So now I know that at least some Linux-es don't clean up after a process exits, and you can get the contents of memory when you have kernel privileges. This is not so easy if you're scanning memory as root since /dev/mem or whatever won't reveal that process memory.

zamnos• 3 years ago

here, I made it clickable: http://pwn.college

djbusby2• 3 years ago

The memory wipe is a kernel build-time config option.

CodesInChaos4• 3 years ago

By default memory is wiped before it's handed out again, not when it's freed. This improves performance, but means secrets can remain in RAM for longer than necessary, where they can be accessed by privileged attackers (software running as root, DMA without MMU, or hardware attacks). For unprivileged processes eager and lazy zeroing look the same.

dathinab6• 3 years ago

don't clean up after a process exits

exactly, the only guarantee is that things are zeroed before handing them out to a different process, but there is some potential time gap between releasing memory back to the kernel and it being cleaned, a gap which can outlive the live of a process

and you can get the contents of memory when you have kernel privileges. This is not so easy [..] as root

yes, root has much less privileges then the kernel, but often can gain kernel privileges.

But this is where e.g. lockdown mode comes in which denies the root users such privilege escalation (oversimplified, it's complicated). Main problem is that lockdown mode is not yet compatible with suspend to disk (hibernation), even through its documentation implies it is, if your have a encrypted hibernation. (This is misleading as it refers to a not yet existing feature where the kernel creates a encrypted image which is also tamper proof even if root tries to tamper. On the other hand suspend to an encrypted partition is possible in Linux, but not enough for lockdown mode to work.)

tomcam• 3 years ago

Thank you! Annoyed at myself for not thinking it through that way.

worthless-trash2• 3 years ago

One day, the general technical population will understand containers are just processes on a system. No magic under the hood.

__turbobrew__• 3 years ago

Hey, if people keep thinking it is magic I keep getting paid to be a wizard

gregw22• 3 years ago

It’s not clear to me why the top rated reply which seemed extremely knowledgeable in discussing Linux kernel auto zeroing of memory mentioned a bunch of caveats including mentioning “some memory management libraries may reuse memory” but didn’t say “including glibc”.

That’s a pretty big library to gloss over from a security or working programmer perspective!

(Corrections to perspective welcome.)

hn92726819• 3 years ago

I thought the shared memory they were referring to would be read only memory or something. So yes, the memory is shared, but your process doesn't have any access to overwrite any of it. Likewise, when you call library functions, when it mallocs something, it uses memory that your process has write access to, not the shared library memory.

In the context of the question, I assume the asker was mostly interested in reading some kind of sensitive data from a previous process, not reading the same librsry-code-only memory or something.

Note: all of this could be wrong, it was just my understanding

Edit: looks like this answers that: https://stackoverflow.com/questions/20857134/memory-write-pr...

IcePic• 3 years ago

Even if your libc IS one of the offenders, this libc code runs in a single process, so even though libc does or does not use evil tricks for your process, it doesn't mean you will get pages from some other process memory map, only that you may get your own old pages back.

gregw22• 3 years ago

Thanks, makes sense. (Looks like the poster commenting about glibc has corrected themself.)

bluedino2• 3 years ago

Is the confusion coming from when you malloc() memory in C you aren't getting zereoed memory?

dale_glass5• 3 years ago

I tried on Linux, and got all zeroes.

Of course that's for a trivial program. If you freed something, that probably wasn't returned to the OS, and the next malloc might just recycle it.

bluedino2• 3 years ago

Malloc a bunch of memory, dirty it, free it, then malloc some more. And then check.

Your first request in your program, you'll get clean memory from the OS.

CodesInChaos4• 3 years ago

Is GPU memory wiped before it's handed to a new process, and is a process prevented from accessing GPU memory of a different process? (assuming low level APIs like Vulkan)

tryauuum• 3 years ago

this is certainly more interesting question than OPs

NotCamelCase• 3 years ago

You can request Windows to zero out every GPU memory allocation, but it's more of a driver thing that's not exposed in APIs. Such option is likely to be off by default in drivers as it might induce additional, unintended overhead. In practice, you are likely to see memory cleared more often than not due to other reasons, though.

You can't just peek another process' GPU memory thru UMD app, either. Per-process virtual memory mechanisms similar to CPUs are also present in GPUs, which is the whole reason that resources are explicitly imported/exported across APIs via special API calls.

johntb86• 3 years ago

By the Vulkan spec it is guaranteed that memory from one process can't be seen by another. https://registry.khronos.org/vulkan/specs/1.3-extensions/htm... says: In particular, any guarantees made by an operating system about whether memory from one process can be visible to another process or not must not be violated by a Vulkan implementation for any memory allocation.

In theory you could have some sort of complex per-process scrambling system to avoid leaking information, but I think implementations actually just zero the memory.

GPU drivers on different operating systems can be more or less buggy; Windows and Linux generally seem to do the right thing, but MacOS is a bit more haphazard.

saagarjha6• 3 years ago

In theory yes. In practice, there have been bugs here.

hansendc• 3 years ago

Uh... Did I miss the patches that add a pre-zeroed page pool to Linux? Wouldn't be the first time I missed something like that getting added, but 6.3-rc5 definitely zeroes _some_ pages at allocation time, and I don't see any indiciation of it consulting a prezeroed page pool: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

pmontra• 3 years ago

I thought that everybody knows how the CPU and the kernel manage memory. Everything else follows. However as a very good developer told me once, "I don't know how networking works, only that I have to type a URL." "How is that possible?" I replied, "They teach it at a school." And he told me "yes, but I studied graphic design, then discovered that I like programming."

Replace and repeat with SQL, machine language, everything below modern programming languages.

ritcgab• 3 years ago

I think the main insight is that user-level memory allocation does not necessarily involve the kernel. If the application uses `malloc` to get memory allocated, the real "application" that will (possibly) request free pages from the kernel is the memory allocator. If the space requested can be served by the pages already assigned to the allocator, it can just hand them out. So when you `malloc`, it is not zeroed, unless you know it is freshly from kernel (but you don't know the internal state of the allocator). That is why one needs to manually zero out space returned from `malloc`, or use `calloc` instead.

jcarrano• 3 years ago

I'm curious, when did OSes first begin zeroing out pages. Did the first operating systems already zero the memory?

saagarjha6• 3 years ago

OSes that have paging (and generally memory protection) have been zeroing them out from very early on, because part of the reason you have this is to avoid processes being able to mess with or spy on each other.

jk_i_am_a_robot• 3 years ago

If I could answer it would be a solid "maybe". If you're willing to consider the Translation Lookaside Buffer as part of the contents of RAM then, if the claims are to be believed, SPECTRE and Meltdown could read the contents of other processes' (and containers) RAM.

coppsilgold• 3 years ago

It is if you tell the kernel.

zero memory on free (3-5% average system performance impact, due to touching cold memory)

  init_on_free=1

zero memory on alloc (<1% average system performance impact)

  init_on_alloc=1

Is RAM wiped before use in another LXC container?

Related Stories