Async hazard: MMAP is blocking IO

https://huonw.github.io/blog/2024/08/async-hazard-mmap/

by mmastrac • 2 years ago

106 115 2 years ago

huonw.github.io

PaulHoule• 2 years ago

No secret. Reading from memory is synchronous and always has been, at least in a normal computer. (Sometimes I think of how you could fit a fancy memory controller in a transport triggered architecture but that’s something different)

xoranth2• 2 years ago

On Linux, you might be able to use userfaultfd to make it async...

dividuum• 2 years ago

I don't see how that would work. The memory access causing the page fault still blocks, but now another thread handles paging in the requested data. So without coordination between those two, nothing really changes. Sounds easier to just use nonblocking reads directly.

Thanks for the pointer to userfaultfd. Didn't know that existed.

01HNNWZ0MV43FF• 2 years ago

By blocking they mean that it can take ballpark non-volatile storage times instead of ballpark RAM times

xoranth2• 2 years ago

I believe they mean that since it bypasses the (Tokio) scheduler, so if you use it in async code you lose the main benefit of async code (namely, the scheduler is able to switch to some other task while waiting for IO to complete.). Basically the same behavior you'd get if you called a blocking syscall directly.

guerrilla• 2 years ago

GP is aware. mmap makes files act like memory. Memory is always synchronous, thus blocking, so mmaped files are always blocking. I'm surprised OP even found this surprising. It should be completely obvious.

bhawks• 2 years ago

Same thing would happen if your memory has been paged to disk too.

mjb3• 2 years ago

I like this point - it's no secret that mmap can make memory access cost the same as an IO (swap can too) - but the interaction with async schedulers isn't immediately obvious. The cost can, sometimes, be even higher than this post says, because of write back behavior in Linux.

Mmap is an interesting tool for system builders. It's super powerful, and super useful. But it's also kind of dangerous because the gap between happy case and worst case performance is so large. That makes benchmarking hard, adds to the risk of stability bugs, and complicates taming tail latency. It's behavior also varies a lot between OSs.

It's also nice to see all the data in this post. Too many systems design conversations are just dueling assertions.

a-dub3• 2 years ago

it outsources buffer management and user thread i/o scheduling to the kernel. for some use cases it's a great way to simplify implementation or boost performance. for others it may not perform as well.

the blog post points (in my mind) at some more general advice when programming which is not to mix and match paradigms unless you really know what you're doing. if you want to do user space async io, cool. if using kernel features tickles your fancy, also cool.

mixing both without a deep understanding of what's going on under the hood will probably give you trouble.

rbanffy4• 2 years ago

Making it work asynchronously would require the compiler to split the memory access into two parts, a non-blocking IO dispatch and a blocking access to the mapped address. The OS would need to support that, however, and the language would need to keep track of what is a materialised array and what’s not.

cbsmith6• 2 years ago

I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads. This is the deal you make with the devil when you use cooperative scheduling without involving the kernel, so I'm surprised it is news to people working with cooperative schedulers. These faults can happen even if you never explicitly memory map files (particularly since executables and shared libraries are often memory mapped into processes), so page faults are a blocking hazard for cooperative schedulers even without mmap.

The MMU in the hardware is aggressively parallel, and the only thread being blocked on the page fault is the one touching the page that needs to be swapped in. In reality, you can get heavily parallelized IO using mmap (indeed, it works quite well when you have a ton of IO you'd like to execute in parallel).

Moto7451• 2 years ago

I think your point here can be more generalized. Why should someone expect reading memory to benefit from async code?

The fact that the memory in this case has an access layer with exploitable latency is where the chatter about this stems from, but it misses the fundamental issue at hand.

If this was a valid concept we’d have async memcpy interfaces.

mjb3• 2 years ago

I'm surprised this is seen as a liability of mmap rather than a cooperative scheduler that isn't using native kernel threads

Indeed. In practice, though, it's easier to write high performance servers and storage systems with async runtimes (like tokio) than with native threads, at least with the current state of the ecosystem. That's not for some fundamental reason - it's possible to get great threaded performance - just the current reality.

So, whoever's fault this is, it's useful to have good evidence of this downside of async runtimes (and worth thinking about ways that OSs could let runtimes know when they were about to block on IO).

raggi2• 2 years ago

changes in the page table block the whole process, doesn't matter what combination of concurrency models you're using. we could do with a sub-process mapping API from the OS, but it's not something any major OS offers today, and requires designing for at a very fundamental level due to interaction with the hardware, and associated hardware constraints.

jandrewrogers• 2 years ago

I would describe it more as a limitation of mmap than a liability.

Modern async models have their origin in addressing serious shortcomings with the traditional POSIX APIs, particularly with respect to mmap and kernel schedulers. You can’t mix the models easily, many parts of POSIX don’t play nicely with async architectures. Traditional async I/O engines use direct I/O on locked memory, and if you use async you need to be cognizant of why this is. Half the point of async is to have explicit control and knowledge of when page faults are scheduled. Async is a power tool, not something that should be used lightly for casual tasks nor accidentally pulled in as a dependency.

The issue here appears to be that it is far too easy for someone to inadvertently mix models, not async per se. Async has substantial advantages versus native kernel threads, so getting rid of async is not a realistic solution. No one is going to give up a several-fold performance increase versus native kernel threads because some developers can’t figure out how to not mix models or the ecosystem doesn’t protect developers against inadvertently mixing models.

Async is used heavily in some C/C++ domains but it doesn’t seem to cause many issues there, perhaps because dependencies are much more explicit and intentional. Async has also been idiomatic for certain domains in C/C++ for decades so there is an element of maturity around working with it.

scienceplease• 2 years ago

cooperative scheduler that isn't using native kernel threads

Can anyone point me towards cooperative thread schedulers that use native kernel threads? Would this effectively mean implementing a cooperative model on top of pthreads?

Retr0id2• 2 years ago

This is thus a worst case, the impact on real code is unlikely to be quite this severe!

I think the actual worst-case would be to read the pages in a (pseudo-)random order.

dbaupp10• 2 years ago

(Author here.)

Good point. Do you think the difference will be observable? Will it be observable on an SSD (vs. HDD)?

Retr0id2• 2 years ago

I think so, yes - only one way to find out, though!

lowbloodsugar• 2 years ago

- register

- shadow register

- L1

- L2

- L3

- RAM

- GPU/SPU/RSP

- SSD

- Network

- HDD

The line is drawn depending on what you are doing and how.

Edit: moved Network above HDD. :-)

tux3• 2 years ago

I would put networks above HDDs (depending on how many miles you need to send your emails)

mjb3• 2 years ago

Modern data center networks offer RTTs about 100x lower than hard drive latency, and comparable to local SSD. It depends, of course, how far over the network you're going, and how fast the other side responds, but <100us is very achievable.

correnos• 2 years ago

IMO this is a strong argument for proper threads over async: you can try and guess what will and won't block as an async framework dev, but you'll never fully match reality and you end up wasting resources when an executor blocks when you weren't expecting.

dan-robertson• 2 years ago

I don’t find this argument super strong, fwiw. It could just mean ‘be wary of doing blocking operations with async, and note map makes reading memory blocking (paging in) and writing memory blocking (CoW pages)’

I think there are reasons to be wary but to me, debugging comes first (this goes two ways though: if you have a single ‘actual’ thread then many races can’t happen) because debuggers/traces/… work better on non-async code. Performance comes second but it’s complicated. The big cost with threads is heavy context switches and per-thread memory. The big cost with async is losing cpu locality (because many syscalls on Linux won’t lead to your thread yielding, and the core your thread is on will likely have more of the relevant information and lots of cache to take advantage of when the syscall returns[1]) and spending more on coordination. Without io_uring, you end up sending out your syscall work (nonblocking fd ops excepted) to some thread pool to eventually pick up (likely via some futex) load into cache, send to the os on some random core, and then send back to you in a way that you will notice such that the next step can be (internally) scheduled. It can be hard to keep a handle on the latency added by all that indirection. The third reason I have to be wary of async is that it can be harder to track resource usage when you have a big bag of async stuff going on at once. With threads there is some sense in which you can limit per-thread cost and then limit the number of threads. I find this third reason quite weak.

All that said, it seems pretty clear that async provides a lot of value, especially for ‘single-threaded’ (I use this phrase in a loose sense) contexts like JavaScript or Python where you can reduce some multithreading pain. And I remain excited for io_uring based async to pick up steam.

[1] there’s this thing people say about the context switching in and out of kernel space for a syscall being very expensive. See for example the first graph here: https://www.usenix.org/legacy/events/osdi10/tech/full_papers... . But I think it isn’t really very true these days (maybe spectre & co mitigations changed that?) at least on Linux.

dathinab3• 2 years ago

It isn't.

Async is for tasks dominated by waiting, e.g. http serving, not computations. This means it's extremely rare to run into mmap blocking related issues if you don't do something strange.

Furthermore async doesn't exclude multi threading:

- having multi threaded worker threads in addition to CPU threads is pretty normal

- having multiple async threads potentially with cross core work stealing is also the nrom

I.e. if you just follow basic advice the huge majority of task interacting in any potential performance problematic way will not be run in async task even if you write an async web server.

but you'll never fully match reality and you end up wasting resources when an executor blocks when you weren't expecting

and you wast tons of resources always even without doing something unusual with non async IO _iff_ it's about waiting dominated tasks as you have way more management overhead

furthermore in more realistic cases it's quite common that some unplanned blocking is mainly casing latency issues (which in worst case could case timeouts) but due async engines still using multi threading it not leading relevant utilization issues. That is if it's just some unplanned blocking. If you do obviously wrong things like processing large files in async tasks things can be different.

An argument against async is that depending what you use it can add complexity and that a lot of use-cases don't benefit form it's benefits enough to make it a reasonable choice. Through that is also a bit language dependent. E.g. JS is already anyway coperative in your program and using async makes things simpler here (as the alternative are callbacks). Or in pythons with GIL the perf. gain of async are much higher compared to the gains in idk. C++.

neonsunset3• 2 years ago

This kind of issue exists only in async executor implementations that cannot detect blocked workers and inject new ones to compensate for the starvation. I'm not aware if Rust has anything like this today (both Tokio and async-std are not like that) or in development for tomorrow, but there are implementations that demonstrate resilience to this in other language(s).

dbaupp10• 2 years ago

Do you have info about current (production) implementations that increase the number of workers?

In https://tokio.rs/blog/2020-04-preemption#a-note-on-blocking (2020), there's reference to .NET doing this, and an explicit suggestion that Go, Erlang and Java do not, as well as discussion of why Tokio did not.

akira25012• 2 years ago

How do other mmap/madvise options influence this (for instance, MADV_SEQUENTIAL, MADV_WILLNEED, MADV_POPULATE, MADV_POPULATE_READ, mlock)? (Hypothesis: these options will make it more likely that data is pre-cached and thus fall into fast path more often, but without a guarantee.)

That probably should have been the first thing to try. Too mad the mmap2 crate does not expose this.

Also looking at the mmap2 crate, it chooses some rather opinionated defaults depending on which function you actually call, and it makes accessing things like HUGEPAGE maps somewhat difficult.. and for whatever reason includes the MMAP_STACK flag when you call through this path.

I feel like a lot of rust authors put faith in crates that, upon inspection, are generally poorly designed and do not expose the underlying interface properly. It's a bad crutch for the language.

dbaupp10• 2 years ago

That probably should have been the first thing to try.

The point of the post is not "how to make mmap work best with async/await" or "how to optimise mmap with async/await", but exploring the consequence of incorrect code (and thus explaining why one might need potential remedies like those). Sorry if that didn't come across!

akira25012• 2 years ago

I think it's harder for you to write "correct code" here because the crate is hiding most of the actual detail from you. I put that in quotes because there's absolutely nothing incorrect about the code, it's really just suboptimal, and most probably because it can't even use the full syscall interface.

Seriously, I hate to be a curmudgeon, but that crate looks like a particularly bad and naive wrapper around mmap. It works very hard to provide you things you don't need when the basic interface is much more flexible. Aside from having to put `unsafe` around the call and re-import the kernel header constants, there's almost no reason to even have this in a crate.

rini17• 2 years ago

I looked into it (as part of optimizing LLM memory throughput) and hugepage file-backed maps aren't supported under linux. Only anonymous mappings.

pengaru• 2 years ago

water is wet

mwcampbell• 2 years ago

Nobody automatically knows everything, and we have limited energy for drawing inferences based on what we do know. So the material covered in this post isn't obvious to everyone in its target audience, especially since Rust has had some success in making systems programming more approachable to inexperienced programmers, which is a good thing.

dbaupp10• 2 years ago

(Author here)

Thank you, you've expressed one of my goals with doing this sort of investigation/getting this data far better than I have. :)

It's something that's feels obvious once the dots are connected, but I was pretty sure many people wouldn't connect these dots automatically.

malkia4• 2 years ago

WIth mmap you have to be prepared to handle unexpected page fault errors due to corrupted volume: Unlike standard read/write, where one can handle the issue, now it can happen anywhere the memory is mapped - your code, third party library, etc.

It gets even unwieldy, and now you have to add additional tracking where access is to be expected. Blindly delegating mmap area to any code path that does not have such handling, and you would have to deal with these failures.

Maybe that's not the case on Linux/OSX/BSD, but definitely is on Windows where you would have it. Also in C/C++ land you have to handle this using SEH - e.g. `__try/__except` - standard C++ handling won't cut it (I guess in other systems these would be through some signals (?)).

In any case, it might seem like an easy path to achieve glory, yet riddled with complications.

krilovsky6• 2 years ago

Yes, on POSIX systems you'd get a SIGBUS if the I/O fails or if there's no available physical memory to back the mapping.

jcalvinowens11• 2 years ago

On Linux, if you get a SIGBUS from poking a memory map that generally means you'd have certainly gotten -ENOMEM or -EIO during an equivalent sequence of syscalls (or been oom-killed, if you overcommit). Those are treated as fatal in the vast majority of programs, so dying to SIGBUS isn't meaningfully different for most usecases.

By your logic, passing a file descriptor to a library is also "unwieldy", because the library might not handle -EIO.

You can use MAP_POPULATE|MAP_LOCKED to ensure you get an error from mmap() instead of getting killed in the ENOMEM case, if you don't overcommit (if you do, you can still be oom-killed). You still get SIGBUS beyond EOF, but that's the behavior you want: it's equivalent to overrunning a buffer.

The behavior when file size isn't a multiple of PAGE_SIZE is legitimately weird (writes to the final page beyond EOF are visible to the entire system in memory but never written back to the file), but it's intuitive if you understand how the page cache works at a high level, and you can avoid it by making the size page aligned.

For more complex usecases where you really do want to handle these sorts of errors, userfaultfd() gives you all the tools you need: https://www.man7.org/linux/man-pages/man2/userfaultfd.2.html

EDIT: Initially described MAP_POPULATE wrong.

loeg3• 2 years ago

On Linux, if you get a SIGBUS from poking a memory map that generally means you'd have certainly gotten -ENOMEM or -EIO during an equivalent sequence of syscalls (or been oom-killed, if you overcommit). Those are treated as fatal in the vast majority of programs, so dying to SIGBUS isn't meaningfully different for most usecases.

No. EIO is not instantly fatal in the same way that SIGBUS is. It allows for printing an error message, associating the error with some context of what failed, and either recovering in a degraded state or exiting cleanly. Doing any of this in a SIGBUS handler ranges from unwieldy to impossible.

malkia4• 2 years ago

Passing file handle is different - a well written library would check and handle errors, but pointer to buffer - and I don't expect a library to "try" and "catch" (even if it's possible) cases. Would strlen do this? No.

loeg3• 2 years ago

It is also the case on Linux/BSD.

dsp_person• 2 years ago

One possible implementation might be to literally have the operating system allocate a chunk of physical memory and load the file into it, byte by byte, right when mmap is called… but this is slow, and defeats half the magic of memory mapped IO: manipulating files without having to pull them into memory

This doesn't defeat the purpose necessarily. How about for example, implementing a text editor: I want the best performance by loading the existing file initially (say it is <1MB), and the convenience and robustness of any writes to this memory being efficiently written to disk.

dathinab3• 2 years ago

While the general point the article is making is correct there are some issues.

- (minor issue) async example is artificially limited to 1 thread (the article states that). The issue is comparing 8 OS threads no async to 1 thread async is fundamentally not very useful as long as you didn't pin all threads to the same physical core.. So in general you should compare something async with num_cpus threads vs. num_cpus*X OS threads. Through this wouldn't have been that useful in this example without pinning the tokio async threads to CPUs to forcefully highlight the page issue, and doing it is bothersome so I wouldn't have done so either.

- (bigger issue) The singled thread async "traditional IO" example is NOT single threaded. Async _file_ IO anything between not a thing or very bad in most OSes hence most async engines including tokio do file IO in worker threads. This means the "single threaded" conventional IO async example is running 8 threads for reading IO and one to "touch the buffer" (i.e. do hardly anything).

To be clear the single threaded not being single threaded issue isn't discrediting the article, the benchmarks still show the problem it's that the 8 threaded conventional and 1 threaded async conventional are accidentally basically both 8 thraded.

dbaupp10• 2 years ago

(Author here)

The singled thread async "traditional IO" example is NOT single threaded

The threads backing the single-threaded IO are an implementation detail of fulfilling the `.read().await` calls. The key is that there's a single coordinator thread that's issuing all the work, with the user-space runtime multiplexing tasks on that thread. I thought the fact that the "start a request and come back when it is finished" behaviour happens to be implemented via user-space threads rather than kernel-level epoll (or similar) is unlikely to affect behaviour.

I considered scaling up the number of files and using a multi-threaded runtime, but I felt that'd make everything more complicated without fundamentally changing behaviour.

However, maybe my theory is incorrect, in which case someone else can do their own experiments to provide more concrete information.

(This is referenced in a footnote: https://huonw.github.io/blog/2024/08/async-hazard-mmap/#fn:t... )

The issue is comparing 8 OS threads no async to 1 thread async is fundamentally not very useful as long as you didn't pin all threads to the same physical core

The point is not to benchmark async vs. non-async, but provide a general reference point for "mmap working well" for comparison. As you suggest you agree with the "minor" issue tag, I don't think the parallelism vs. concurrency distinction matters much here... but again, definitely happy to see some concrete data that suggests otherwise!

dathinab3• 2 years ago

yes no of the issues are affecting the outcome of the blog about memory mapped files

The threads backing the single-threaded IO are an implementation detail of fulfilling the `.read().await` calls.

IMHO it's not just an implementation detail, it's a very relevant design aspect for anything related to blocking and benchmarks. Through yes it doesn't matter to much for this blog.

charleshn• 2 years ago

See also the classic "Are You Sure You Want to Use MMAP in Your Database Management System?" which mentions this common pitfall of mmap, and others, in the context of DBMS.

[0] https://db.cs.cmu.edu/papers/2022/cidr2022-p13-crotty.pdf

colonwqbang• 2 years ago

Function calls are also blocking IO then because executables and libraries are mmapped.

rmholt2• 2 years ago

While author said that C's mmap suffers the same issue, I would argue C's mmap is fine, because C doesn't have async. The issue arises from the mmap crate not having an async read and the confusion around how does async work.

rmholt2• 2 years ago

I feel like author used a lot of words to say "mmaped reads are IO (obvious) but critically, they are usually not awaitable like IO (bad)"

gpderetta4• 2 years ago

No, that misses the point. Async doesn't require an async keyword. Evented programming (which as the same issues) has been common in C for decades.

davesque• 2 years ago

I always thought that one of the use cases of memory mapping was to improve multiprocessing workloads, where a group of processes don't have to duplicate the same region of a working set. In that sense, maybe it's not surprising that single-threaded concurrency can't leverage all of the benefits of memory mapping.

dekhn• 2 years ago

I used to really like mmap for a wide range of uses (having noticed its performance in the BLAST DNA/protein search command) but over time I've come to consider it a true expert tool with deep subtlety, like a palantir.

Async hazard: MMAP is blocking IO

Related Stories