My Fear of Commitment to the First CPU Core

https://www.jabperf.com/my-fear-of-commitment-to-the-1st-cpu-core/

by tanelpoder • 3 years ago

221 40 3 years ago

snvzz4• 3 years ago

But for as long as the kernel defines a cpumask_first() function, I will always worry that some other Linux feature might employ it to force unrelated, latency-spiking workloads upon that core.

Very Linux. It's complex, very complex, and full of footguns like this.

The real answer is to use seL4[0]. Among its extensive formal proofs, it has proof of WCET (Worst Case Execution Time), therefore unlike Linux and other hopelessly non-deterministic unorthogonal toy OSs, seL4 can actually guarantee latency bounds.

0. https://sel4.systems/About/seL4-whitepaper.pdf

im_down_w_otp4• 3 years ago

We've built things on seL4 (https://github.com/auxoncorp/ferros) before. We like to joke that it's the most perfect piece of nearly featureless software ever made.

There's... A LOT... of work to do before seL4 is going to be anywhere near usability parity with something like Linux, unfortunately.

Rather than make a general purpose OS, we decided to use it more like a unikernel or "library OS" where you're trying to make a well defined kind of "appliance" image to deploy to specific hardware rather than try to fake being a POSIX-y shaped OS. At some point we plan to leverage the WASM ABI to help with stability across versions & toolchains and stuff a WASM runtime in there too, so that support for "applications" written in something other than Rust is easier. Though, it's a pretty low priority at the moment since it's not at all core to our business presently.

PaulDavisThe1st• 3 years ago

At some point we plan to leverage the WASM ABI to help with stability across versions & toolchains and stuff a WASM runtime in there too, so that support for "applications" written in something other than Rust is easier.

the GP was talking about guaranteed low latency execution. You're orbiting a different star here.

cmrdporcupine2• 3 years ago

huh, cool, I had just been looking at ferros a few hours ago, and now coincidentally it comes up for discussion on HN.

I was meaning to circle back and see how difficult it is to throw together uni-kernel type work with it, but didn't see much in the way of documentation for how to ... use it.

Is the general idea here to target use of sel4+Rust for embedded work? Or is it feasible to think about this stuff in the context of server workloads on e.g. x86_64 machines?

I have a fantasy of writing a database engine that sits close to baremetal, completely managing its own memory for page buffers and so not engaging in negotiations with the Linux virtual memory system at all.

But such an application would need a robust networking and block device I/O stack. Something I'm not sure I can expect in sel4 land.

fbdab103• 3 years ago

What kind of appliances have you created/planned?

I love the idea of seL4, and was sorely disappointed to see Australia cut their funding. Decades of exploits have me convinced that microkernels are the only real path towards security.

RobotToaster• 3 years ago

If you want determinism, wouldn't you use a RTOS?

mnd999• 3 years ago

Usually because the tooling for development on Linux is better and/or more familiar and they doing high frequency trading, not trying to fly a plane.

Some HFT systems are written in Java - they just give it loads of RAM, disable the garbage collector and restart it once a day.

vgatherps• 3 years ago

For HFT you generally just shove the OS out of the way. It exists to provide basic access the machine and for the non-fast-paths, while the hot paths never touch it in the first place. You also aren't hard-real-time, where it's literally failure if you ever exceed a latency target.

In this context, Linux is fairly sensible as it has decent knobs for getting out of the way and is well supported for all of your non-fast-path-code.

mort96• 3 years ago

Don't RTOSes usually sacrifice average latency for max latency guarantees? That seems like the wrong trade-off for HFT.

kjs3• 3 years ago

unlike Linux and other hopelessly non-deterministic unorthogonal toy OSs

You should team up with the Hurd folks. They have, what, 30 years of the same impotently shaking their fists at the rest of the world touting how their perfect little academic exercise and its tens of users is the future of computing if everyone else on the planet wasn't so st00pid. Really. Any day now.

They really have the angry old CS professor schtick down. You could learn a lot.

snvzz4• 3 years ago

You should team up with the Hurd folks.

I use every chance to drive people away from the Hurd.

That project is a waste of time. It has failed to move away from Mach or address the Hurd critique document. It has no direction and thus no future.

Instead, I try to promote efforts that merit the attention, such as seL4 or Genode.

xyzzy_plugh• 3 years ago

Once when I was much younger first encountering NUMA I came across a bit of code that always did a few magical things with the first physical CPU. We had a fairly elaborate hardware configuration and several peripherals were bound to the first core.

Ever since, when I do any sort of CPU or core pinning or grouping, I leave the first core or two alone if possible. It's convenient to say the first two cores are for all the ancillary stuff, but the last 30 are for our workload.

Now I realize why I never witnessed something like this in all benchmarks I performed... I always ignore the first CPU core.

anonymousiam• 3 years ago

Smart move! I've always known about core 0 and IRQ processing, but this article introduced me to new dangers (under Linux) associated with core 1.

Love your user name too! Plover! Zorton!

masklinn• 3 years ago

Does that work with the recent moves to asymmetric cores? Do they set the E cores as the first cores of a socket?

formerly_proven• 3 years ago

No, P cores are first, then E cores, at least on ADL.

trollied2• 3 years ago

Thanks for submitting this Tanel, very interesting! (If anyone doesn't know who Tanel is, he's an Oracle performance expert)

daviddever23box• 3 years ago

Agreed - multi-core processing is now ubiquitous, from a Raspberry Pi on down, and negative affinity concepts ought to be in every junior engineer's toolkit (NOT that they should be cargo-culted, but understood and deployed where it is beneficial).

medawsonjr• 3 years ago

Hi, I'm the author. I'm glad you enjoyed it, and pleasantly surprised that Tanel posted it already. I was right in the middle of posting it myself an hour ago when I noticed it had been posted and had fostered a discussion already :)

trollied2• 3 years ago

Brilliant, thank you! I used to be a performance engineer many moons ago & love this content. Makes me want to get back into it!

tanelpoder2• 3 years ago

Thanks for writing this up and doing the work preceding the writeup. Every time I read your articles, I learn something important and new (and I consider myself pretty good in the Linux perf/troubleshooting space :-)

tanelpoder2• 3 years ago

Thanks for the props :-) This might not mean that much here. But one very useful thing that came out of the long Oracle RDBMS performance career is that now I'm not afraid of addressing (attacking?) systems of any complexity. As the Oracle DB is one of the most complex pieces of software out there (it can also be a cache-coherent distributed system, BTW) so once you learn how to handle that, plus all the OS, storage interactions, you'll be capable of dealing with most (if not all) RDBMS out there :-)

teddyh• 3 years ago

Possibly related talk at FOSDEM two weeks ago: https://fosdem.org/2023/schedule/event/sched_tracing/

The talk is ostensibly about generating graphs, but it also describes a patch to make Linux prefer the previously-used core for any task, to make massively parallel tasks not jump around and end up with two threads on the same core.

mrguyorama• 3 years ago

I've always considered the second, or second fastest, CPU core to be the real King for this reason. The OS will want to use the first CPU core for various things, for whatever reasons they came up with, sometimes by accident, and without warning.

I doubt OS's ever purposely utilize the second core lol.

My Fear of Commitment to the First CPU Core

Related Stories