afr0ck

Why you say that? Nuvia made a massively great success with Oryon CPUs which are now all over the place.

It's also an opportunity, from a different perspective

Is this vibe-coded?

David Hildenbrand, another very involved memory management legend is picking up the role. It will be fine.

I created my first Linux from scratch when I was a freshman in college in a third world country (not India). Fast forward few years later, I now write Linux kernel code for a living. Not sure what you did wrong, bud, to end up miserable like this.

That's not how operating systems work. KVM is both an interface and a hypervisor. Just as we have different hypervisor implementations for amd, intel, arm and others all abstracted behind the same KVM interface, there is no reason the same can't be done for Gunyah. Userspace does not have to know anything about that. KVM already supports svm and vmx for amd and intel on x86. Why is something similar can't be done for Arm? Plus now there is pKVM.

I just don't understand this argument of a separate interface. The only reason you want to do that is to decouple from the KVM community, but that introduces a shit tone of duplicated effort and needless fragmentation to the virtualisation software ecosystem hindering your users from enjoying the existing upstream tools they already know about. In other terms, vendor locking and shitty downstream experience.

I worked at Linaro, who was contracting for Qualcomm. Qualcomm were pushing for some protected hypervisor called Gunyah (which had its own Linux interface and needed a new qemu port) that apparently no one liked. I tried to port it to KVM [1], but upstream folks (mostly Google) outright rejected the port. Otherwise KVM would have been available on QCOM boards. You can still try it. I have a Linux kernel and a Qemu port on my github [2,3]

[1] https://lore.kernel.org/kvm/20250424141341.841734-1-karim.ma...

[2] https://github.com/karim-manaouil/linux-next/tree/gunyah-kvm

[3] https://github.com/karim-manaouil/qemu-for-gunyah

Linux kernel + bootloaders + firmware

The Linux kernel side is mostly device trees, device drivers and the like.

u-boot is very famous as a bootloader in the embedded space

Firmware for board bring up and devices

There are Qualcomm laptops now I believe (at least that's what I heard when I was last working for them). NXP also made some boxes (I own a bunch of them). The server market is also growing with Ampere and Cavium (now Novell) which I have both.

NUMA is only useful if you have multiple sockets, because then you have several I/O dies and you want your workload 1) to be closer to the I/O device and 2) avoid crossing the socket interconnect. Within the same socket, all CPUs shared the same I/O die, thus uniform latency.

I think Meta has already rolled out some CXL hardware for memory tiering. Marvell, Samsung, Xconn and many others have built various memory chips and switching hardware up to CXL 3.0. All recent Intel and AMD CPUs support CXL.

CXL uses the PCIe physical layer, so you just need to buy hardware that understands the protocol, namely the CPU and the expansion boards. AMD Genoa (e.g. EPYC 9004) supports CXL 1.1 as well as Intel Saphire Rapids and all subsequent models do. For CXL memory expansion boards, you can get from Samsung or Marvell. I got a 128 GB model from Samsung with 25 GB/s read throughput.

It's not that deep. The futex was developed just to save you from issuing a special system call to ask the OS to put you on a wait queue.

The whole point is that implementing a mutex requires doing things that only the privileged OS kernel can do (e.g. efficiently blocking/unblocking processes). Therefore, for systems like Linux, it made sense to combine the features for a fast implementation.

right-wing religious revival rooted in Christianity, combined with technological acceleration and a reimagined political order that prioritizes heroic individuals and hierarchical impulses

Sounds like a fast path to totalitarianism a la 1930.

Inference runs like a stateless web server. If you have 50K or 100K machines, each with a tons of GPUs (usually 8 GPUs per node), then you end up with a massive GPU infrastructure that can run hundreds of thousands, if not millions, of inference instances. They use something like Kubernetes on top for scheduling, scaling and spinning up instances as needed.

For storage, they also have massive amount of hard disks and SSD behind planet scale object file systems (like AWS's S3 or Tectonic at Meta or MinIO in prem) all connected by massive amount of switches and routers of varying capacity.

So in the end, it's just the good old Cloud, but also with GPUs.

Btw, OpenAI's infrastructure is provided and managed by Microsoft Azure.

And, yes, all of this requires billions of dollars to build and operate.

France has its own independent military production including jet fighters (Rafale), tanks, ballistic missile, nuclear submarines and nuclear heads.

I use vim with mutagen for syncing files. It's simple and works fine, but you have to duplicate storage.

All of this drama is coming from the Rust email thread. Please, explain to me how this is irrelevant? I have nothing against Rust in the kernel, but Rust people are definitely dramatic.

This is not how the kernel works. You cannot rely on someone's "commitment" or "promise". Kernel maintainers was to have very good control over the kernel and they want strong separation of concern. As long as this is not delivered, it will be very hard to accept the Rust changes.

"some Russian-sounding names are banned, but we still have to demonstrate there is a due process".

That's not true! There are still many Russian maintainers in the kernel, but they are not based in Russia. They only banned individuals, based in Russia, who are employed by sanctioned companies.

Yes, when support is not upstream.

Linux kernel is not complex. Most of the code runs lock-free. For example, the slab allocator in the kernel uses only a single double_cmpxhg instruction to allocate an object via kmalloc(). The algorithm scales to any number of CPUs and has NUMA awareness. Basically, the most concurrent, lowest allocation latency allocator you can get in the market, which also returns the best objects for the requesting process on big memory systems.

The complexity on the other hand is architectural and logical to achieve scale to hundreds of CPUs, maximise bandwidth and reduce latency as much as possible.

Any normal Rust kernel will either have issues scaling on multi-cores or use tax-heavy synchronisation primitives. The kernel RCU and lock-free algorithm took a long time to be discovered and become mature and optimised aggressively to cater for the complex modern computer architectures of out-of-order execution, pipelining, complex memory hierarchies (especially when it comes to caching) and NUMA.

What you wrote doesn't make any sense. Arm has DTB [1]. Most SoCs re-use a lot of hardware IP blocks and they require very little modifications to DTB files in the kernel and device drivers to get them working. PCIe and USB support discoverability so no issue from that side.

Arm ecosystem is cleaner in my experience and learned from the mistakes of the past. Arm CPUs are still not as fast as high-end x86 chips, but it's just a matter of time before that market is also eaten by Arm.

[1] https://community.arm.com/oss-platforms/w/docs/525/device-tr...

NUMA gives you more bandwidth at the expense of higher latency (if not managed properly).

So, essentially, you're just doing cache eviction in software. That's obviously a lot of overhead, but at least it gives you eviction control. However, there is very little to do when it comes to cache eviction. The algorithms are all well known and there is little innovation in that space. So baking that into the hardware is always better, for now.

Well, it's not impossible. It's just software after all. You can mmap a remote device file, but you need OS support to do the magical paging for you, probably some sort of page ownership tracking protocol like in HMM [1], but outside a coherence domain.

I was once working on CXL [2] and memory ownership tracking in the Linux kernel and wanted to play with Nvidia GPUs, but then I hit a wall when I realised that a lot of the functionalities were running on the GSP or the firmware blob with very little to no documentation, so I ended up generally not liking the system software stack of Nvidia and I gave up the project. UVM subsystem in the open kernel driver is a bit of an exception, but a lot of the control path is still handled and controlled from closed-source cuda libraries in userspace.

tldr; it's very hard to do systems hacking with Nvidia GPUs.

[1] https://www.kernel.org/doc/html/v5.0/vm/hmm.html [2] https://en.wikipedia.org/wiki/Compute_Express_Link

That's what most hypervisors (e.g. Qemu) do on Linux when THP are enabled and allowed for the process.

- Does it mean userspace is free to allocate up to a maximum of 1G? I took pages to have a fixed size. > - Or, you mean CPUs reserve TLB sizes depending on the requested page size?

The TLB is a hardware cache with a limited number of entries that cannot dynamically change. Your CPU is shipped with a fixed number of entries dedicated for each page size. Translations of base 4 KiB pages could, for example, have 1024 entries. Translations of 2 MiB pages could have 512 entries and those of 1 GiB usually have a very limited number of only 8 or 16. Nowadays, most CPU vendors increased their 2 MiB TLBs to have the same number of entries dedicated for 4 KiB pages.

If you're wondering why they have to be separate caches, it's because, for any page in memory, you can have both mappings at the same time from different processes or different parts of the same process, with possibly different protections.

- Would memory allocators / GCs need to be changed to deal with blocks of 1G? Would you say, the current ones found in popular runtimes/implementations are adept at doing so?

- Does it not adversely affect databases accustomed to smaller page sizes now finding themselves paging in 1G at once?

Runtimes and databases have full control and Linux allows per-process policies via madvise) system call. If a program is not happy with huge pages, it can ask the kernel to be ignored, as it can choose to be cooperative.

If the dissertation is public, please do link it, if you're comfortable doing so.

I'm still in the PhD process, so no cookies atm :D

In current Linux systems, there are two main ways to benefit from huge pages. 1) There is the explicit, user-managed approach via hugetlbfs. That's not very common. 2) Transparently managed by the kernel via THP (userpsace is completely unaware and any application using mmap() and malloc() can benefit from that).

As I mentioned before, most major Linux distributions ship with THP enabled by default. THP automatically allocates huge pages for mmap memory whenever possible (that is when, at least, the region is 2 MiB aligned and is at least 2 MiB in size). There is also a separate kernel thread, khugepaged, that opportunistically tries to coalesce/promote base 4K pages into 2 MiB huge pages, whenever possible.

Library support is not really required for THP, but could be detrimental for its performance and availability on the long run. A library that is not aware of kernel huge pages may employ suboptimal memory management strategies, resulting in inefficient utilization, for example by unintentionally breaking those huge pages (e.g. via unaligned unmapping), or failing to properly release them to the OS as one full unit, undermining their availability on the long run. Afaik, Tcmalloc from Google is the only library with extensive huge page awareness [1].

Do the heuristics used by Linux THP (khugepaged) really allow completely ignoring whether pages have actually been page-faulted in or even initialised? Is a possibility unlikely to happen in practice?

Linux allocates huge pages on first touch. For khugepaged, it only coalesces the pages if all the base pages covering the 2 MiB virtual region exist in some form (not necessary faulted-in. For example, some of those base pages could be in swap space and Linux will first fault them in then migrate them)

[1] https://www.usenix.org/system/files/osdi21-hunter.pdf

Because of virtual address translation [1] speed up. When a memory access is made by a program, the CPU must first translate the virtual address to a physical address, by walking a hierarchical data structure called a page table [2]. Walking the page tables is slow, thus CPUs implement a small on-CPU cache of virtual-to-physical translations called a TLB [1]. The TLB has a limited number of entries for each page size. With 4 KiB pages, the contention on this cache is very high, especially if the workload has a very large workingset size, therefore causing frequent cache evictions and slow walk of the page tables. With 2 MiB or 1 GiB pages, there is less contention and more workingset size is covered by the TLB. For example, a TLB with 1024 entries can cover a maximum of 4 MiB of workingset memory. With 2 MiB pages, it can cover up to 2 GiB of workingset memory. Often, the CPU has different number of entries for each page size.

However, it is known that larger page sizes have higher internal fragmentation and thus lead to memory wastage. It's a trade off. But generally speaking, for modern systems, the overhead of managing memory in 4 KiB is very high and we are at a point where switching to 16/64 KiB is almost always a win. 2 MiB is still a bit of a stretch, though, but transparent 2 MiB pages for heap memory is enabled by default on most major Linux distributions, aka THP [2]

Source: my PhD is on memory management and address translation on large memory systems, having worked both on hardware architecture of address translation and TLBs as well as the Linux kernel. I'm happy to talk about this all day!

[1] https://blogs.vmware.com/vsphere/2020/03/how-is-virtual-memo... [2] https://docs.kernel.org/admin-guide/mm/transhuge.html