benlwalker

SPDK will be able to fully saturate the PCIe bandwidth from a single CPU core here (no secret 6 threads inside the kernel). The drives are your bottleneck so it won't go faster, but it can use a lot less CPU.

But with SPDK you'll be talking to the disk, not to files. If you changed io_uring to read from the disk directly with O_DIRECT, you wouldn't have those extra 6 threads either. SPDK would still be considerably more CPU efficient but not 6x.

DDIO is a pure hardware feature. Software doesn't need to do anything to support it.

Source: SPDK co-creator

For an expanding array in a 64 bit address space, reserving a big region and mmaping it in as you go is usually the top performing solution by a wide margin. At least on Linux, it is faster to speculatively mmap ahead with MAP_POPULATE rather than relying on page faults, too.

And, if you find you didn't reserve enough address space, Linux has mremap() which can grow the reserved region. Or map the region to two places at once (the original place and a new, larger place).

Compared to libraries like bgfx and sokol at least, I think there are two key differences.

1) SDL_gpu is a pure C library, heavily focused on extreme portability and no depedencies. And somehow it's also an order of magnitude less code than the other options. Or at least this is a difference from bgfx, maybe not so much sokol_gfx.

2) The SDL_gpu approach is a bit lower level. It exposes primitives like command buffers directly to your application (so you can more easily reason about multi-threading), and your software allocates transfer buffers, fills them with data, and kicks off a transfer to GPU memory explicitly rather than this happening behind the scenes. It also spawns no threads - it only takes action in response to function calls. It does take care of hard things such as getting barriers right, and provides the GPU memory allocator, so it is still substantially easier to use than something like Vulkan. But in SDL_gpu it is extremely obvious to see the data movements between CPU and GPU (and memory copies within the CPU), and to observe the asynchronous nature of the GPU work. I suspect the end result of this will be that people write far more efficient renderers on top of SDL_gpu than they would have on other APIs.

The eBPF programs are strictly bounded. And they're scoped to their own memory that you have to pre-load from the actual storage with separate commands issued from the CPU (presumably from the kernel driver which is doing access control checks). It's no different than uploading a shader to a GPU. You can burn resources but that's about the extent of the damage you can cause.

We tried to standardize exactly this - eBPF programs offloaded onto the device. The NVMe standard now has a lot of infrastructure for this standardized, including commands to discover device memory topology, transfer to/from that memory, and discover and upload programs. But one of the blockers is that eBPF isn't itself standardized. The other blockers are vendors ready and willing to build these devices and customers ready to buy them in volume. The extra compute ability will introduce some extra cost.

I'm still hopeful that we see it happen some day.

Yes, I've seen some clearer cases made for networking. In networking there is no standard for the hardware interface. Every vendor does their own thing. Except many can at least handle virtqueues carrying virtio-net messages for the data path, so some framework like vDPA may make sense (I'd prefer to see a full NIC interface standard emerge instead).

In storage, however, the industry has agreed on NVMe. This is a full standard for control and data plane. All storage products on the market, including DPUs and SmartNICs, just present NVMe devices. So there's no case to be made for vDPA at all. It just isn't necessary.

I don't get it either, and I'm a maintainer of SPDK which provides multiple implementations of virtualized devices and is frequently used inside DPUs to present storage devices.

If I'm implementing a hardware device anyway, why would I not just use NVMe as the interface? NVMe is superior to virtio-blk in every way that I can think of.

Even for a software device in userspace, why not use a technology like vfio-user to present an NVMe device, or just use vhost-user to present the virtio-blk device?

I've never really been able to get a clear value proposition for vDPA for storage laid out for me. Maybe I'm missing something critical - it's certainly possible.

Is it really that much code? I don't know GPU hardware, but the NVMe spec header file in SPDK is around 4k lines[0]. If there's 7 of them and they're twice as complicated each, we're still well under 100k from register map headers. I didn't actually look through Linux to see how big they are, so maybe it is that much more complex.

0: https://github.com/spdk/spdk/blob/master/include/spdk/nvme_s...

Pre-commit means before committed to the canonical repo, not before commit locally.

The SPDK project has an elaborate pre-commit review and test system all in public. See https://spdk.io/development . I wouldn't want to work on a project that doesn't have infrastructure like this.

Even mailing lists with patches are really a pre-commit review system, as are GitHub pull requests. Pre-commit testing seems more elusive though.

iWARP maybe, but I don't think you want to offload all of TCP to hardware. You want to leave congestion control and all that to software. I don't entirely know if that's why iWARP isn't very popular, but I suspect that's the reason. You want a software TCP stack that can land the data where you want it directly.

At 200+ Gbps, the copy from LLC where the packet landed to the userspace buffer dominates the performance profiles on most systems. The TCP processing isn't bad and the expensive parts can often be offloaded. I'd contend that this data copy on the TCP recv path is one of the most important performance issues to solve for the entire industry right now. DDIO let us kick the can way down the road, but it seems like the network speeds have outpaced the CPU speeds so much that it can't save us for much longer.

RDMA is great and similar, but behaves very differently from TCP in the face of network congestion and longer distance traversals. This is essentially trying to get the best parts of TCP and the best parts of RDMA combined.

XRP is a regular BPF hook in Linux and requires no additional standardization. The device never "calls out to BPF programs in the driver" - it generates a normal completion interrupt and Linux runs a BPF hook in the completion path. This is no different than other kernel BPF hooks elsewhere and doesn't provide any additional reason or need to standardize BPF.

The article misstated that XRP was a framework used for offloading BPF programs to NVMe devices. That's not correct, and XRP is not one of the emerging use cases for BPF that is driving standardization.

I am very closely tied to what the NVMe vendors want, having written the first internal draft of the proposal to the standards body (since that draft many smart people have taken the pen and done a lot of great work).

XRP is unrelated to offloading eBPF to NVMe devices.

Associating the desire of NVMe vendors to allow users to ship down eBPF programs to run on the device and XRP is a major mistake in the article. XRP has nothing to do with what the NVMe vendors want to do, and XRP is a pure kernel solution that doesn't need any participation from NVMe vendors. I think it's unclear whether XRP even has real value - it certainly may, but I believe the benchmarking in the paper was deeply flawed[1].

1. https://github.com/xrp-project/BPF-KV/issues/3

For many syscalls, the primary overhead is the transition itself, not the work the kernel does. So doing 50 operations one by one may take, say, 10x as much time as a single call to io_uring_enter for the same work. It really shouldn't be just moving latency around unless you are doing very large data copies (or similar) out of the kernel such that syscall overhead becomes mostly irrelevant. If syscall overhead is irrelevant in your app and you aren't doing an actual asynchronous kernel operation, then you may as well use the regular syscall interface.

There are certainly applications that don't benefit from io_uring, but I suspect these are not the norm.

You can also use umwait on the next completion entry in the ring

Without the submission batching you lose your system call reduction and that is far and away the biggest benefit.

Imagine you have a piece of software that runs in an event loop (as many things do). On each loop, queue up all system calls you'd like to perform. At the end of the loop, do one syscall to execute the batch. At the start of the loop, check if anything has completed and continue the operation.

If you're processing a set of sockets and on any given loop N are ready, then with epoll you do N+1 syscalls. With io_uring you do 1. It's independent of N.

IOCP certainly was ahead of its time, but it only does the completion batching, not the submission batching. io_uring is significantly better than anything available on Windows right now.

I've spent essentially the last year trying to find the best way to use io_uring for networking inside the NVMe-oF target in SPDK. Many of my initial attempts were also slower than our heavily optimized epoll version. But now I feel like I'm getting somewhere and I'm starting to see the big gains. I plan to blog a bit about the optimal way to use it, but the key concepts seem to be:

1) create one io_uring per thread (much like you'd create one epoll grp)

2) use the provided buffer mechanism to post a pool of large buffers to an io_uring. Bonus points for the newer ring based version.

3) keep a large (128k) async multishot recv posted to every socket in your set always

4) as recvs complete, append the next "segment" of the stream to a per-socket list.

5) parse the protocol stream. As you make it through each segment, return it to the pool*

6) aggressively batch data to be sent. You can only have one outstanding at a time per socket, so make it a big vectored write. Writes are only outstanding until they're confirmed queued in your local kernel, so it is a fairly short time until you can submit more, but it's worth batching into a single larger operation.

* If you need part of the stream to live for an extended period of time, as we do for the payloads in NVMe-oF, build scatter gather lists that point into the segments of the stream and then maintain a reference counts to the segments. Return the segments to the pool when it drops to zero.

Everyone knows the best way to use epoll at this point. Few of us have really figured out io_uring. But that doesn't mean it is slower.

And io_uring itself was more directly inspired by NVMe and RDMA, which of course work with these same queues as GFX cards. The original io_uring patch compares itself to SPDK, whose premise is "what if we expose an abstraction for a hardware queue per thread to an application " - basically the same programming model as io_uring. And SPDK was just taking techniques from networking (DPDK) and applying them to storage.

source - I helped create SPDK.

Windows did already have async ("overlapped") IO, and a completion aggregator (IOCP) kind of like io_uring. What Windows didn't have, and the reason they're now adding their own IORing, is the ability to submit batches of operations in a single system call. Batching operations to reduce system calls on the submission side is one of the most important features of io_uring.

The Windows IORing is only storage today, but hopefully becomes a generic system for making batched, async system calls just like on Linux.

As an alternative, may I offer you a wasm port? https://play.classicuo.org/

(I've never used this but I heard it runs well)

There's also now an open source recreation of the original client, all written in C# that actually uses a GPU so it renders 4k at 250fps instead of 800x600 at 12.5fps. It's a very mature and stable reproduction at this point.

https://github.com/classicuo/ClassicUO

For me, the killer use case for this is presenting logical volumes to containers. There just has not been an efficient mechanism for a local storage service in one container to serve logical volumes to another container on the same system until this. For VMs there is virtio/vfio-user, but for containers the highest performing option until this was NVMe-oF/TCP loopback.

Basically, you can implement a virtual SAN for containers efficiently with this.

The SPDK project is certainly looking to use this to replace our limited use of NBD, as well as present SPDK block devices as kernel block devices, including devices backed by userspace implementations of iSCSI, NVMe-oF, and various other network protocols.

It's the architectural choices more than the language (one thread per core, async, event loops). But I'm sure the c++ does have some benefit over Java.

Nope. The reason this is so complex is that these devices are actually highly parallel machines with multiple queues accepting commands. It's quite difficult to even define "before" in terms of command sequence. For example, if you have a device with two hardware queues for submitting commands and a software thread for each, if you submit a flush on one queue, which commands on the other queue does it affect?

Or what if the device issues a pci write to the completion entry that passes a flush command being submitted on the wire?

I think the only interpretation that makes sense is from the perspective of a single software thread. If that particular thread has seen the completion via any mechanism and then that thread issues the flush, then you know the write is durable. Other than that, the device makes no promises.

A flush command only guarantees, upon completion, that all writes COMPLETED prior to submission of the flush are non-volatile. Not all previously sent writes. NVMe base specification 2.0b section 7.1.

That's a very important distinction. You can't assume just because a write completed before the flush that it's actually durable. Only if it completed before you sent the flush.

I'm not very confident that software is actually getting this right all that often, although it probably is in this fsync test.