Towards fearless SIMD, 7 years later

https://linebender.org/blog/towards-fearless-simd/

by raphlinus • 1 year ago

177 175 1 year ago

Very interesting! I posted a vector and quaternion lib here a few weeks ago, and got great feedback on the state of SIMD on these things. I since have went on a deep dive, and implemented wrapper types in a similar way to this library. Used macros to keep repetition down. Split into three main sections:

  - Floating point primitives. Like this lib. Basically, copied `core::simd`'s API. Will delete this part once core::simd is stable. `f32x8`, `f64x4` types etc, with standard operator overloads, and utility methods like `splat`, `to_array` etc.

  - Vec and Quaternion analogs. Same idea, similar API. Vec3x8, Quaternionx8 etc.

  - Code to convert slices of floating point values, or non-SIMD vectors and quaternions to SIMD ones, including (partial) handling of accessing valid lanes in the last chunk.

I've incorporated these `x8` types into a WIP molecular dynamics application; relatively painless after setting up the infra. Would love to try `Vec3x16` etc, but 512-bit types aren't stable yet. But from Github activity on Rust, it sounds like this is right around the corner!

Of note, as others pointed out in the thread here I mentioned, the other vector etc libs are using the AoS approach, where a single f32x4 value etc is used to represent a Vec3 etc. While with this SoA approach, a `Vec3x8` is for performing operations on 8 Vec3s at once.

The article had interesting and surprising points on AVX-512 (Needed for f32x16, Vec3x16 etc). Not sure of the implications of exposing this in a public library is, i.e. might be a trap if the user isn't on one of the AMD Zen CPUs mentioned.

From a few examples, I seem to get 2-4x speedup from using the x8 intrinsics, over scalar (non-SIMD) operations.

camel-cdr2• 1 year ago

Why do the x4/x8 types seem to be the default in rust?

A portable SIMD feature should encurage portable SIMD and not a specific vector register size.

the__alchemist4• 1 year ago

This sounds like a great idea. I went with this approach because I'm new to SIMD, so I aped the most promising API (core::simd), extending it naturally.

I need to think through the consequences. It might involve feature gates, and/or an enum. So, for example, instead of:

  pub struct f32x8(__m256);

It might be this internally, with some method to auto-choose variant based on system capability?:

  pub enum f32_simd {
    X8(__m256),
    X16(__m512), 
  }

etc. Thoughts?

janwas5• 1 year ago

Amen! I really do not understand this. It has been 7 years since SVE was introduced.

Writing an application in terms of a specific lane count loses performance portability - either it's too many, or too few, for the particular CPU. And it also enables/encourages antipatterns like putting RGB in Vec4.

dzaima16• 1 year ago

Seems rustc nightly does successfully vectorize the first sigmoid example: https://rust.godbolt.org/z/e1WYexqWY

Also there's progress on making safe intrinsics safe: https://github.com/rust-lang/stdarch/pull/1714

ashvardanian7• 1 year ago

I’ve said it before and I’ll say it again: Rust feels like a Python developer’s idea of a high-performance computing language. It’s a great language for many kinds of applications — just not when you need to squeeze out every bit of performance from advanced hardware.

Even before getting into SIMD, try using Rust for concurrent, succinct, or external-memory data structures. It quickly becomes clear where the friction is.

Cargo is fantastic — clean, ergonomic, and a joy compared to many toolchains. But it’s much easier to keep things simple when you don’t have to support dozens of AVX-512 variants, AMX, SME, different CUDA generations, ROCm, or any of the other modern hardware capabilities.

Standardising SIMD in the standard library — in Rust or C++ — has always been a questionable idea. Most of these APIs cater to operations that compilers already auto-vectorize reasonably well, and they barely touch the recent capabilities of SIMD. Just consider how hard it is to build any meaningful abstraction over the predicate/register models across AVX-512, SVE, and RVV.

RVV aside, this should illustrate the point: https://www.modular.com/blog/understanding-simd-infinite-com...

the__alchemist4• 1 year ago

I guess it comes down to application. If you don't attempt to find the most general solution, you can dodge those pitfalls. Case in point, abstracting over AVX-512, SVE, and RVV may be tough, but picking one is fine (On nightly only for now), can with the right abstractions can be almost as straightforward as using normal scalar values. I don't have a solution on the CUDA variants either; have been hard-coding that as well... (Cudarc lib with CUDA-version feature gates and GPU-series-specific code). Haven't hit a brick wall yet, but might... or might not.

lifthrasiir• 1 year ago

Just consider how hard it is to build any meaningful abstraction over the predicate/register models across AVX-512, SVE, and RVV.

Note that Highway mentioned in the post does take care of this, which is no easy feat but also a proof that it is doable.

dzaima16• 1 year ago

I don't think Rust is particularly problematic here. As long as you don't want to do funky things like use immutable argument memory as temporary scratch space (with you restoring the values afterwards of course), all it means is some `unsafe`ing at worst, compared to C/C++. And there are some safe abstractions you can make over loads/stores (everything else being safe, even if not yet marked as such).

Do agree that a standard SIMD type is rather pointless, if not immediately, then in like 5 years. (and, seemingly, both Rust and C++ are like over 10 years behind on SIMD, so they're already out-of-date)

Maybe somewhat useful if you just want the simple ~8x speedup, and not squeeze out the last 1.4x or whatever, but autovectorization should be capable of covering a significant amount of such.

ashvardanian7• 1 year ago

… except for byte-level processing, variable-length codecs, or mixed-precision numerics. That never works with autovectorization and can’t be solved with general-purpose SIMD wrappers. For me the solution was to implement those manually, and even at a scale of just 2 libraries I’ve eneded up with somewhat different project layouts & dispatch mechanisms: https://github.com/ashvardanian/SimSIMD , https://github.com/ashvardanian/StringZilla

One big family not covered there, is sparse data-strictures and related algorithms. I’ve only started integrating scatter/gather in AVX-512 and SVE, and on synthetic benchmarks both look promising: https://github.com/ashvardanian/less_slow.cpp/releases/tag/v...

Those should probably unlock a much wider set of applications for SIMD, but designing libraries for those may benefit from yet another project structure.

scottlamb2• 1 year ago

Do agree that a standard SIMD type is rather pointless, if not immediately, then in like 5 years. (and, seemingly, both Rust and C++ are like over 10 years behind on SIMD, so they're already out-of-date)

What language would you consider to have cutting-edge SIMD support?

I've dipped my toes into SIMD with Rust, [1] on stable with platform-specific intrinsics (SSE2, AVX2, NEON). I would have liked to use stable `std::simd`. I learned that (particularly on AVX2) getting things into the right lanes efficiently is a pain. I would have liked to just use `simd_swizzle!` for that part, and mix that with intrinsics calls. My approach of writing a small C++ or unstable Rust program that does the swizzling and then copying the intrinsics operations it chose into my program's "source" code worked, but I prefer to not have a manual copy'n'paste step between compilation and assembly.

If there's something much better out there in another language, well, I'd be very interested to see it.

[1] I wrote this: https://github.com/infiniteathlete/pixelfmt/blob/main/docs/s...

creata• 1 year ago

Rust feels like a Python developer’s idea of a high-performance computing language.

I might be wrong, but I think it sounds more like Rust doesn't move as far away from the C or C++ way of doing things as you want it to. At the very least, Rust is no worse than C or C++ at any of the things you mentioned.

ashvardanian7• 1 year ago

In my (biased) experience, Rust is much harder to use for advanced projects, than C and C++. On the bright side, it’s also harder to misuse :)

pclmulqdq11• 1 year ago

Rust is easy to understand as "a language by browser writers for writing browsers." That statement alone gives you most of Rust's design choices:

* Safety over everything else

* Very good single threaded performance

* Javascript-like syntax and a Javascript-like package manager

I have been working on some software in Rust recently that needs bit and byte manipulation, and we have "unsafe" everywhere and hugely complicated spaghetti compared to the equivalent code in C.

19328122675• 1 year ago

I love your username, btw :)

pjmlp6• 1 year ago

Most likely because many keep forgetting that bit and byte manipulation in C is a mix of implementation defined and UB, depending on how it is coded.

pcwalton5• 1 year ago

I have been working on some software in Rust recently that needs bit and byte manipulation, and we have "unsafe" everywhere and hugely complicated spaghetti compared to the equivalent code in C.

I'm curious what makes this so different from my experience. I rarely ever have to write "unsafe", and I'm writing quite low-level engine code that certainly uses bit manipulation. In fact, crates like bitflags and fixedbitset make it so easy that I tend to get dinged in code reviews for using bit flags when structs of booleans would be simpler :)

motorest5• 1 year ago

* Javascript-like syntax and a Javascript-like package manager

I think this is not serious criticism. The "javascript-like package manager" reference actually refers to fixing a major problem with the developer experience playing legacy programming languages such as C or C++. Java has those, .NET has those, every single mainstream programming language has those. Except C or C++.

Rust might be riddled with "the emperor has no clothes" aspects, but having a package manager is not it.

19328122675• 1 year ago

I've written a fair bit of SIMD code in Rust, and it definitely had lots of sore spots.

The main advantage was that, because Rust doesn't use TBAA, it's completely legal (and safe, if you use bytemuck) to freely cast pointers and values around. TBAA in C++ makes it much easier to hit undefined behavior.

But also, because of various miscompilations, Rust refuses (or at least refused) to pass SIMD arguments in registers, so every non-inlined function call passes arguments via the stack. There were also miscompilations if you enabled a target_feature for just one function, so we ended up just passing `-C target-cpu=...` globally, and if we wanted to support a different microarchitecture, we just recompiled the whole program. On top of that, there's no good way to check to see what microarchitecture you're compiling for, so we had to resort to specifying the target cpu in multiple places, with comments reminding us to keep the places in sync.

pcwalton5• 1 year ago

Even before getting into SIMD, try using Rust for concurrent, succinct, or external-memory data structures. It quickly becomes clear where the friction is.

It's the exact opposite for me: I use concurrent data structures more often in Rust than I do in C++ because I don't have to worry about dumb data race bugs. If one of my Bevy systems is slow, I slap par_iter() on the query and if it compiles it probably works, or at least fails for a not-stupid reason.

SleepyMyroslav• 1 year ago

What do you think about 'par_iter' having to wait for work imbalance to return execution back? With 'when_all' like primitive one can continue execution on any thread without losing one for waiting. ps as someone who does not have rust job i would like to see an example how rust deals with task based systems if you have public one at hand ofc.

pclmulqdq11• 1 year ago

Concurrent data structures are rarely faster than "lock + non-concurrent data structure" and if you're putting constructs like par_iter() in a lot of places, there's a good chance you would be better off with the "dumb" pattern than the concurrent data structure.

The same goes with Arc - if you're using it a lot there's a good chance that code with a lot of Arcs is slower than equivalent GC-ed code.

jandrewrogers2• 1 year ago

One of the things I find interesting about SIMD is that a lot of behavior that is “undefined” for scalar types in C-derived languages is explicitly fully defined when you use SIMD intrinsics with the same integral types. UB exists to hide the fact that major CPU architectures give different results for basic ALU operations in some cases. SIMD makes no such pretense of abstraction. If I am using AVX-512 I explicitly get the full Intel architecture experience, the implementation details are not hidden behind UB. Same with ARM, etc.

For example, shift overflows are masked on x86, zero-filled on ARM, and undefined in C/C++. In SIMD-land, none of this is hidden and so you design your code to leverage the reality that those instructions behave differently, whereas in C/C++ only the behavior they have in common is “defined”.

The vector ISAs are sufficiently different from each other (and normal CPUs) that it is like trying to build a compiler that can automagically produce optimized code for both CPUs and GPUs from the same source tree. I am not optimistic that this will happen anytime soon. AVX-512 essentially started life as a GPU ISA, which probably explains the interesting fact that a modern x86 CPU core has more AVX-512 registers than x86 registers.

grandempire20• 1 year ago

Of course. Many things are UB because dictating a policy for all machines doesn’t make sense. Whereas AVX is a specification for a specific hardware capability.

athrowaway3z• 1 year ago

https://xkcd.com/378/

imtringued• 1 year ago

Python doesn't have anything remotely similar to Cargo that isn't written in Rust (uv is). It's exactly backwards. Python is benefitting from Rust philosophy.

fulafel4• 1 year ago

Rust feels like a Python developer’s idea of a high-performance computing language.

I'd say the language complexity (and it keeps piling more things) means it's pretty far from Python's ideas.

Julia is closer.

andrepd3• 1 year ago

Rust is an extraordinary simple language. Compare it to the complexity of C++.

beng-nl• 1 year ago

I’m seriously interested; what is the best language/franework/method that will squeeze out every bit of performance from advanced hardware that you know of?

isusmelj• 1 year ago

I’ve been playing around with SIMD since uni lectures about 10 years ago. Back then I started with OpenMP, then moved to x86 intrinsics with AVX. Lately I’ve been exploring portable SIMD for a side project where I’m (re)writing a Numpy-like library in Rust, mostly sticking to the standard library. Portable SIMD has been super helpful so far.

I’m on an M-series MacBook now but still want to target x86 as well, and without portable SIMD that would’ve been a headache.

If anyone’s curious, the project is here: https://github.com/IgorSusmelj/rustynum. It's just a learning exercise for learning Rust, but I’m having a lot of fun with it.

IshKebab2• 1 year ago

A problem for RISC-V is going to be that there's currently no way for user code to detect the presence of RVV. I have no idea how you can do multiversioning with that limitation.

hmry• 1 year ago

The solution is to ask the OS to detect it for you. Linux offers a syscall for this (riscv_hwprobe). Has the drawback that it requires OS support, of course. But RVV requires OS support anyway (e.g. managing mstatus, saving vector registers on context switch), so that seems fine to me.

dzaima16• 1 year ago

There is some work on an OS-agnostic feature detection C API: https://github.com/riscv-non-isa/riscv-c-api-doc/blob/main/s.... Still quite new though, and potentially might change (as it did a month ago).

janwas5• 1 year ago

Or also getauxval? Highway has code to check for this, including that vectors are at least 128 bits: https://github.com/google/highway/blob/master/hwy/targets.cc...

thomashabets2• 1 year ago

Oh? Isn't that what this does? std::arch::is_riscv_feature_detected!("v")

Hmm… now that I actually experiment with it, I can't get it to return `true` on hardware that does support it, unless I also compile with -Ctarget-feature=+v. And if I do, then the binary crashes with SIGILL before getting to that point on hardware without rvv.

So if it's always equal to cfg!(target_feature="v"), then what does that even mean?

I have created https://github.com/rust-lang/rust/issues/139139

fulafel4• 1 year ago

What happens there when you try to execute a missing RVV instruction ? On other archs you get a SIGILL which you can handle.

dzaima16• 1 year ago

With RISC-V being an open ISA a vendor can freely add some non-RVV thing on the instruction space used by RVV if it doesn't desire to support RVV. And that already exists with Xtheadvector (aka RVV 0.7.1) where thead made hardware with pre-ratification RVV that's rather incompatible with the ratified RVV1.0 but still uses generally the same encoding space.

IshKebab2• 1 year ago

It's "reserved" which is basically the same as C's UB - anything can happen (nasal dragons etc.) so you can't rely on it.

quesomaster9000• 1 year ago

It's a shame that the `MISA` CSR is in the 'Privileged Architecture' spec, otherwise you could just check bit 21 for 'V', but that appears to only be available in the highest privilege machine-mode.

Presumably your OS could trap attempts to read the CSR and allow it, but if not then it's a fatal error and your program shits the bed, otherwise you rely on some OS-specific way of getting that info at runtime.

DeathArrow• 1 year ago

Using SIMD in C#:

https://xoofx.github.io/blog/2023/07/09/10x-performance-with...

https://learn.microsoft.com/en-us/dotnet/standard/simd

Towards fearless SIMD, 7 years later

Related Stories