AVX 512 will be the future

https://www.realworldtech.com/forum/?threadid=209249&curpostid=209596

by fulafel • 4 years ago

203 271 4 years ago

PaulHoule2• 4 years ago

AVX 512 has a number of instructions that make it useful for UTF-8 parsing, floating point parsing, XML parsing, JSON parsing, things like that. It is tricky coding

https://lemire.me/blog/2022/05/25/parsing-json-faster-with-i...

All things that are good for HFT but also good for speeding up your web browser and maybe even saving power because you can dial down the clock rate. It's a tragedy that Intel is fusing off AVX 512 in consumer parts so they can stuff the chips with thoroughly pizzled phonish low-performance cores.

eesmith3• 4 years ago

I think that's what "if you aren't into parsing (but not using) JSON" is supposed to be referencing.

And, "It's not only some JSON parsing library, you'll find it in other libraries too. I suspect the most use it gets in many situations is - drum roll - implementing memmove/strchr/strcmp style things. And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it."

PaulHoule2• 4 years ago

One thing I learned in grad school is you can do a stupendous number of floating point operations in the time it takes to serialize a matrix to ASCII numbers and deserialize it. Whatever it is you are doing with a JSON document might be slower than parsing it.

It's true that autovectorization accomplishes very little but specialized libraries could have a notable effect on perceived performance if they were widely developed and used.

Frankly Intel has had less interest in getting you to buy a new computer by making it the best computer you ever bought than it has been in taking as much revenue as it can from the rest of the BOM, for instance the junk integrated graphics that sabotaged the Windows Vista launch and has been making your computer crash faster ever since. Another example is that they don't ship MKL out of the box on Windows or Linux although they do on MacOS. And Intel wonders why their sales are slipping...

zeusk2• 4 years ago

They have a replacement coming in 2023/24

kristianp2• 4 years ago

Replacement cpu generation or replacement for avx-512? Will it be variable length?

jeffbee3• 4 years ago

I don't think Intel has even decided whether to fuse them off or not. In my newest Intel desktop, a Core i7-13700K, AVX-512 is actually available. It wasn't available on the i7-12700K that was in the same system on the same motherboard a few weeks ago.

compsciphd• 4 years ago

i thought the reason they wanted to fuse them off was for all cores being completely compatible (just slower), without the need for the OS to force scheduling on specific cores (I'd guess what would have to happen is if an instruction not available on efficiency core was called, it trap, the OS would see it and then mark the process to only be executable on performance core?)

borissk• 4 years ago

Are you sure? According to Intel spec AVX-512 is not available - https://ark.intel.com/content/www/us/en/ark/products/230500/...

ilyt7• 4 years ago

Didn't read the post ? THat is explicitly mentioned

Yes, yes, yes, you can use it in random places. It's not only some JSON parsing library, you'll find it in other libraries too. I suspect the most use it gets in many situations is - drum roll - implementing memmove/strchr/strcmp style things. And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it.

bluestreak• 4 years ago

I couldn't agree more. I don't think compiler vectorization is that useful even for columnar (!) database we're building. The specialized JIT doesn't even use AVX512 because too much effort for little to no gain.

gumby• 4 years ago

And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it.

How much of that is due to language semantics? Can't FORTRAN compilers make better use of the AVX instructions than, say, a C or Rust compiler could due to memory reference semantics?

sitkack4• 4 years ago

Vector instructions (ala Cray) are future proof in ways that SIMD are not. Eventually every high performance CPU will have HBM, I am not sure SIMD is the mechanism we will use in to extract the most efficiency out of the platform.

https://people.eecs.berkeley.edu/~krste/papers/yunsup-ms.pdf

https://parlab.eecs.berkeley.edu/sites/all/parlab/files/mave...

https://www.cs.uic.edu/~ajayk/c566/VectorProcessors.pdf

https://inst.eecs.berkeley.edu/~cs152/sp18/lectures/L16-RISC...

janwas26• 4 years ago

I feel that SIMD vs vector is not the most useful distinction to make. Yes, at the ISA and binutils level there is a difference in terms of number of instructions and ugly encodings. But the actual application source code can actually look the same for SIMD and vector - vector-length agnostic style is helpful for both.

Instead, it seems more interesting to focus on preparing more applications for either SIMD or vector: with data-parallel algorithms, avoiding branches etc.

lifthrasiir• 4 years ago

Of course when you only have packed instructions (a la SIMD) you have no choice but using them in place of vector instructions, but vector instructions in general do not fully subsume packed instructions. I think packed instructions are better at exploiting opportunistic parallelism from either a short loop (where vector instructions would have higher overhead) or a series of similar but slightly different scalar instructions.

fooblaster• 4 years ago

There have been attempts to remedy the fixed width simd vector problem that are already shipping:

https://developer.arm.com/Architectures/Scalable%20Vector%20...

torginus3• 4 years ago

Generally speaking, the idea that the average person uses more than 4 cores is insane. Even as a power user/dev I can count the times when I needed more than 4 cores on one hand.

All the benchmarks that these CPUs are evaluated by, test either games, or run the same crazy stuff like fluid simulations that OPs article complains about. Hardware reviewers need a reality check.

Iwan-Zotow• 4 years ago

good porno will saturate all available cores

jl6• 4 years ago

Eh, a modern desktop can have a lot of concurrent processes running.

A browser, one process per tab, software updates continually spawning in the background…

Though I do agree that the benefits of high core count CPUs are oversold as far as most users are concerned.

eyegor• 4 years ago

I hate to break it to you, but outside of gaming, fluid sim, and battery life / power draw tests, nothing else matters for modern chips. Schmucks only need the kind of horsepower a phone is capable of, or want to know how long it will last. The only people who care about performance either really care because more perf = more dollars (fluid sim, hft, etc.) or they're gamers.

fomine32• 4 years ago

Games are really really popular application

gpderetta3• 4 years ago

I'm a C++ dev. I'll use as many cores as I can get for the build! Even my IDE can consume all the cores just by opening some files.

bitcharmer• 4 years ago

All things that are good for HFT (...)

Not sure if you're referring to high frequency trading here, but if that's the case then it's just not true. AVX is infamously bad for low latency becuse these instructions result in insane amounts of heat which forces the CPU to get down clocked (sometimes the cores on the neighboring tiles on the SoC too). P-State transitions are a serious source of execution jitter and a big no-no in our business.

Many HFT shops actually disable AVX with noxsave. It's fantastically useful for HPC thought.

dan-robertson2• 4 years ago

I think I found there’s also a quite expensive ‘warmup’ which makes much of AVX only worth it for larger loops. But my memory or original understanding might be incorrect and my understanding of why that warmup might be the case (maybe microcode caches in the front end??) is very poor.

37ef_ced3• 4 years ago

Linus Torvalds:

  And I claim that that is the real problem with AVX-512 (and pretty much any vectorization). I personally cannot find a single benchmark that does anything I would ever do - not even remotely close. So if you aren't into some chess engine, if you aren't into parsing (but not using) JSON, if you aren't into software raytracing (as opposed to raytracing in games, which is clearly starting to take off thanks to GPU support), what else is there?

Answer? Neural net inference, e.g., https://NN-512.com

If you need a little bit of inference (say, 20 ReNet50s per second per CPU core) as part of a larger system, there's nothing cheaper. If you're doing a small amount of inference, perhaps limited by other parts of the system, you can't keep a GPU fed and the GPU is a huge waste of money.

AVX-512, with its masked operations and dual-input permutations, is an expressive and powerful SIMD instruction set. It's a pleasure to write code for, but we need good hardware support (which is literally years overdue).

quelsolaar• 4 years ago

True, but the problem is that that is today better done on vector hardware like a GPU or other ML hardware. The world has sort of diverged int to two camps: vectorizable problems that can be massivly paralleleized (graphics, simulation, ML) and for that we use GPUs, and then everything else is CPU. What i think linus is saying is that there are few reasons to use AVX-512 on a CPU, when the is a GPU much better siuted for those kinds of problems.

You could say that the intersecting area in the ven diagram of "Has to run on CPU" and "Can use vector instructions" is small.

dtx1• 4 years ago

You could say that the intersecting area in the ven diagram of "Has to run on CPU" and "Can use vector instructions" is small.

People are forgetting the "Could run on a GPU but I don't know how" factor. There's tons of Situations where GPU Offloading would be fast or more energy efficient but importing all the libraries, dealing with drivers etc. really is not worth the effort, whereas doing it on a CPU is really just a simple include away.

dragontamer12• 4 years ago

few reasons to use AVX-512 on a CPU,

Memcpy and memset are massively parallel operations used on a CPU all the time.

But lets ignore the _easy_ problems. AES-GCM mode is massively parallelized as well, each 128-bit block of AES-GCM can run in parallel, so AVX512-AES encryption can process 4 blocks in parallel per clock tick.

Linus is just somehow ignorant of this subject...

titzer• 4 years ago

You could say that the intersecting area in the ven diagram of "Has to run on CPU" and "Can use vector instructions" is small.

I dunno, JSON parsing is stupid hot these days because of web stacks. Given the neat parsing tricks by simdjson mentioned upthread, it seems like AVX512 could accelerate many applications that boil down to linear searches through memory, which includes lots of parsing and network problems.

fulafel6• 4 years ago

GPUs are still an unworkable target for wide end user audiences because of all the fragmentation, mutually incompatible APIs on macOS/Windows/Linux, proprietary languages, poor dev experience, buggy driver stacks etc.

Not to mention a host of other smaller problems (eg no standard way to write tightly coupled CPU/GPU codes, spotty virtualization support in GPUs, lack integation in estabilished high level languages, etc chilling factors).

The ML niche that can require speficic kinds of NVidia GPUs seems to be an island of its own that works for some things, but it's not great.

mhh__6• 4 years ago

Firing up a GPU just to do a little bit of inference sounds quite expensive.

CPUs (let alone CPUs talking to a GPU) spend huge numbers of cycles shunting data around already.

WithinReason2• 4 years ago

We already have GPUs for that

adgjlsfhk13• 4 years ago

gpus have a high enough latency that for O(n) operations, the time it takes to move the data to the GPU will be higher than the time it takes to run the problem on a CPU. AVX-512 is great because it makes it easy to speed up code to the point that it's memory bottlenecked.

dragontamer12• 4 years ago

I'd say AES Encryption/Decryption (aka: every HTTPS connection out there), and SHA256 Hashing is big. As is CRC32 (the VPMULDQ instruction), and others.

There's.... a ton of applications of AVX512. I know that Linus loves his hot-takes, but he's pretty ignorant on this particular subject.

I'd say that most modern computers are probably reading from TLS1.2 (aka: AES decryption), processing some JSON, and then writing back out to TLS1.2 (aka: AES Encryption), with probably some CRC32 checks in between.

--------

Aside from that, CPU signal filtering (aka: GIMP image processing, Photoshop, JPEGs, encoding/decoding, audio / musical stuff). There's also raytracing with more than the 8GB to 16GB found in typical GPUs (IE: Modern CPUs support 128GB easily, and 2TB if you go server-class), and Moana back in 2016 was using up 100+ GB per scene. So even if GPUs are faster, they still can't hold modern movie raytraced scenes in memory, so you're kinda forced to use CPUs right now.

ilyt7• 4 years ago

AES Encryption/Decryption (aka: every HTTPS connection out there),

that already have dedicated hardware on most of the x86 CPUs for good few years now. Fuck, I have some tiny ARM core with like 32kB of RAM somewhere that rocks AES acceleration...

So even if GPUs are faster, they still can't hold modern movie raytraced scenes in memory, so you're kinda forced to use CPUs right now.

Can't GPUs just use system memory at performance penalty ?

323• 4 years ago

I am confused, CPUs have dedicated instructions for AES encryption and CRC32. Are they slower than AVX512?

rjsw• 4 years ago

The two ARM64 systems on the desk next to me have neural net engines built in.

bobowzki2• 4 years ago

Software defined radio.

paulmd9• 4 years ago

"boy, if I exclude all the places where it's actually used, there's really not any places that use it, amirite guys?"

begging the fucking question to the max, post better linus

superkuh• 4 years ago

UH... didn't Intel actually stop including AVX 512 in their newest (alder lake) processors? This seems unlikely. ie, https://www.tomshardware.com/news/intel-nukes-alder-lake-avx...

"Although AVX-512 was not fuse-disabled on certain early Alder Lake desktop products, Intel plans to fuse off AVX-512 on Alder Lake products going forward." -Intel Spokesperson to Tom's Hardware.

Kon-Peki3• 4 years ago

On the other hand, you can now virtually guarantee that a GPGPU is present on Intel consumer chips. So now you can write your code 3 times; no vector acceleration (for the really old stuff), AVX-512 for the servers, and GPU for the consumer chips!

convolvatron3• 4 years ago

or we can finally starting looking into languages with implicit fine grained parallelism and run on whatever

dragontamer12• 4 years ago

Yeah but AMD Zen4 has AVX512 enabled by default. And they're not too shabby with the performance either.

AVX512 is the future, but maybe not Intel's future, ironically.

CamperBob23• 4 years ago

If I need to crunch small amounts of data in a hurry, existing instructions are fine for that.

If I need to crunch large amounts of data in a hurry, I'll send it to a GPU. The CPU has no chance to compete with that.

I honestly don't understand who/what AVX512 is really for, other than artificial benchmarks that are intentionally engineered to depend on AVX512 far more often than any real-world application would.

Aardwolf3• 4 years ago

I switched from Intel to AMD Zen 4 over this.

Whether or not AVX 512 is the future, Intel's handling of this left a sour taste for me, and I'd rather be future-proof in case it does gain traction, since I do use a CPU for many years before building a new system. Intel's big/little cores (with nothing else new of note compared to 5+ years ago) offer nothing that future-proofs my workflows. 16 cores of equally performant power with the latest instruction sets does.

kolbe7• 4 years ago

Have you been programming with the Zen 4? I bought one, and I've been using the avx512 intrinsics via C++ and Rust (LLVM Clang for both), and I've been a little underwhelmed by the performance. Like say using Agner Fog's vector library, I'm getting about a 20% speedup going from avx2 to avx512. I was hoping for 2x.

ls65536• 4 years ago

I very recently upgraded and had considered both Raptor Lake and Zen 4 CPU options, ultimately going with the latter due to, among other considerations, the AVX-512 support.

Future proofing is no doubt a valid consideration, but some of these benefits are already here today. For example, in a recent Linux distro (using glibc?), attach a debugger to just about any process and break on something like memmove or memset in libc, and you can see some AVX-512 code paths will be taken if your CPU supports these instructions.

microtonal• 4 years ago

Wasn't the issue that efficiency cores don't support AVX-512 and that operating system schedulers/software don't deal with this yet and end up running AVX-512 code on the efficiency cores?

bogwog• 4 years ago

That was the exact same story I read, probably on this site.

cogman102• 4 years ago

That's a terrible CPU design. You might as well ship arm cores if you are going to have a mismatch of instruction set support on efficiency vs power cores.

Especially since apps using AVX-512 will likely sniff for it at runtime. So now, you have a thread that if rescheduled onto an efficiency core it will break on you. So now what, does the app dev need to start sending "capabilities requests" when it makes new threads to ensure the OS knows not to put it's stuff on an efficiency core?

What a dumb design decision.

pantalaimon2• 4 years ago

That's why it was only enabled if you disabled the efficiency cores - until Intel fused it off.

And if Intel would have wanted this, they could have deviced the kernel to handle the fault and pin the process to the big cores.

monocasa• 4 years ago

AVX-512 in general purpose CPUs was designed to not really make sense but start seeding the market at 10nm, vaguely make sense at 7nm, and really make sense starting at 5nm (or Intel's 14nm (Intel10) -> 10nm (Intel7) -> 7nm (Intel4)).

So Intel's process woes have been hurting their ability to execute in a meaningful way. Additionally Alder Lake was hurt by needing to pivot to heterogeneous cores (E cores and P cores), which hurt their ability to keep pushing this even in a 'maybe it makes sense, maybe it doesn't' state it had been in.

willis936• 4 years ago

AVX-512 is in Alder Lake P-cores but not E-coree. AVX-512 is disabled in Alder Lake CPUs because the world is not ready for heterogenous ISA extensions. AVX-512 could be enabled by disabling E-cores.

It was supposedly maybe actually taken out of Raptor Lake, but the latest info I can find on that is from July: long before it was released. I have a Raptor Lake CPU but haven't found the time to experiment with disabling E-cores (far too busy overclocking memory and making sensible rainmeter skins).

paulmd9• 4 years ago

Early Alder Lake could but they fused it off in the later ones and newer microcode also blocks it on the older processors.

Raptor Lake is basically "v-cache alder lake" (the major change is almost doubling cache size) so it's unsurprising they still don't have an answer there, and if they did it could be backported into Alder Lake, but they don't seem to have an immediate fix for AVX-512 in either generation.

Nobody really knows why, or what's up with Intel's AVX-512 strategy in general. I have not heard a fully conclusive/convincing answer in general.

The most convincing idea to me is Intel didn't want to deal with heterogeneous ISA between cores, maybe they are worried about the long-term tail of support that a mixed generation will entail.

Long term I think they will move to heterogeneous cores/architectures but with homogeneous ISA. The little cores will have it too, emulated if necessary, and that will solve all the CPUID-style "how many cores should I launch and what type" problems. That still permits big.little architecture mixtures, but fixes the wonky stuff with having different ISAs between cores.

there are probably some additional constraints like Samsung or whomever bumped into with their heterogenous-ISA architectures... like cache line size probably should not vary between big/little cores either. That will pin a few architectural features together, if they (or their impacts) are observable outside the "black box".

pantalaimon2• 4 years ago

AVX-512 is disabled in Alder Lake CPUs because the world is not ready for heterogenous ISA extensions

It was already opt-in (disabled unless you also disable efficiency cores), that is no justification to make it impossible to use for people who want to try it out.

But I suppose Intel just doesn't want people to write software using those new instructions.

jeffbee3• 4 years ago

For some reason this doesn't hold on Raptor Lake, the successor to Alder Lake. AVX-512 works if you disable all the efficiency cores.

the8472• 4 years ago

Some of the AVX512 instructions are nice additions though, even if you only use it on YMM or XMM registers. All the masked instructions make it much simpler to vectorize loops over variable amounts of data without special treatment for the head/tail of the data.

stephencanon3• 4 years ago

This is exactly right—-the 512b width is the least interesting thing about AVX512. All the new instructions on 128, 256, and 512b vectors, (finally) good support for unsigned integers, static rounding modes for floating point, and predication are much more exciting.

anonymoushn8• 4 years ago

I don't mind head/tail stuff so much but the masked instructions and vcompress make a lot of "do some branchy different stuff to different data or filter a list" easier and faster compared to AVX2, so I'm a big fan

cmrdporcupine6• 4 years ago

SIMD on x86 has become a ridiculous minefield of standards. I honestly pity the compiler writers who have to navigate this nonsense. It was stressful enough trying to just manage the compiler flagging in an application (music synthesizer) I wrote recently.

The core amd64 ISA is stable and that's good. I wouldn't call it elegant or nice, but it's there. The patchwork of which processor supports which SIMD extension on top of it, though, has gotten out of control and is unpleasant.

I'm sure AVX512 is great. Except as an application author I can't rely on it being there.

I am not a MacOS user or MacOS developer, but I envy the consistency of that platform now that they control their own ISA.

paulmd9• 4 years ago

SIMD on x86 has become a ridiculous minefield of standards.

people make the AVX-512 compatibility thing more complex than it has to be. there's consumer AVX-512 and server AVX-512 and within those series there has always been strictly increasing compatibility, with a common core between them. and AMD supports the complete thing with Zen4 so there is no differentiation for them.

intel didn't want to put bfloat16 ML instructions in consumer chips and people turned that into a whole thing because they like to meme about AVX-512, but that's literally the only difference. it's just not that complex a story though, if you don't turn it into a thing.

intel's consumer and server teams have been operating quasi-independently and some of them are on "newer generations" than the other team. currently that's the client team that's ahead of the server team. so client got a couple gens ahead of server, and server has this one extra instruction in their versions. that's it, that's the whole deal.

frankly it's actually a simpler story than SSE4 where AMD just kinda went and did its own thing for a while and Intel went and did AVX and there truly was total divergence between what the companies were doing. you can probably turn SSE into a real ugly venn diagram if you try to capture every nuance too! After all, some of them were introduced in K8, K9, Phenom... and there's lots of sub-versions of each.

xeon phi is also its own thing but you'll know if you're targeting that. it has a weird 4-wide AVX thing to go along with its 4-wide SMT, and it's its own specific thing.

hajile2• 4 years ago

Based on Wikipedia's compatibility chart, it sure looks like a minefield to me.

https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512

janwas26• 4 years ago

Except as an application author I can't rely on it being there.

I'm surprised to keep hearing this concern. We can write code once in a vector-length agnostic way, compile it for multiple targets, and use whatever is there at runtime.

p0nce8• 4 years ago

Yes, but if you write consumer software you need to optimize more for the low-spec machines. It is those users that need the speed.

p0nce8• 4 years ago

I sell consumer software and you can get bad reviews from people not having SSSE3 (2008). So not ony I hardly need larger vectors (vs 128-bit) vs memory throughput, but it would take 10+ years before being able to use that. The larger the SIMD vector, the more useless it is in a way that Linus describes.

kierank• 4 years ago

This solves all the problems:

https://blogs.gnome.org/rbultje/2017/07/14/writing-x86-simd-...

janwas26• 4 years ago

hm, nifty - templates for x86 assembly language. Doesn't solve our requirement for portability across SVE/RISC-V, though :)

Aardwolf3• 4 years ago

Seriously, go look at random code on github - I dare you. Very little of it is clearly delineated loops over arrays with high repeat counts. It's not very helpful to be able to vectorize some "search for a value in an array", when people end up using hash tables or other more complex data structures instead of arrays for searching.

Sure, but many parts that matter for performance, the inner loops to optimize, may be like that.

Multimedia encoding/decoding, emulation, ML, matrix multiplications, ...

Tuna-Fish• 4 years ago

Multimedia encoding/decoding,

Done on special-purpose hardware that is >10 times more efficient than any implementation on GP hardware.

emulation,

Only current practical use of AVX-512.

ML, matrix multiplications, ...

Much better to do on the GPU.

In general wide SIMD on CPU has the problem that to make effective use of it, you have to massage your code enough that you are ~95% away from just running it on the GPU anyway, and you can gain much more performance if you do that. The best niche of AVX-512 would have been as the baseline common target for things that also get optimized for GPUs... except that Intel has eliminated this possibility by heavy product segmentation right from the start.

Aardwolf3• 4 years ago

Done on special-purpose hardware that is >10 times more efficient than any implementation on GP hardware.

That locks users in to only the codecs that have been implemented in this particular hardware, and increases hardware size for particular vendor implementations rather than provide common building blocks that many codecs use (DCT, ...)

Much better to do on the GPU.

Yep and that's what I run ML stuff on, not all systems have a GPU available though, and for some applications it's faster to do something immediately on the CPU, than have the overhead of going to/from GPU

In general wide SIMD on CPU has the problem that to make effective use of it, you have to massage your code enough that you are ~95% away from just running it on the GPU anyway,

Better tooling and libraries could help with this imho. Note that the situation for GPU is here also not great at all, since the good tools are locked into 1 vendor.

dahfizz• 4 years ago

you have to massage your code enough that you are ~95% away from just running it on the GPU anyway

Depends on your usecase, I guess. Having AVX-512 on the CPU allows you to just re-implement one critical function to be faster, and let the rest of your code be clean and simple. Communicating with a GPU comes with a large latency penalty that is not acceptable in some use-cases.

mhh__6• 4 years ago

That 5% is pretty big arguably.

sliken2• 4 years ago

Much better to do on the GPU

Oh? Which GPU? iGPU? Discrete? Intel? Nvidia? AMD? From which generation? Using what libraries? Assuming you are running on an x86-64? Or maybe arm? Something else? How are you going to handle underflow/overflow? Did you need IEEE FP support? How many $1000s were you going to spend on hardware to test/verify your code on different GPUs?

Also depends on how big the matricies are, if they are too small and the latency of the GPU isn't worth it, too large and it won't fit. The too small/too large decision depends on which GPU, how many PCIe lanes, and which library.

So not such a simple decision.

hajile2• 4 years ago

you are ~95% away from just running it on the GPU anyway

Vector code with lots of branches absolutely exists. You can run it on a GPU, but because they don't dedicate transistors to OoO, branch prediction, and good prefetchers, the code won't run very well.

jiggawatts2• 4 years ago

There are several chicken & egg problems with this kind of thing.

First, there are only a handful of languages optimised for "structure of arrays" instead of "array of structures" programming (SoA vs AoS). To my knowledge only some array-based languages like J/K/APL, Jonathan Blow's Jai, and Google's Rune.

Why aren't array languages more popular? Because the performance delta has historically not been worth the switching costs.

Why has the performance delta not been big enough? Because AVX-512 and similar architectures haven't been available to the mass market.

Why isn't AVX-512 included in every processor? Because there's not enough software for it yet.

Etc...

Once AVX-512 becomes widespread, languages are developed for highly-parallel architectures with SIMD, then you'll start seeing new applications materialise driving demand.

Dylan168073• 4 years ago

Similar architectures have been available for a plenty of time! 256 bits at once with multiple execution units is a lot of compute power and has been the standard for a decade. Let alone SSE.

singularity20013• 4 years ago

I wonder how wasms proposed vector pipeline operations will relate

frozenport• 4 years ago

These dudes are biased: they don't want to support something they don't use.

Razengan• 4 years ago

The PC dev world in a nutshell.

ak217• 4 years ago

Some of the biggest investments into general purpose Linux performance, reliability, and sustainability were made by people with "specialty engines" that Linus is talking about. The specialty applications often turn out to be important because they push the platform to excel in unexpected ways.

With that said, the biggest practical problem we saw with AVX-512 was that it's so big, power hungry, and dissimilar to other components of the CPU. One tenant turning that thing on can throttle the whole chip, causing unpredictable performance and even reliability issues for everything on the machine. I imagine it's even worse on mobile/embedded, where turning it on can blow out your power and thermal budget. The alternatives (integrated GPUs/ML cores) seem to be more power efficient.

cogman102• 4 years ago

and dissimilar to other components of the CPU

It's not terribly dissimilar from the SSE instructions before it. The biggest weirdness about AVX is it's CPU throttling. Which, IMO, should not be a thing for the reasons you mention.

I imagine that future iterations of AVX will likely ditch CPU throttling. I'm guessing it was a "our current design can't handle this and we need to ship!" sort of solution. (If they haven't already? Do current CPUs throttle AVX-512 instructions?)

I imagine it's even worse on mobile/embedded

Fortunately, x86 mobile/embedded isn't terribly common, certainly not with AVX-512 support.

Kon-Peki3• 4 years ago

I imagine that future iterations of AVX will likely ditch CPU throttling

Already happening:

https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#Dow...

paulmd9• 4 years ago

I imagine that future iterations of AVX will likely ditch CPU throttling. I'm guessing it was a "our current design can't handle this and we need to ship!" sort of solution. (If they haven't already? Do current CPUs throttle AVX-512 instructions?)

exactly what it looks like to me, and it was already fixed in Ice Lake and probably doesn't exist at all in Zen4. Just nobody cares about Ice Lake-SP when Epyc is crushing the server market.

https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre...

and some measurements of Skylake-X displaying the effect, note the voltage swings in particular: https://travisdowns.github.io/blog/2020/01/17/avxfreq1.html

the whole thing looks very much to me like "oops we put not just AVX-512 but dual AVX-512 in our cores and on 14nm it can hammer the chip so hard we need more voltage... guess we're gonna have to stop and swing the voltage upwards".

not only is that less of a problem with higher-efficiency nodes, but the consumer core designs drop the idea of dual-AVX 512 entirely which reduces the worst-case voltage droop as well...

janwas26• 4 years ago

The turbo curves depend on the exact type of Xeon. Were they Bronze or Silver? Those are much worse in that regard.

It is also unclear to me that AVX-512 is necessarily problematic in terms of power and thermal effects. Scalar code on many threads actually uses more energy (dispatch/control of OoO instructions being the main driver) than vector code (running far fewer instructions). The much-maligned power-up phase does not matter if running vector code for at least a millisecond.

I agree dedicated HW can be more power-efficient still, assuming it does exactly what your algorithm requires. And yet they make the fragmentation and deployment issues (can I use it?) of SIMD look trivial by comparison :)

sliken2• 4 years ago

I have some water cooled Xeon Gold (6100 series) chips. Generally they never throttle, unless running a code well known for heavy use of AVX 512.

So despite running many codes that use all cores 100%, using AVX512 is the only way I've seen them throttle.

nomel2• 4 years ago

One tenant turning that thing on can throttle the whole chip,

This wasn't related to AVX-512, it was related to its implementation on that specific, somewhat old, architecture. Current AMD and next generation Intel don't have this limitation, so it shouldn't be considered when thinking of "the future".

dathinab6• 4 years ago

One tenant turning that thing on can throttle the whole chip, causing unpredictable performance and even reliability issues for everything on the machine

This is where ironically where the double pumping of Zen 4 shines, it's less of an problem with it.

fulafel6• 4 years ago

The main thing seems to be that we don't have general purpouse languages in common use that fit SIMD auovectorization in semantics, without carefully "code to the compiler" portable assembly mindset. Even after 25 years of x86 SIMD support in hardware.

It's just not been an aim even in more recent-ish languages like Rust, Swift, Go etc. Partly because of the unworkably hard task of targeting fragmented, undebuggable, proprietary driver/OS programming interfaces from POV of portable languages.

cmrdporcupine6• 4 years ago

Exactly. But also he inadvertently actually touches on this in the post. Talking about people using hash tables and so on. I think if people had the right guidance from language and libraries and from education, they would choose array-based patterns more often. They're often in fact simpler to write.

But many languages and common practices push people in a highly incremental piece by piece approach. I'd argue we're basically taught to think in terms of loops, from our earliest days of programming. We could just as easily be taught to think of things in terms of bulk arrays. The win in programming this way is extremely high.

I was very disappointed to find the SIMD story in Rust so underdeveloped.

Yes, people are not running weather forecasting on their home computers. But they're playing video, synthesizing audio, doing speech recognition, etc. all the time. And more of this ever day.

And if there's any chance of rescuing the uses of ML from a hellish privacy violating landscape worth of a Butlerian Jihad... it would by making sure most of this type of computation was done locally on-device.

Guess APL has the last laugh.

acdha3• 4 years ago

Yes, people are not running weather forecasting on their home computers. But they're playing video, synthesizing audio, doing speech recognition, etc. all the time. And more of this ever day. > And if there's any chance of rescuing the uses of ML from a hellish privacy violating landscape worth of a Butlerian Jihad... it would by making sure most of this type of computation was done locally on-device.

I think SIMD support is useful too but most of the examples you gave are things which are handled on-device now using dedicated hardware. Video playback has been hardware accelerated for ages, ML acceleration is multiple hardware generations in, etc.

bsder• 4 years ago

Exactly. But also he inadvertently actually touches on this in the post. Talking about people using hash tables and so on. I think if people had the right guidance from language and libraries and from education, they would choose array-based patterns more often. They're often in fact simpler to write.

A) The vast majority of programmers have zero idea about any of this. They want a key-value store and have no idea about the underlying implmentation. And that's okay. You can write an awful lot of useful software without understanding Big-O notation.

B) Pointer chasing has been dogshit on modern architectures for quite a while, and most people have no idea. Look at all the shit Rust gets for being annoying to implement linked lists which are a garbage data structure on modern CPUs.

C) Array-of-struct to struct-of-array type transformations are just starting to get some traction in languages. Entity component systems are common in games but haven't really moved outside of that arena. These are the kind of things that vectorization can go to town on but haven't yet moved to mainstream programming.

D) Most of this stuff is contained inside libraries anyway. So, only a few people really need to care about this.

candrewlee14• 4 years ago

Check out Zig’s `Vector` type. I don’t think I’ve seen SIMD built as elegantly into any other systems language.

anonymoushn8• 4 years ago

I use Zig every day and I consider Zig's vector type and SIMD support a good stepping stone to the destination of just writing the intrinsics.

Competing efforts including std::simd or Highway are either unusable because they don't expose pshufb or more difficult to use than the tools they are abstracting away.

int_19h2• 4 years ago

C# has Vector<T> as well, although it's a bit more low-level.

sorenjan• 4 years ago

Are there any libraries that allow me to write different versions of the same function (AVX-512, AVX2, SSE, etc) and then automatically choose the best one that the system supports at runtime? Or maybe even better, the compiler generates multiple versions for me.

In other words, one binary that takes advantage of new instructions but still runs on older hardware. It doesn't really have to be older either, plenty of brand new CPUs doesn't support AVX-512.

bobowzki2• 4 years ago

Libvolk works exactly like you describe.

https://www.libvolk.org/

nynx• 4 years ago

Rust has built-in support for this, I believe.

3836293648• 4 years ago

Rust's native support for this is super verbose and repetetive. There's a macro library that can deal with all the unsafe and version picking for you though.

curiousmindz• 4 years ago

In C#, you can write (pseudo-code): if (AVX2) { ... } else if (SSE) { ... } else ...

Then, when you run the program, the JIT will pick the first supported option and eliminate the other ones.

janwas26• 4 years ago

Cool, I wasn't familiar with libvolk. It seems to be a collection of prewritten kernels, so it only helps if the function you want to write is among them.

github.com/google/highway seems to be closer to what is requested here. It provides around 200 portable intrinsics using which you can write a wide variety of functions (only a single implementation required). Disclosure: I am the main author.

mhh__6• 4 years ago

GCC can do it for you with autovectorization.

Doing the feature detection is fairly trivial. If they don't run the invalid instructions don't error.

astrange• 4 years ago

Autovectorization basically doesn't work. The effort to test it properly (that it got vectorized the way you expect or at all) is more maintenance than writing it yourself.

If you insist on abstractions, autoscalarization (the opposite approach) would be better, which is kind of how Fortran works… but I unironically recommend just writing assembly like ffmpeg does.

fmajid• 4 years ago

http://maskray.me/blog/2021-01-18-gnu-indirect-function

eesmith3• 4 years ago

https://gcc.gnu.org/wiki/FunctionMultiVersioning ?

And in clang - https://releases.llvm.org/7.1.0/tools/clang/docs/AttributeRe... .

I've never used them.

Don't know about other compilers.

fsfod• 4 years ago

MSVC recently added there own version[1] of it controlled with a [[msvc::dispatch]] attribute thats needs an experimental compiler flag I think still.

1: C++ Function Multiversioning in Windows https://www.youtube.com/watch?v=LTM1was1dTU

wazoox• 4 years ago

Oooh, that's exactly the problem I had these days. The Firefox extension "Firefox translations" uses a tool (bergamot-translator) which requires SSE4 (or is it SSE3.1?) to run at all, so on all slightly old machines it fails miserably with a weird error. (90% of the machines I see around are 5 to 10 years old, work perfectly and don't need to be replaced, thank you very much, save money, think of the planet, this sort of things).

Rafuino• 4 years ago

Generally speaking the vendor libraries have dynamic dispatch support, which can identify which functions are available on a CPU and then deploy to the best at runtime. Intel got into some hot water for having their dispatch hurt performance on AMD CPUs, but it seems that's been fixed. Their IPP-Crypto and AMD's AOCL-Cryptography libraries both support dynamic dispatch these days, for example.

paulmd9• 4 years ago

End result: go look for AVX-512 benchmarks. You'll find them. And then ask yourself: how many of these are relevant to what I bought my PC/mac/phone for?

... do you guys not do JSON? cause we're doing json.

https://www.phoronix.com/review/simdjson-avx-512

or how about ARM emulation? anybody doing ARM development on x86? do you guys have phones, maybe?

https://www.reddit.com/r/emulation/comments/lzfpz5/what_are_...

seriously, linus is wrong on this issue, and he just keeps digging. lose the ego and admit you were spewing FUD.

avx-512 had legit problems in Skylake-SP server environments where it was running alongside non-AVX code (although you could segment servers by AVX and non-AVX if you wanted).

None of that has been true for any subsequent generations where the downclocking is essentially nil, and the offset has always been configurable for enthusiasts who get more buttons to push.

It also was never a "one single instruction triggers latency/downclocking" like some people think, it always took a critical mass of AVX-512 instructions (pulling down the voltage rail) before it paused and triggered downclocking, and "lighter" operations that did less work would trigger this less.

https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre...

AMD putting AVX-512 in Zen4 sealed the deal. AMD isn't making a billion-dollar bet on AVX-512 because of "benchmarksmanship", they're doing it because it's a massive usability win and a huge win for performance. Over time you will see more AVX-512 adoption than AVX2 - because it adds a bunch of usability stuff that makes it even more broadly applicable, like op-masking and scatter operations, and AVX is already very broadly applicable. Some games won't even launch without it anymore because they use it and they didn't bother to write a non-AVX fallback.

It's also a huge efficiency win when used in a constrained-power environment - for your 200W-per-1RU power budget, you get more performance using AVX-512 ops than scalar ones. Or you can run at a lower power budget for a fixed performance target. And you're seeing that on Zen4, when it's not tied to Intel's 14nm bullshit (and really it's probably also apparent on Ice Lake-SP if anybody bothered to benchmark that). That's also a huge win in laptops too, where watts are battery life. Would anybody like more efficiency when parsing JSON web service responses when they're on battery? I would.

Sorry Linus it's over, just admit you're wrong and move on.

chrchang5232• 4 years ago

We'll see.

There's now another game in town, exemplified by Apple's switch to ARM chips for its Macs. I'll keep an eye on the AVX2 vs. AVX-512 performance gap on Zen4+, but my working hypothesis is that my SIMD-handcoding time will be better spent on improving ARM support than upgrading AVX2 code to AVX-512 for the foreseeable future.

janwas26• 4 years ago

How about porting to Highway, that gets you AVX-512, NEON and SVE(2) from a single rewrite :)

paulmd9• 4 years ago

I'm a huge simp for M1 too (and there's SVE there too). Yeah for client stuff if you can get people to just buy a macbook that's the best answer right now, if that does their daily tasks. Places need to start thinking about building ARM images anyway, for Ampere and Graviton and other cost-effective server environments if nothing else. If you are that glued at the hip to x86 is time to look at solving this problem.

Apple's p-cores get the limelight but the e-cores are simply ridiculous for their size... they are 0.69mm^2 vs 1.7mm2 for gracemont, excluding cache. Gracemont is Intel 7, so it's a node behind, but, real-world scaling is about 1.5-1.6x between 5nm and 6nm so that works out to about 1.1mm2 for Avalanche if it were 7nm, for equal/better performance to gracemont, at much lower power.

https://www.reddit.com/r/hardware/comments/qlcptr/m1_pro_10c...

Sierra Forest (bunch of nextmont on a server die, like Denverton) looks super interesting and I'd absolutely love to see an Apple equivalent, give me 256 blizzard cores on a chiplet and 512 or 1024 on a package. Or even just an M1 Ultra X-Serve would be fantastic (although the large GPU does go unutilized). But I don't think Apple wants to get into that market so far from what I've seen.

(tangent but everyone says "Gracemont is optimized for size not efficiency!" and I don't know what that means in a practical sense. High-density cell libraries are both smaller and more efficient. So if people meant that they were using high-performance libraries that would be both bigger and less efficient (but clock higher). If it's high density it'd be smaller and more efficient but clock lower. Those two things go together. And yes everyone uses a mix of different types of cells, with high-performance cells on the timing hot-path... but "gracemont is optimized for size not efficiency" has become this meme that everyone chants and I don't know what that actually is supposed to mean. If anyone knows what that's supposed to be, please do tell.)

(also, as you can see from the size comparison... despite the "it's optimized for size" meme, gracemont still isn't really small, not like Blizzard is small. they're using ~50% more transistors to get to the same place, and it's almost half the size of a full zen3 core with SMT and all the bells and whistles... I really think e-cores are where the music stops with the x86 party, I think i-cache and decoders are fine on the big cores but as you scale downwards they start taking up a larger and larger portion of the core area that remains... it is Amdahl's Law in action with area, if i-cache and decoding doesn't scale then reducing the core increases the fraction devoted to i-cache/decoding. And if you scale it down then you pay more penalty for x86-ness in other places, like having to run the decoder. And you have to run the i-cache at all times even when the chip is idling, otherwise you are decoding a lot more. It just is a lot of power overhead for the things you use an e-core for.)

fulafel6• 4 years ago

JSON acceleration and handful of other peripheral spot improvements aren't going to yield big whole application speedups in most apps.

Using hand coded accelerated SIMD kernels in specific places like JSON codec hits Amdahls' law[1]. The archievable whole app speedup is going to be low in most cases unless you get pervasive performant compiler generated SIMD code throughout your code, done by JITs of managed languages etc.

[1] https://en.wikipedia.org/wiki/Amdahl%27s_law

paulmd9• 4 years ago

if there were big general-case speedups still possible from auto-vectorization that juice would have been squeezed already, even if it was only possible at runtime the core would watch for those patterns and pull those onto the units.

it's like the old joke about economists: an economist is walking down the street with his friend, the friend says "look, a 20 dollar bill laying on the ground!" and bends to pick it up. But the economist keeps walking, and says "it couldn't be, or someone would already have picked it up!". That joke but with computer architecture.

we are inherently talking about a world where that's not possible anymore, that juice has been squeezed. Throwing 10% more silicon at some specific problems and getting 2.5-3x speedup (real-world numbers from SIMD-JSON) is better than throwing 10% more silicon at the general case and getting 1% more performance. If 10-20% of real-world code gets that 3x speedup (read: probably 2x perf/w) that's great, that's much better than the general-case speedups!

ilyt7• 4 years ago

... do you guys not do JSON? cause we're doing json.

Go ahead, benchmark how much of JSON parsing consists of runtime of the entire pipeline between browser and server. I bet it is in low single digit unless you're literally doing "deserialize, change a variable, serialize"

Yes, it is use case that works, but unless you're keeping gigabytes in JSON and analyzing it, it isn't making most users job any faster, low single digits increases at best.

It's also a huge efficiency win when used in a constrained-power environment - for your 200W-per-1RU power budget,

Which is not the use case Linus was talking about ? He didn't argue it doesn't make sense in those rare cases where you can optimize for it and GPUs are not an option.

rashkov• 4 years ago

If I understand correctly, this discussion is specifically about auto-vectorization, which is when the compiler automatically re-writes your code to use highly parallel (and performant) vectorized assembly instructions.

Linus acknowledges that vectorized instructions are valuable, so his rant seems to be specifically about auto-vectorization:

Yes, yes, yes, you can use it in random places. It's not only some JSON parsing library, you'll find it in other libraries too. I suspect the most use it gets in many situations is - drum roll - implementing memmove/strchr/strcmp style things. And in absolutely zero of those cases will it have been done by the compiler auto-vectorizing it.

In other words.. the people writing foundational parsing libraries (json / strchr / strcmp) are writing their vectorized instructions by hand.

cmrdporcupine6• 4 years ago

Well, also in other words, extant languages and CS education is failing people and they're writing these things the wrong way.

kolbe7• 4 years ago

I use intrinsics by hand all the time. It's very easy to make a problem too complicated to autovectorize. And even if you do get it to autovectorizie, it's not exactly future proof against compiler changes.

mhh__6• 4 years ago

Note that if you don't write with simd in mind you'll probably write code that cannot ever be autovectorized.

SIMD lexing/parsing could mean being less precise with error messages and so on.

anonymoushn8• 4 years ago

Linus is correct that auto-vectorization is basically useless, but I have some minor disagreements. JSON is obviously not the only format or protocol that can be parsed using SIMD instructions. In general computers spend a lot of time parsing byte streams, and almost all of these could be parsed at gigabytes per second if someone familiar with the protocol, the intel intrinsics guide, and the tricks used in simdjson spent an afternoon on them. It's sort of unfortunate that our current software ecosystem makes it difficult for many developers to adopt these sorts of specialized parsers written in low-level languages and targeting specific vector extensions.

If someone wants to use a hash table or TLS, they may find AES-NI useful.

jackmott42• 4 years ago

One of the specific design properties of AVX-512 is to make auto-vectorization less useless. If Intel had committed to more predictable and wide support, we might be seeing the fruits of that now. Whether it would have been a "Game changer" or just a slight improvement, I don't know, but even slight improvements in cpu performance are worth it these days.

dathinab6• 4 years ago

"Game changer"

made me smirk, as games is what likely would have profited from it

games are also what tends to be some of the more havy compute applications outside of servers

through it likely would be mostly just some code here and there deep withing the game engine

but then this is also an example where auto-vectorization can matter as it means writing platform independent code which happens to be fast on the platforms you care about most but also happens to compile and run with some potential (but only potential) performance penalty when porting to other platforms. Not only makes this porting easier, it also for some games might be fine if the engine is a bit slower. Main drawback is that writing such "auto vectorized" code isn't easy and can be brittle.

eklitzke• 4 years ago

Part of his argument though was that for most cases where you're parsing JSON (or some other protocol), the parsing time is small compared to the amount of time you spend actually doing things with the data. And for most of the examples where the code is mostly parsing (say, an application doing RPC routing, where the service just decodes an RPC and then forwards it to another host) you can often use tricks to avoid parsing the whole document anyway. For example, if you have a protocol based on protobuf (like gRPC) you can put the fields needed for routing in the lowest number fields, and then use protobuf APIs to decode the protobuf lazily/incrementally; this way you will usually just need to decode the first (or perhaps the first few) fields, not the entire protobuf.

Making code faster and more efficient is always better, it's just that you might not see much improvement in end-to-end latency times if you make your protocol parsing code faster.

anonymoushn8• 4 years ago

I think this is maybe true in practice because business logic is usually written really badly, but a lot of the time the work that must be done is ~linear in the size of the message.

cat_plus_plus• 4 years ago

It's called GPUs and TPUs? Turned out it's more practical to design hardware for known problems comprising big parts of computing workload than "build it and they will come" Itanium/AVX 512 plans. Also, GPUs have been repurposed for science, machine learning and mining crypto and, as additional useful applications are discovered, new hardware can support them more effectively.

alkonaut2• 4 years ago

The problem with AVX isn’t that compilers find it difficult to emit enough of them or that people don’t often work on problems that are easily transformed to simd at all.

It’s enough if a small number of libraries use these instructions, and a lot larger number of programs can benefit.

The problem with AVX-512 in my opinion is the sparse hardware support, power/thermal problems and noisy neighbor phenomenon.

To be useful simd instructions need to be widely available and not have adverse power effects such as making other programs go much slower.

mhh__6• 4 years ago

Be warned that a lot of the power issues are basically Chinese whispers from processors that are getting on for a decade old now.

There are still measurable effects but not like on skylake, to my knowledge at least.

alkonaut2• 4 years ago

In the latest iteration, they removed AVX-512 iirc. So was there even as window when it worked well and shipped on a majority of new CPUs?

jmoss20• 4 years ago

No, 512 still fries skylake.

gok• 4 years ago

Firstly, there are in fact a lot of problems that benefit from parallelization, but are not quite wide enough to justify dealing with (say) a GPU. For example, even in graphics you often have medium-sized rasterization tasks that are much more convenient to run on CPUs. Audio processing is similar. Often you have chunks of vectorizable work inter-mixed with CPU-friendly pointer chasing, and the ability to dispatch SIMD work within nanoseconds of SISD work is very helpful.

Secondly, AVX-512 (and Arm's SVE) support predication and scatter/gather ops, both of which open the door to much more aggressive forms of automatic vectorization.

oconnor663• 4 years ago

AVX-512 is fabulous for cryptography, especially software-oriented algorithms like ChaCha and BLAKE3 that lean on general-purpose SIMD features rather than dedicated hardware acceleration. Most of the added value is just the larger registers, but the native bit-rotate instructions are also great. (But agreed with Linus that autovectorization isn't very helpful for this stuff.)

Havoc2• 4 years ago

Requiring special hardware to avoid other more different special hardware seems a little questionable

dathinab6• 4 years ago

Not if a few docent different "special cases" all can use the same special hardware instead of a few docent different hardwares.

And the thing with a lot of this "special cases" is that while most code doesn't write them a lot of systems have them running somewhere.

raggi• 4 years ago

trigger warning: possible rust advocacy. I hope the warning can help some folks relax, and stave off the usual repeating meta discussion.

I've looked at a lot of rust generated outputs from iterator usage in godbolt. If you have tried this (and if not, go look), you'll know that you almost certainly never want to look at debug output for iterator usage.

In release mode builds of rust programs using the iterator package you'll often see SIMD. The author doesn't need to do special things to get this, the combination of aggressive inlining and encouragement to use owned and intrusive structures goes a long way to making this viable more often in practical code - so much so that I see it working _a lot_. It's not a magic bullet, and I'm not advocating that it's some be-all end-all, but it works well there.

If you'd like to see an example, search for "godbolt rust iterator" and look at the optimized output. I'm sure one of the near top results will do.

joshfee• 4 years ago

Rust or otherwise, the more semantics that can be expressed in a language, the more optimizations that can be done for you (at the cost of lacking full control in certain scenarios).

nitwit005• 4 years ago

A lot of people in C++ land probably just have the vector optimizations off. If you use clang, you will get vector instructions at -O2. Gcc seems to need -O3, or the specific flags enabled.

fulafel6• 4 years ago

I think the interesting question is that how big a speed delta does a typical Rust app enjoy when built with autovectorization enabled vs not.

You get points for going faster, not for sometimes emitting the instructions.

protastus• 4 years ago

SSE, NEON and AVX are fundamental for anything reducible to matrix and vector arithmetic (e.g., signal processing and ML inference on the CPU).

It is hard to find simple enough code to vectorize at all.

It's hard to find code simple enough for the compiler to efficiently auto-vectorize.

Anything that reduces to GEMM will parallelize in practice extremely well, and there are many excellent libraries with SIMD support (MKL, BLAS, ATLAS, Eigen, etc.). However, these libraries rely on kernels carefully written by experts and benchmarked extensively over decades. They're not the output of running naively written code through a super smart compiler.

All of this is extremely relevant to what you bought your PC or phone for. It's also not in the kernel, and therefore Linus seems to be unaware of their pervasiveness and utility.

andromeduck• 4 years ago

Wish RWT had better UI.

vgatherps• 4 years ago

I'm freshly out of the weeds of implementing some fixed-point decimal parsers, and only found ~1 cycle that I could save (at best) per iteration using avx512.

I largely agree with linus here. It's still a huge pain to do anything non-trivially-parallelizable with simd even in an avx512 world

1. I have NEVER been satisfied with the results of a compiler autovectorization, outside of the most trivial examples. Even when they can autovectorize, they fail to do basic transformations that greatly unlock ILP (to be fair these aren't allowed by FP rules since it changes operation ordering)

2. Gather/scatter still have awful performance, worse than manually doing the loads and piecemeal assembling the vector. The fancy 'results-partially-delivered-in-order-on-trap' semantics probably don't help. Speeding up these operations would instantly unlock SO much more performance (see my username)

3. The whole 'extract-upper-bit-to-bitmask' -> 'find-first-set-in-bitmask' really deserves an instruction to merge the last two (return index of first byte with uppermost bit set, etc). It's such a core series of instructions and ime often the bottleneck (each of those is 3 cycles)

4. The performance of some of the nicer avx512 intrinsics leaves a lot to be desired. Compress for example is neat in theory but is too slow for anything I've tried in real life. Masked loads are another example, cool in theory but in practice MUCH faster to replicate manually.

5. It still feels like there's a ton of really random/arbitrary holes in the instruction set. AVX512 closed some of these but introduced many more with the limited operations you can do on masks and expensive mask <-> integer conversions.

gonzo• 4 years ago

AVX 2 and AVX 512 are very good for VPN transforms.

10gbps on a single core.

singularity20013• 4 years ago

Abstract vectorization instructions in wasm will make life a lot easier

https://github.com/WebAssembly/flexible-vectors/blob/main/pr... great proposal!

Mapping to whatever hardware is available as some sort of micro library

AVX 512 will be the future

Related Stories