janwas

Author/Highway TL here. Happy to discuss.

Impressive result. Congrats!

Oops, the final T got cut off somehow, sorry about that.

https://gcc.godbolt.org/z/KM3ben7ET

Any suggestions for improvement? We went through >5 iterations of the dispatching and I am fairly confident this is about as good as it gets in current C++. I suppose "macro hell" is a matter of taste. Objectively, we have six dispatch related macros in the example: https://gcc.godbolt.org/z/KM3ben7E The ~two dozen lines of boilerplate are generally copied from an example. But why multi-file?

Working on one together with fastcode.org :)

To be clear, "better abstractions" here seems to mean macros for assembly language. To each their own.

What bothers me is advocating for this, or denigrating more generally useful alternatives, without mentioning the very narrow niche where this sits.

Video codecs only change every few years. This makes it more worthwhile/feasible to spend eng time on a few kernels.

Even then, not supporting SVE (you don't, right?) gives less incentive for the Arm CPU ecosystem to invest in it, helping keeping us stuck in the NEON local minimum. Not ideal :/

Correction (typo): Z13 lacks fp32.

Oh, interesting :) I meant Fastcode.org.

Is this a good faith reply? The particular abstraction we built, and is being discussed, is manifestly and obviously not a lowest common denominator. Looks like you are deploying a second straw man, that of zero cost. In other comments here I acknowledge a cost to intrinsics.

?? Where did you see mention of AI?

Thanks for sharing. The first link seems non public indeed. I can imagine there is some compile issue we could reasonably fix, with the help of someone who has Z13 access. Please encourage them to raise an issue. I will be back on May 26. After that, it should at least be able to use the scalar fallback. The issue with Z14 is that it lacks fp32 support. Would their usage be integer only?

Fair point. If it helps, our security team has called Highway critical infrastructure and helped to harden the repo. The flip side of standardization is that it would be much harder and slower to add ops as the need arises, which we do regularly.

:) I figure there is always something left to improve. For some kernels which really want to keep 30+ live registers, the compiler might not do as good a job as careful manual tuning, so intrinsics can have a bit of a cost. But I also figure optimization time is limited, so better to get 90% of several kernels rather than one to 99%.

Yes, the EMU128 target is scalar only, with for loops. This is a fun way to see how well autovectorization works, with the same source code. That works on any CPU. Curious which projects have such concerns, any link?

In such discussions, whenever you mention abstractions are universally "pretty poor", to the extent anyone is listening, I think this hyperbole can do real damage. Maybe it prevents people from getting relevant performance gains, even if not 100% of the optimum, which is anyway unattainable. And what is the alternative? Not many projects can afford to hand write intrinsics for all platforms. And are you aware that Highway is basically a thin wrapper over intrinsics, which you can still drop down to where it helps?

This works today :) Highway provides such an abstraction for arbitrary vector lengths and maps them to intrinsics. All on the library level, no need to wait years for compiler or language updates.

:) I agree a tutorial would be helpful. We are working on one with Fastcode.

Have you considered our Highway library? Runtime dispatch need not be a PITA :) It's basically portable intrinsics, and a much more complete set (>300) than the ~50 in std.

Highway TL here. I agree with the main points, with a few clarifications:

tag-dispatched free functions like hn::Mul(d, a, b)

We only require tags for certain ops, mainly memory, casts and reduction; not arithmetic. Operator overloading is supported but until recently compilers didn't allow that for SVE vectors.

It’s a Google project with Google-scale maintenance, but the bus factor is real — the core development is driven by a small team

We have 101 contributors, including 14 current or former Googlers in several teams.

being length-agnostic means you can’t easily express fixed-width algorithms that depend on knowing the vector size at compile time, which is common in cryptography and codec work

We explicitly support fixed-length 128-bit vectors, acknowledging that these are common and important.

Looks like the ratification plan for Zvzip is November. So maybe 3y until HW is actually usable? That's a neat trick with wmacc, congrats. But still, half the speed for quite a fundamental operation that has been heavily used in other ISAs for 20+ years :(

Great that you did a gap analysis [1]. I'm curious if one of the inputs for that was the list of Highway ops [2]?

[1] https://gist.github.com/camel-cdr/99a41367d6529f390d25e36ca3... [2]: https://github.com/google/highway/blob/master/g3doc/quick_re...

(Personal opinion) I get the impression that RISC-V-related discussions often lack of awareness of prior work/alternatives. A large amount of (x86) software actually uses our Highway library to run on whatever size vectors and instructions the CPU offers.

This works quite well in practice. As to leaving performance on the table, it seems RVV has some egregious performance differences/cliffs. For example, should we use vrgather (with what LMUL), or interesting workarounds such as widening+slide1, to implement a basic operation such as interleaving two vectors?

:D Your code was nicely written and it was a pleasure to port to SIMD because it was already very data-parallel.

Gemma.cpp has nested thread pools, one per chiplet, and one across all chiplets. With such core counts it is quite important to minimize any kind of sharing, even RMW atomics.

performance of general solutions without using SIMD, is good enough too, since all of which will eventually dump right down to the uops anyway, with deep pipeline, branch predictor, superscalar and speculative execution doing their magics altogether

A quick comment on this one point (personal opinion): from a hyperscalar perspective, scalar code is most certainly not enough. The energy cost from scheduling a MUL instruction is something like 10x of the actual operation it performs. It is important to amortize that cost over many elements (i.e. SIMD).

Wow, that number requires STRONG caveats, lest it be called out as completely false. Take away the tensor cores (unless you only do matmuls?), and an H100 has roughly 2x as many f32 flops as a Zen5 CPU, which is considerably cheaper. I suspect brute force HW/algorithms are not going to age well: https://www.sigarch.org/dont-put-all-your-tensors-in-one-bas... (/personal opinion)

CPU-time would over-emphasize regions where many threads are running, right? I find wall-time useful for finding serial regions that aren't yet parallelized.

More detail here: https://github.com/dvyukov/perf-load. We recently implemented the same idea without requiring context-switch events: https://github.com/google/highway/blob/master/hwy/profiler.h...

:) Yes indeed, it's about 500 LOC in https://github.com/google/highway/blob/master/hwy/ops/generi....

I appreciate your efforts to nudge readers towards SoA data structures and varying SIMD widths. FWIW I have observed that advice is more effective if communicated with some kindness.

Thanks for expanding on your viewpoint.

Why would writing an optimizing compiler qualify as territory for directly writing SIMD code, but anything else is off the table?

I understood "directly writing" to mean assembly or even intrinsics. In general, I would advise not touching intrinsics directly, because the intrinsic definitions themselves have in several cases had bugs. Here's one AVX-512 example: https://github.com/google/highway/commit/7da2b760c012db04103....

When using a wrapper library, these can be fixed in one spot, but every direct user of intrinsics has to deal with it themselves.

it's extremely easy to write your own wrappers (like I did) and not take a dependency. A good trade IMO

I understand wanting to reduce dependencies. The tradeoff is a bit more complex: for example many readers would be familiar with Highway terminology. We have also made efforts to be a lightweight dependency :)

doing it yourself as a learning exercise has value.

Understandable :) Though it's a bit regrettable to tie your user code to the library prototype - if used elsewhere, it would probably have to be ported.

The fact is, the library code is super fucking boring.

True for many ops. However, emulating AES or other complex ops is nontrivial. And it is easy to underestimate the sheer toil of keeping things working across compiler versions and their many bugs. We recently hit the 3000 commit mark in Highway :)

You also get the automatic support for newer instructions (and multiversioning) with a wrapper library such as our Highway :)