Measuring energy usage: regular code vs. SIMD code

https://lemire.me/blog/2024/02/19/measuring-energy-usage-regular-code-vs-simd-code/

by ashvardanian • 2 years ago

62 21 2 years ago

jsheard2• 2 years ago

It makes intuitive sense that going wide in SIMD uses less power than going wide over cores, because extra SIMD lanes take up much less silicon area than adding another whole core. If they didn't then we wouldn't bother with SIMD, we'd just make more cores and save ourselves the hassle of juggling two different types of parallelism.

GPUs take this even further by going even wider, with each "core" typically executing 1024-bit SIMD operations (i.e. 32x FP32, 64x FP16, etc). CPUs have roughly settled at 128-bit (most ARM) or 256-bit (most x86) with a little bit of 512-bit (x86 with AVX512) which grew out of Intels earlier aborted attempt at making a dedicated GPU.

dist-epoch• 2 years ago

Not only that, but you need to decode the SIMD instruction only once, and not N times. Same with other stages of the pipeline.

sakras• 2 years ago

Yep, I've heard that actually something like 80% of the energy of the CPU is used by the front-end! Beyond decoding, you also have all the energy spent on scoreboarding data dependencies and reordering the instructions and scheduling to execution ports. If you can do that 8x less as in the case for SIMD, that's a huge win!

andy99• 2 years ago

There's a talk by a Nvidia scientist I just saw where he talks about compute efficiency in terms of flops/joule and explicitly mentions vectorized instructions as one of the energy savers. I'll see if I can find it.

Edit: found it (I think) I can't remember where in the presentation but he does mention energy efficiency as a proxy for performance. https://m.youtube.com/watch?v=kLiwvnr4L80

Edit 2: about 26:40 he starts talking about energy use in the context of performance.

01HNNWZ0MV43FF• 2 years ago

I guess it also makes sense if you figure that dedicated hardware for things like AES, video codecs, etc. are similar to SIMD

zokier• 2 years ago

if your tasks complete much faster, you are likely to use less power, even if you are using more energy per unit of time.

isn't this another formulation of the race-to-idle thinking?

ashvardanian• 2 years ago

The 3.5x energy-efficiency gap between serial and SIMD code becomes even larger when

A. you do byte-level processing instead of float words;

B. you use embedded, IoT, and other low-energy devices.

A few years ago I've compared Nvidia Jetson Xavier (long before the Orin release), Intel-based MacBook Pro with Core i9, and AVX-512 capable CPUs on substring search benchmarks.

On Xavier one can quite easily disable/enable cores and reconfigure power usage. At peak I got to 4.2 GB/J which was an 8.3x improvement in inefficiency over LibC in substring search operations. The comparison table is still available in the older README: https://github.com/ashvardanian/StringZilla/tree/v2.0.2?tab=...

Havoc• 2 years ago

Anybody know whether the same is true for gpu vid decode va cpu?

SomeoneFromCA• 2 years ago

everyone who has early batches of Alder Lake should try AVX512-FP16. Extremely fast for AI tasks.

BenoitP• 2 years ago

Please look at these number with the following grain of salt, when optimizing programs. They are trumped by cache usage patterns [1]:

    Loading (fig IV)
    
    Location    Energy (pJ = pico Joules)
    L1          64   pJ/Byte
    L2          121  pJ/Byte
    L3          254  pJ/Byte
    RAM         1250 pJ/Byte
    
    Adding integers with SIMD (fig VI)
    428 pJ/op, for 8 Byte/op; this means: 
    53 pJ/byte

So it takes actually 20 times to fetch data from RAM than to add it to something! And most often this is also the source of latency. Generally energy is linear to the distance signal has to travel, and that's the same for latency.

That's why successful data structures are sized a tiny bit under L1/L2 sizes! (BTree chunks, ring buffers).

If you've been following hardware, it's all about putting RAM closer to compute at the moment with chiplets at the moment.

[1] https://tu-dresden.de/zih/forschung/ressourcen/dateien/abges...

Measuring energy usage: regular code vs. SIMD code

Related Stories