Failing to reach DDR4 bandwidth

https://unum.cloud/post/2022-01-29-ddr4/

by ashvardanian • 4 years ago

101 71 4 years ago

inetknght2• 4 years ago

The spreadsheet showing a bunch of different approaches:

Attempt Bandwidth Max Bandwidth Saturation Time to Code
Parallel STL 87 GB/s 204 GB/s 42.6% 1m
Best CPU run 122 GB/s 204 GB/s 59.8% 60m
Thrust 743 GB/s 936 GB/s 79.4% 1m
Custom CUDA 817 GB/s 936 GB/s 87.3% 30m
CUB 879 GB/s 936 GB/s 93.9% 5m

Then, benchmarks output:

8,115,337,000,378 instructions # 0.18 insn per cycle # 4.71 stalled cycles per insn (83.33%)
1,820,092,697,347 branches # 168.291 M/sec (83.33%)

It's not clear to me which approach represents the output from the benchmarks. But that's a _lot_ of branches for the number of instructions executed. I suggest that might be a reason that the CPU didn't reach high RAM throughput.

ashvardanian13• 4 years ago

The numbers are for the last approach, committed to the public repo. I agree that it's a lot of branches, but expectedly so. Only 0.13% of all branches were missed, so the speculative execution almost always works. Even manual 4x unrolling didn't help.

inetknght2• 4 years ago

Indeed? I wonder if you've tried different memory performance settings in your BIOS (which "might" invalidate your warranty) or different memory modules altogether

323• 4 years ago

On the other hand, high branch usage is the ideal scenario for using CPUs vs GPUs.

dragontamer10• 4 years ago

Not necessarily.

GPUs are outstandingly good at uniform branches. Only divergent branches are GPUs bad at.

jhokanson3• 4 years ago

It is not exactly clear to me what is going on with threads (I guess you are using all of them?). I haven't done too much in this space but anecdotally I've had better luck if my summation is explicitly split into sub-summation tasks. It is not clear if that is being done here. It looks like a single summation loop that the author is expecting the computer to magically split across multiple threads. I'd be interested in seeing what this looks like if instead the task were to add chunks of the original dataset into results per thread (e.g, first 8000 samples on first thread, next 8000 on 2nd thread, etc.), with a final accumulation loop across all threads. Again, the author may be trying this and this is not my area of expertise but I've had decent luck saturating the memory bus with a similar approach.

ashvardanian13• 4 years ago

Here is the source and the threads: https://github.com/unum-cloud/ParallelReductions/blob/fd16d9...

OFC we don't expect the compiler to instantiate them for us, it's not OpenMP :) That one we covered in previous articles. OpenMP gave us about 50 GB/s with all cores enabled and 80 GB/s with part of them disabled.

jhokanson3• 4 years ago

My c++ is not great (so it is hard for me to tell what is going on) and I'm used to OpenMP where my understanding has always been that you tend to get a single thread per processor (or per hyper-thread) -- not sure if that is guaranteed with the way your code is laid out? Perhaps it really is a NUMA issue as others suggest. I will note that one other variation I had (as it looks like you are already splitting across threads) is that the chunk sizes were actually smaller than the # of threads which meant a faster thread would take more chunks rather than waiting on the slowest thread. Good luck!

a_t48• 4 years ago

Is there an advantage to using taskflow for parallel for, if you already have another threadpool implementation? I recently removed taskflow in a project that was only being used for a parallel for loop (as part of a larger refactor, the code had a number of issues...), and I'm wondering if that was a mistake now that I see that pattern somewhere else. :)

hinkley2• 4 years ago

Doug Lea of Java Memory Model and concurrency note went pretty far down this rabbit hole. Not only do you use separate counters/queues per thread/core, but you also put empty space around them so that you don't accidentally share cache lines. I don't know what they do now, but at the time some of the data structures in that library used arrays where only every 8th or 16th entry is used to avoid two cores trying to read from the same cache line.

Typically allocating a separate data structure per actor also accomplishes this as a happy accident. If the thread does the allocation, then it has a better chance of being in the right bank as well.

ashvardanian13• 4 years ago

Yes, thats needed when you have counters in global memory. In that case, instead of just having vector<double> you would put each double into a stricture aligned to 64 byte addresses. Here all the counters are on local stack, so that trick unfortunately wont help

dragontamer10• 4 years ago

GPUs are in fact easier to reach high performance these days IMO. Sure, you gotta learn how to stride memory and possibly do it across workgroups, but those GPUs really have patterns that easily hit their specs (be it FLOPs or Memory Bandwidth)

GPUs are dumb, but that makes them kinda simpler for these kinds of simple tests.

CPUs, in particular the Thread ripper Pro, have very complex memory hierarchies.

Unless they experiment with NUMA mode (4 or 8 nodes maybe), I wouldn't expect much improvement. To max out the memory Bandwidth on a CPU, you really gotta understand MESI / cache snooping / false sharing, split up your accesses across the chip let's (easier to do with NUMA turned on) and code in a NUMA aware manner.

---------

I'm not saying GPUs are always better. But this particular test is really bloody simple and ideal for GPUs.

ashvardanian13• 4 years ago

Yeah, completely agree. Sadly we live in the convergence era. Now GPUs are also multi-chip (starting with MI200) and the latencies will likely become unpredictable once again :)

tuetuopay• 4 years ago

You are missing one critical element: NUMA awareness.

Because of the Chiplet design, Threadrippers and EPYC chips are basically NUMA systems, like you would with a multi-socket motherboard. This has an impact on both memory latency and bandwidth.

Redo your test, but pin each thread to a CPU thread, and its memory pool to a stick of ram attached directly to the CCX of said thread. The bandwidth you are measuring is the Infinity Fabric's bandwidth, NOT the DDR4 bandwidth.

You can read more about infinity fabric and its bandwidth here: https://en.wikichip.org/wiki/amd/infinity_fabric

ashvardanian13• 4 years ago

Thanks, that's a valid suggestion! But in that case I would be forced to split my array into smaller chunks in different disjoint region of address space, right?

vardump• 4 years ago

Came to say exactly this.

A lot of code is wasting a ton of performance on servers because of not being NUMA aware. We should also test high throughput/performance code on NUMA systems.

Remember to allocate local memory for your threads — IOW, be NUMA aware. Going through interconnect(s) is slow.

jjoonathan• 4 years ago

How is that generally done? Platform specific libraries? Utilities that ask the OS to pin a program to a node?

hardolaf• 4 years ago

I'm in HFT and a huge part of our job for the FPGA drivers is managing the NUMA awareness of our software. And we have to constantly educate new devs coming into the low level software teams about this. Luckily, the SW teams are starting to do this training for us but it's always a major headache when there's issues. You can easily lose 50%+ of your performance by just failing to set a core affinity without even considering anything else.

bravetraveler2• 4 years ago

Hadn't even opened the article yet and I was thinking exactly this, 'bet they forgot NUMA'

ashvardanian13• 4 years ago

On a second thought, are you sure Infinity Fabric would be that slow? I would expect it to be compared to L3 bandwidth not RAM. Plus, the article you gave suggests 40 GB/s P2P IFOP with slower 2.6 GHz RAM. I can't find the right diagram , but as I remember, this CPU had 8 chiplets, still allowing for much more bandwidth, depending on the topology.

dragontamer10• 4 years ago

I would expect it to be compared to L3 bandwidth not RAM.

Infinity Fabric is well known to be slower / lower-bandwidth than DRAM actually.

Just one of the peculiarities of this architecture.

------

That's why understanding MESI is important. If Core#10 writes something to DRAM: then Core#45 will not be able to write to it until Core#10 releases it (aka: reaches the "invalid" state of MESI).

In contrast, writing to DRAM that's "not owned" by anybody is one-and-done. No need to communicate to other cores or chiplets.

------

No _singular_ core can reach 204GB/s spec of the architecture. You only reach that 204GB/s spec if all the cores (or chiplets really) are visiting different sections of RAM independently.

This happens in practice often: the whole EPYC / Threadripper design is really aimed for virtual machines doing independent tasks. If you're trying to write high-performance code on EPYC / Threadripper, its important to keep this model in your brain.

-------

Once you fix this problem, your next problem is probably going to be the page-table / virtual memory system. Maybe you should be looking up huge-pages in preparation?

wmf• 4 years ago

AFAIK each CCD gets 50 GB/s over Infinity Fabric which adds up to 200 GB/s which is effectively the same as memory bandwidth. IF should only be a bottleneck if you want to get high bandwidth from few CCDs.

lostmsu2• 4 years ago

I upvoted this knowing the explanation but not feeling myself confident enough to actually write it down in the hope that someone more authoritative will. Thank you!

lmilcin• 4 years ago

I would add that seeing somebody "forgetting" NUMA makes it likely there is a lot of other problems.

NUMA isn't something you "forget" when you try to test bandwidth limits. It suggest lack of understanding of how that stuff works.

It is as if somebody failed to reach hard drive transfer speeds and they "forgot" they need to be looking at the access pattern.

bravetraveler2• 4 years ago

I think that's fair - this reads heavily like someone that thinks throwing hardware at the problem is all it takes. The marketing materials are guarantees. That kind of thing.

They even quoted direct AMD statements that would indicate the memory access pattern isn't uniform. They glossed over it when they saw numbers.

I used the term 'forget' in one of my posts, but it's not a true representation - NUMA isn't the kind of thing you forget when you're actually doing HPC. It's the bread and butter.

It's like your mechanic forgetting your vehicle has a motor that needs consideration...

tester756• 4 years ago

but it only proves his point

namibj3• 4 years ago

No, these are Zen2. Both Zen2 and Zen3 single-socket systems are not NUMA systems in the traditional view (i.e., w.r.t. DRAM).

The L3 is CCX-local, but with Zen2 they re-introduced a separate northbridge die for each socket, integrated with the CCDs as a separate chiplet.

Dual-socket systems remain NUMA, and for workloads that feel L3$ more than DRAM, treating CCX-as-NUMA still helps (but that's a scheduler thing, where kernel-level threads/processes (Linux barely distinguishes) that do fine-grained shared memory with _collective_ temporal locality should be accessed from within a single CCX).

hindsightbias• 4 years ago

But isn’t Zen2 a MCM? It’s not a single uniform processor and there is another level of abstraction on top of a socket

paulmd2• 4 years ago

This is not correct. Despite only having a single IO die, the Zen2/Zen3 IOD is actually divided into four quadrants, and each chipset (or pair of chipsets) has preferential access to that quadrant, and has to cross an interconnect to get to the other quadrants (and their memory). This is still a non-uniform topology.

This is explicitly called out in the BIOS of these systems: the terminology is "NPS4 mode" (Nodes Per Socket) and is documented in AMD's reference materials. The system can also be run in what's called "NPS1 mode", where all memory channels are interleaved, but since this obviously requires waiting for all four quadrants to finish their memory access, this increases latency somewhat.

https://developer.amd.com/wp-content/resources/56338_1.00_pu...

Anandtech did some really good coverage of this in their 3rd Gen Epyc (Milan) deep-dive, including latency measurements. Zen2 is largely similar architecturally in this area. You can see from comparing the NPS1 and NPS4 charts that Zen2 has around 12.8% higher memory latency in NPS1 mode and this is reduced to 6.4% in Zen3.

https://www.anandtech.com/show/16529/amd-epyc-milan-review/4

If you ignore the physical placement of functionality and simply look at a birds-eye view of the data paths, it's not that different from Zen1. Zen1 had four "chiplets" that each have their own memory controller (and associated uncore). In Zen2, the uncore has simply been pulled from the chiplets onto the standalone IOD, but it is still implemented as four quadrants - just as Naples was four monolithic dies interconnected and packaged together.

As mentioned, since effectively the entire uncore is divided into quadrants, this has a few other quirks. The one that most frequently comes up is populating memory channels - you really really want to populate all memory channels on Epyc, even when running in NPS1 mode. If you're not going to populate sets of 8, you really really want to make sure it's one of the "balanced" configurations, as otherwise some quadrants don't have access to memory at all, and the performance hit can be substantial. For example, Lenovo's documentation shows that populating only 6 out of 8 channels can result in a 29% performance hit (relative to the theoretical potential of a 6-DIMM configuration) even in the "correct" configuration (two quadrants each lose one channel), and an improper configuration (one quadrant with no attached memory) will have a 60% performance hit. Populating 7 channels will result in a 65% performance hit from the theoretical maximum - you are losing 2/3rds of your performance largely due to the NUMA topology!

https://lenovopress.com/lp1268.pdf

PCIe latency is also slightly higher when crossing quadrants. Not the sort of thing most people will be paying attention to, but to use an example here, the guy above who's doing HFT and paying attention to NUMA affinity is probably paying attention to what cores go to what PCIe lanes for talking to his FPGAs, because it does matter. Netflix also ran into similar issues around bandwidth - needlessly pushing data across the NUMA domains will eventually bottleneck performance, if you are pushing enough data. Keeping it inside the quadrant doesn't incur that bottleneck.

It really is a testament to how well AMD made NUMA work that it doesn't "feel" like NUMA - and I think they even turn the NPS1 mode on by default now. But architecturally, it is NUMA underneath, and you can extract a small amount of additional performance by pulling the veneer of UMA away and addressing the hardware as it is actually implemented.

StillBored• 4 years ago

I would try something like this on gravaton2/3 or a recent POWER machine, in my experience its a lot easier to get reasonable memory perf out of !intel machines.

bee_rider2• 4 years ago

The threadripper is also a !intel chip.

ashvardanian13• 4 years ago

Threadripper is by AMD

tryp• 4 years ago

Some folks are just not TradeMarksters. They'll teach you that Intel makes processors that implement x86 and amd64 instruction sets. You can Google things on DuckDuckGo. They'll blow their nose on the Kleenex that they bought from GenericPaperProductsCo. At some restaurants in the US South you'll have to let your server know what kind of Coke you want: cola, lemon-lime, or the Doctor one.

jhokanson3• 4 years ago

Does AMD have folks that you can reach out to regarding this? I know Intel has MKL and all the work around its own compiler for maximum speed. This seems like it should be trivial for someone at AMD to put together as an example of how to do things like this correctly ...

ggreg843• 4 years ago

I bet AMD itself is not able to get even close to the 204 GB/s bandwidth that they "promise" for these CPUs.

AMD instructions for measuring peak bandwidth are here: https://developer.amd.com/spack/stream-benchmark/ (essentially does a multi threaded memcpy). The script there will pin 1 thread per CCD, and with 8 threads you'll get close to peak bandwidth on CPUs with 8 CCDs (adding more threads doesn't help).

I've run it in the same CPUs that AMD uses there, following their instructions to the letter, which should be able to achieve 204 GB/s, but the best I've seen is about 165 GB/s. That's 80% of what AMD advertises.

HOWEVER, THIS NUMBER IS A LIE. It is using the AMD compiler with a flag called "-fnt-store" to turn all stores into non-temporal stores. This is something that no real-world application in practice would actually do because it does not make sense doing in general.

Without this flag, the peak bandwidth that AMD scripts report is about 124 GB/s. That's 60% of what AMD advertises and the best that most applications can aim to obtain in practice (the blog post author is really close to it).

The fine print (https://www.amd.com/en/products/cpu/amd-epyc-7742) does say that 204 GB/s is a "theoretical" number.

So I bought a "theoretical" 204 GB/s CPU that delivers 60% of what it "theoretically" promised.

I feel scammed.

---

Instead, if I run STREAM on my GPU, I get close to 100% of what the GPU promises, and about 90% or so of it in practice.

ashvardanian13• 4 years ago

Those are great links, thanks for sharing! My memcpy-s were even slower :)

sliken2• 4 years ago

You seem surprised, this is pretty much the rule for ALL CPUs. The 204GB/sec number is simply DDR4-3200 x 64 bits (width of one memory channel) x 8 (number of memory channels = 204,800MB/sec or "roughly" 204GB/sec (assuming GB = 10^9 not 2^30). This is called "peak" bandwidth, as is a never to exceed number.

60-80% of peak is common with tuned code, and less without. Sure GPUs, which are all about bandwidth do better. Much of a GPU design is about throughput and hiding latency with multiple requests/threads/whatever. CPUs have different design goals and worry much more about latency and often have to worry about NUMA/memory coherency. Not to mention in the general case often you have one cache miss per core, which isn't necessarily enough to saturate the memory bus.

Try apple M1, other ARMs, Alpha, PA-risc, Sparc, Intel, etc and you'll find the same story.

ggreg843• 4 years ago

This is called "peak" bandwidth, as is a never to exceed number.

This is the peak bandwidth of DDR4, saying that would be fine.

AMD claims that this is the theoretical peak bandwidth of their CPU, not of DDR4, yet their CPUs memory subsystem are too poor to even get close to this number.

Try apple M1, other ARMs, Alpha, PA-risc, Sparc, Intel, etc and you'll find the same story.

I tried the Ampere Altra and Graviton and they achieve ~170-180 GB/s without non-temporal stores. You can get an AWS graviton instance and verify it yourself.

That's 90% of "theoretical DDR4 peak" for ARM vs 60% for AMD.

So no, i disagree. This is not the case for ALL cpus, this is the case for AMD cpus. And no, 60% vs 90% "out of the box" perf is not the same.

dragontamer10• 4 years ago

I'm somewhat surprised that AMD's stream-benchmark uses "transparent_hugepage" rather than explicitly using huge-pages.

Has Linux's transparent_hugepage feature gotten good enough to use in today's environments? Maybe its good enough for STREAM-benchmarking (simple memory patterns) ??

Or is "STREAM" unable to use huge-pages directly (so we require the OS to try and guess appropriate usage?) ??

blihp• 4 years ago

Yes, it works quite well. Explicit 1gb huge pages also provides significant improvements for the right workloads. You pretty much have to start taking advantage of features like these to get anywhere near the theoretical maximum throughput in real world usage even on the higher core count desktop chips.

Failing to reach DDR4 bandwidth

Related Stories