AMD's CDNA 3 Compute Architecture

https://chipsandcheese.com/2023/12/17/amds-cdna-3-compute-architecture/

by ksec • 3 years ago

265 133 3 years ago

jdewerd3• 3 years ago

AMD diverged their GPU architecture development into separate CDNA and RDNA lines specialized for compute and graphics respectively.

Ooooh, is that why the consumer cards don't do compute? Yikes, I was hoping that was just a bit of misguided segmentation but this sounds like a high level architecture problem, like an interstate without an on ramp. Oof.

CuriousCosmic2• 3 years ago

They added support for high end radeon cards 2 months ago. Rocm is "eventually" coming to RDNA in general but it's a slow process which is more or less in line with how AMD has approached rocm from the start (targetting a very small subset of compute and slowly expanding it with each major version).

https://www.tomshardware.com/news/amd-enables-rocm-and-pytor...

mnau• 3 years ago

That's a good step forward. This separation is a massive barrier to entry. Making CUDA work reliably on every product was a masterstroke by NVIDIA.

treprinum3• 3 years ago

The issue is that RDNA and CDNA are different so when an enthusiast makes a fast RDNA code it doesn't mean it would work well on CDNA and vice-versa. Not sure why AMD had to go this route, only high-end pros will write software for CDNA and they won't get any mindshare.

DarkmSparks2• 3 years ago

I was quite surprised to find at least one of the the top500 is based on AMD gpus, it seems to me not bringing that to the consumer market was a conscious choice made some time ago.

fomine3• 3 years ago

Some HPC applications are specially made for the HPC, so minor API is acceptable for some HPCs. That's not the case for consumer GPGPU use case.

slavik812• 3 years ago

There are eleven CDNA 2 systems in the top500. https://www.top500.org/lists/top500/list/2023/11/?page=1

nolok• 3 years ago

They're far behind Nvidia in gaming, to they try to compete on price to get market share back.

Eg the 7900 XTX competes with the 4090 in rastering but is between 30 and 50% cheaper, 7900 XT / 4080, 7800 XT / 4070 are all winning by decent amount in pricing. Frankly this focus has allowed them to catch back in a field they've behind for more than a decade now.

The rise of RT and other compute tools in gaming (DLSS and co) is hurting this strategy, but their APU being on PS5 AND Xbox Series keep them in the race, game engines and game developpers need to make it work on AMD tech.

With Nvidia pricing and memory choice this generation, it's the first time in a while that bying an AMD card for gaming is the better choice at almost every price point (unless you can afford the 4090).

cogman10• 3 years ago

Looks like the split was around 2016? Which could make sense given the state of crypto at the time. One problem that hit nvidia more than AMD is consumer cards getting vacuumed up by crypto farms. AMD making a conscious split effectively isolated their compute cards from their gamer cards.

That being said, I can't imagine this is something that's been good for adoption of AMD cards for compute tasks. The wonderful thing about CUDA is you don't need a special accelerator card to develop cuda code.

wmf• 3 years ago

You can't really isolate crypto mining. If a gaming card is profitable to mine with, miners will buy it and use the gaming drivers.

Zardoz84• 3 years ago

if i remember correctly, AMD GPU cards was being sold very well to crypto miners. The split wasn't because that. They did, because gamming and computing having different hardware optimization requisites, and that allowed to get better and more competitive GPUs and computing cards on the market.

dotnet00• 3 years ago

The split had more to do with GCN being very compute heavy and AMD's software completely failing to take advantage of that. GCN got transitioned into CDNA, and got heavily reworked for graphics as RDNA. This is why while RDNA is nowhere near as good at compute oriented features, it tends to outperform NVIDIA in pure rasterization.

In contrast, current NVIDIA GPUs are very compute heavy architectures (to the point that they often artificially limit certain aspects of their consumer cards other than fp64 to not interfere with the workstation cards), and they take great advantage of it all with their various features combining software and hardware capabilities.

ColonelPhantom6• 3 years ago

Ooooh, is that why the consumer cards don't do compute?

No, not really. Sure, the consumer cards are based on a less suitable architecture, so they won't be as good, but it's not like it's not a general purpose GPU architecture anymore.

The real problem is that ROCm has made the very weird technical decision of shipping GPU machine code, rather than bytecode. Nvidia has their own bytecode NVPTX, and Intel just uses SPIR-V for oneAPI.

This means that ROCm support for a GPU architecture requires shipping binaries of all the libraries (such as rocBLAS, rocDNN, rocsparse, rocFFT) for all architectures.

As the cherry on top, on paper one GPU family consists of different architectures! The RX 6700 XT identifies as a different 'architecture' (gfx1031) than the RX 6800 XT (gfx1030). These are 'close enough' that you can tell ROCm to use gfx1030 binaries on gfx1031 and it'll usually just work, but still.

I feel like this is a large part of the reason AMD is so bad about supporting older GPUs in ROCm. It'd cost a bit of effort, sure, but I feel like the enormous packages you'd get are a bigger issue. (This is another factor that kills hobbyist ROCm, because anyone with a Pascal+ GPU can do CUDA, but ROCm support is usually limited to newer and higher end cards.)

slavik812• 3 years ago

As the cherry on top, on paper one GPU family consists of different architectures! The RX 6700 XT identifies as a different 'architecture' (gfx1031) than the RX 6800 XT (gfx1030). These are 'close enough' that you can tell ROCm to use gfx1030 binaries on gfx1031 and it'll usually just work, but still.

As far as I can tell, the gfx1030 and gfx1031 ISAs are identical in all but name. LLVM treats them exactly the same and all tests for the ROCm math libraries will pass when you run gfx1030 code on gfx1031 GPUs using the override.

The Debian package for HIP has a patch so that if the user has a gfx1031 GPU and there are no gfx1031 code objects available, it will fall back to using gfx1030 code objects instead. When I proposed that patch upstream, it was rejected in favour of adding a new ISA that explicitly supports all the GPUs in a family. That solution is more complex and is taking longer to implement, but will more thoroughly solve that problem (as the solution Debian is using only works well when there's one ISA in the family that is a subset of all the others).

JonChesterfield2• 3 years ago

There was a ship-bytecode idea branded HSAIL, associated with opencl finalisers in some way. I'm not sure what killed that - would speculate that the compiler backend was annoyingly slow to run as a JIT. It's somewhat conceptually popular under the spirv branding now.

Some of the ROCm libraries are partially written in assembly. Hopefully they all have C reference implementations that can be compiled for the other architectures.

My guess is that the AMD strategy of only shipping libraries for some cards + open source means ROCm-ish as built by Linux distros, who are likely to build the libs for all the architectures and not just the special few, can be expected to drive people away from the official releases.

beebeepka• 3 years ago

I thought trolling wasn't welcome on HN

vegabook2• 3 years ago

AMD has always been a terrible steward of ATI.

They're fundamentally a hardware company (as per Lisa Su's CV) which didn't cotton on to the fact that Cuda was the killer. I remember how @Bridgman kept playing a rearguard action on Phoronix, his job being to keep devs on board. Losing battle.

I kinda understand it because [80/90]s era hardware people intrinsically think hardware is the top dog in the stack, and that's where all AMD management come from including Su.

Kodura understood that Nvidia was killing AMD because the consumer cards could run CUDA. So he pushed through, against Lisa Su, the Radeon VII, which until very recently, and for many years, was the only consumer card that ROCm supported. He was basically fired shortly thereafter, and the RVII, which was a fantastic card, was shutdown sharpish. Then they brought in Wang who crystallised the consumer/pro segmentation.

Now they're furiously trying to backpedal, and it's too late. There are multiple attempting competitors, but basically the only one worth talking about is AAPL and Metal.

AMD lost the window.

simfree• 3 years ago

ATI on its own did not have a solid future, they could have easily ended up like their other local Canadian counterparts Nortel or Blackberry.

ColonelPhantom6• 3 years ago

I'd really not call the Radeon VII a "fantastic card" by any stretch of the imagination, outside of specialized use cases like FP64 compute. Its only selling point is that and the enormous 1 TB/s memory bandwidth with the for the time large 16 GB, which a friend summarized as "I can push 1 TB/s of garbage to nowhere!".

Sure, if those things are relevant to you, it was a killer card, but most consumers judge GPUs first and foremost by their gaming performance, in which the VII was a total dud. The 5700 XT was nearly as fast in those with far less hardware because getting proper utilization on GCN/CDNA is a pain compared to RDNA.

RDNA2 completed the gaming oriented flip with Infinity Cache (which is also useful outside of games, but still has limited application in GPGPU) which allowed them to offer competitive gaming performance to Nvidia with far less hardware, a big difference from the Vega era.

sva_• 3 years ago

Well the APU does, there are compute intrinsics and even a small guide by AMD on how to use them:

https://gpuopen.com/learn/wmma_on_rdna3/

pauljurczak• 3 years ago

How do you use these intrinsics on an APU? https://llvm.org/docs/AMDGPUUsage.html ?

Const-me10• 3 years ago

the consumer cards don't do compute?

Typically, software developers only support a single GPGPU API, and that API is nVidia CUDA.

Technically, consumer AMD cards are awesome at compute. For example, UE5 renders triangle meshes with compute instead of graphics https://www.youtube.com/watch?v=TMorJX3Nj6U Moreover, AMD cards often outperform nVidia equivalents because nVidia prioritized ray tracing and DLSS over compute power and memory bandwidth.

The issue is, no tech company is interested in adding D3D or Vulkan backend to AI libraries like PyTorch. nVidia is not doing that because they are happy with the status quo. Intel and AMD are not doing that because both hope to replace CUDA with their proprietary equivalents, instead of an open GPU API.

boppo1• 3 years ago

Vulkan backend to AI libraries like PyTorch

How many man hours is a project like this?

GeekyBear• 3 years ago

The issue is, no tech company is interested in adding D3D or Vulkan backend to AI libraries like PyTorch.

AMD is interested in making PyTorch take advantage of their chips, and has done so.

we are delighted that the PyTorch 2.0 stable release includes support for AMD Instinct™ and Radeon™ GPUs that are supported by the ROCm™ software platform.

When NVIDIA is charging 1000% profit margins for a GPU aimed at use as an AI accelerator, you can expect competitors to be willing to do what it takes to move into that market.

https://www.tomshardware.com/news/nvidia-makes-1000-profit-o...

zozbot2342• 3 years ago

Vulkan Compute backends for numerical compute (as typified by both OpenCL and SYCL) are challenging, you can look at clspv https://github.com/google/clspv project for the nitty gritty details. The lowest-effort path so far is most likely via some combination of Rocm/HIP (for hardware that AMD bothers to support themselves) and the Mesa project's RustiCL backend (for everything else).

einpoklum7• 3 years ago

Typically, software developers only support a single GPGPU API, and that API is nVidia CUDA.

There is very little NVIDIA and CUDA cards off of X86_64 and maybe OpenPower architectures. So I disagree. Also, OpenCL, despite being kind of a "betrayed standard", enjoys quite a lot of popularity even on x86_64 (sometimes even with NVIDIA hardware) - even if it is not as popular there.

AMD cards often outperform nVidia equivalents

Can you link to benchmarks or other analysis supporting this claim? This has not been my impression in recent years, though I don't routinely look at high-end AMD hardware.

because nVidia prioritized ray tracing and DLSS over compute power and memory bandwidth.

Has it really?

jorvi2• 3 years ago

Intel and AMD are not doing that because both hope to replace CUDA with their proprietary equivalents, instead of an open GPU API

Like AMD did with FreeSync (now VRR) and FSR3 (whatever open standard it will morph into)?

I'll never understand the hate AMD gets from the open source community. Mind-boggling.

spookie2• 3 years ago

They aim to replace CUDA with their proprietary equivalents

Neither oneAPI, nor ROCm are proprietary?

pjmlp4• 3 years ago

DirectML is part of DirectX 12.

0xDEADFED5• 3 years ago

The issue is, no tech company is interested in adding D3D or Vulkan backend

i'm using Apache TVM right now to run LLMs via Vulkan on Intel Arc cards.

Intel has their own pytorch wheels here for their GPUs: https://developer.intel.com/ipex-whl-stable-xpu

paulmd6• 3 years ago

AMD's products have generally failed to catch traction because their implementations are halfassed and buggy and incomplete (despite promising more features, these are often paper features or resume-driven development from now-departed developers). all of the same "developer B" stuff from openGL really applies to openCL as well.

http://richg42.blogspot.com/2014/05/the-truth-on-opengl-driv...

that's the reason blender kicked their openCL implemention to the curb after years of futzing with it. it wasn't portable because AMD's openCL was so broken they had to have their own special codepath anyway, and it didn't perform well and wasn't worth the effort.

AMD has left a trail of abandoned code and disappointed developers in their wake. These two repos are the same thing for AMD's ecosystem and NVIDIA's ecosystem, how do you think the support story compares?

https://github.com/HSA-Libraries/Bolt

https://github.com/NVIDIA/thrust

in the last few years they have (once again) dumped everything and started over, ROCm supported essentially no consumer cards and rotated support rapidly even in the CDNA world. It offers no binary compatibility support story, it has to be compiled for specific chips within a generation, not even just "RDNA3" but "Navi 31 specifically". Etc etc. And nobody with consumer cards could access it until like, six months ago, and that still is only on windows, consumer cards are not even supported on linux (!).

That's fine for non-commercial and HPC, but it makes it difficult to imagine ever shipping commercial products to end-users on it. I guess you... ship as source, and your users have to figure out ROCm and compile it in-situ? Or ship a binary for every single GPU to ever release (including NVIDIA, Intel, ....)

https://geohot.github.io/blog/jekyll/update/2023/06/07/a-div...

This is on top of the actual problems that still remain, as geohot found out. Installing ROCm is a several-hour process that will involve debugging the platform just to get it to install, and then you will probably find that the actual code demos segfault when you run them.

AMD's development processes are not really open, and actual development is silo'd inside the company with quarterly code dumps outside. The current code is not guaranteed to run on the actual driver itself, they do not test it even in the supported configurations, because all the development is happening on internal builds etc, not release code.

oh and AMD themselves isn't using the open driver of course... they only test and validate on AMDGPU-PRO.

it hasn't got traction because it's a low-quality product and nobody can even access it and run it anyway, let alone ship commercial products on it. and now everyone regrets it because it's turned into a money fountain.

7speter3• 3 years ago

The big kicker is that cdna is still based on the much maligned

Pomfers• 3 years ago

AMD consumer cards do compute, but do not have a developed or well supported ecosystem (ROCm is a mess).

Misguided segmentation or a high level architecture problem however, it is not. A specialized product will perform its specialty better than a general purpose product. There's minimal demand for a card that's good at both compute and gaming. These people exist, but are few compared to those who only care about one or the other.

The benefit of splitting GCN into RDNA and CDNA was immediate. Comparing the Radeon VII (GCN 5) vs RX 5700 XT (RDNA 1), you'll see that they're fairly evenly matched on gaming, trading blows with the Radeon VII slightly ahead when averaged. The RX 5700 XT takes a very heavy loss on compute benchmarks. Both are TSMC 7nm, but the RX 5700 XT has fewer shaders (2560 vs 3840), a smaller die (251 vs 311 mm2), and lower power consumption (225 vs 300 w) which shows how much more efficient it is at gaming. That much lower power consumption and noise, and a couple hundred dollars cheaper made it a much more compelling card for gamers.

CDNA cards are apparently missing components needed for gaming, like render output units. Hence, they have no official support for DirectX, OpenGL, or Vulkan. I've never seen anyone get one of these working for gaming. In exchange, their compute performance is so good that a number of companies are buying these cards over Nvidia's, despite the overwhelming CUDA ecosystem. In 2013, a GCN based supercomputer made the top 100. That is the one and only GCN based system to ever make the top 100. Compare to now, where 8 of the top 10 most energy efficient super computers use CDNA accelerators, as does the number 1 fastest supercomputer outright.

diego_sandoval• 3 years ago

These people exist, but are few compared to those who only care about one or the other.

They're few, but they're very important, because 5 years from now, they will be the ones working for the companies that AMD will want to sell their compute cards to.

The reasons you cite may still be more important. I'm not saying AMD made a bad decision. I'm just saying that neglecting the path from hobbyist to professional (or student to professional) is not insignificant.

kaycebasques• 3 years ago

I wasn't familiar with VLIW. Sounds cool!

Very long instruction word (VLIW) refers to instruction set architectures designed to exploit instruction level parallelism (ILP). Whereas conventional central processing units (CPU, processor) mostly allow programs to specify instructions to execute in sequence only, a VLIW processor allows programs to explicitly specify instructions to execute in parallel. This design is intended to allow higher performance without the complexity inherent in some other designs.
The traditional means to improve performance in processors include dividing instructions into substeps so the instructions can be executed partly at the same time (termed pipelining), dispatching individual instructions to be executed independently, in different parts of the processor (superscalar architectures), and even executing instructions in an order different from the program (out-of-order execution).[1] These methods all complicate hardware (larger circuits, higher cost and energy use) because the processor must make all of the decisions internally for these methods to work.

https://en.wikipedia.org/wiki/Very_long_instruction_word

mpweiher• 3 years ago

The most prominent example of VLIW processors was the Itanic, er, Itanium.

It, er, didn't work out well.

Hence Itanic.

Their premise was that the compiler could figure out dependencies sufficiently statically so that they could put multiple sequential and some divergent execution paths in the same instruction. It turned out the compilers couldn't actually do this, so processors figure out dependencies and parallelizable instructions dynamically from a sequential instruction stream.

Which is a lot of work, a lot of chip resources and a lot of energy. And only works up to a point, after which you hit diminishing returns. Which is where we appear to be these days.

kimixa2• 3 years ago

VLIW is also still heavily used in DSPs and similar. And arguably the dual issue instructions in both AMD's and Nvidia's latest GPUs is a step back towards that.

It feels like a lot of these things are cyclical - the core ideas aren't new and phase in and out of favor as other parts of architecture design changes around and increases or decreases benefits accordingly.

Symmetry• 3 years ago

I'd argue that the Hexagon DSPs shipped with all Qualcomm's phone SoC which a large number of us have in our pockets right now are probably more prominent example.

Itanium's problem wasn't so much divergent control flow as not being able to handle unpredictable latencies on load instructions without halting all execution entirely. There were some features that were, in theory, supposed to deal with this but using them was where the magic compiler problem came in.

For DSPs you can usually predict memory latency so this isn't an issue and VLIW works great there even with some amount of branching.

ip262• 3 years ago

It worked fine for graphics chips for a long time. Graphics workloads have embarrassing levels of static ILP.

yarg• 3 years ago

Read up on SIMD in general.

(The means of processing as opposed to the language used to dispatch commands.)

(And worth baring in mind is the fact that terms such as VLIW4 and VLIW5 refer to specific implementations.)

https://en.wikipedia.org/wiki/Single_instruction,_multiple_d...

gpderetta2• 3 years ago

simd != vliw though.

einpoklum7• 3 years ago

Compute has been outpacing memory for decades. Like CPUs, GPUs have countered this with increasingly sophisticated caching strategies.

I'd say it's rather the contrary. Unlike CPUs, GPUs don't attempt to directly counter this. By accepting higher latencies, they have let themselves parallelize much more widely (or wildly) relative to CPUs - and the high number of parallel pseudo-threads provides a "latency hiding" effect.

This effect is illustrated for example, in this presentation on optimizing GPU code:

https://www.olcf.ornl.gov/wp-content/uploads/2019/12/03-CUDA...

(lame) animation on slide 11 and onwards.

ColonelPhantom6• 3 years ago

GPUs do deal with memory in other ways than parallelism! That's why GPUs tend to offer huge register files (up to 256 architectural registers per thread on RDNA1!) and local memories (up to 64KB LDS per workgroup on RDNA1). This means lots of work can be done purely in registers and LDS, and trips to global memory are way rarer compared to CPUs where global memory contain everything except 16 or so architectural registers.

Even then, global memory is an issue. And not just because of latency, but also bandwidth! That's why RDNA2 and Ada both added a ton of last-level cache, not to better hide latency (although it's a welcome addition), but mostly as a 'bandwidth amplifier'.

einpoklum7• 3 years ago

Two valid points, +1.

soundarana• 3 years ago

CPUs have been growing their register files too - AVX512 has 32 x 512 bits registers.

m_mueller2• 3 years ago

I think register files, shared memory and even texture memory patterns are basically the GPU's way of having fully programmable caches, as opposed to cache being a grey box like on CPUs. Overall I think for HPC code it works out better. Looking at vectorized and cache-optimized CPU code gives me the shivers...

Mostly due to the above I think Intel's years long push to try to have people just run their CPU-optimized OpenMP codes on their Xeon Phis was the downfall of that architecture - it just can't win against a data-parallel-first framework like CUDA, mainly because there the data parallel compute thread can be programmed just as such, rather than trying to get the compiler to behave exactly as you want (and in the end giving up and dropping down to assembly-like SSE instructions).

feanaro• 3 years ago

A bit off topic, but when did "compute" become a noun? It is extremely grating to my ears.

synergy20• 3 years ago

it's pretty common these days due to AI with GPU-alike chips.

EarthLaunch• 3 years ago

I first recall it from the rise of AWS at least. "Amazon Elastic Compute Cloud (EC2)" launched 2006 [0]. Google Trends [1].

0: https://en.m.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud

1: https://trends.google.com/trends/explore?date=all&q=Compute&...

overstay8930• 3 years ago

It was a thing in Deep Space Nine (1999) so either the lingo was really on point or it's just something that has fallen in and out of favor.

ip262• 3 years ago

I don’t know the reason, but I suspect you can reason it out.

greggsy• 3 years ago

I hear, read and use the term on a daily basis, and I would say that it emerged in my workplaces around five years ago, and came into common use around two.

Geisterde• 3 years ago

5 points for the Luddite (me) that said amd would leverage their knowledge of chiplets and bus fabrics to make a comeback in AI. I wont pretend to be able to read this article, or read in general, just wanted to plant my flag.

de6u99er• 3 years ago

I don't think AMD will be able to compete with NVidia in the neat future because many scientists developing core libraries in the ML/AI space, are getting free or heavily discounted GPUs from NVidia. If they had to buy GPU's from their own or grant money, and pay the same price as regular consumers, the situation might be a different one.

IMHO the way NVidia is infiltrating academics and academia, is highly unethical.

dagw• 3 years ago

There is an also a lot of bad blood towards AMD in the space. I know several people who spent a lot of time trying to support both Nvidia and AMD GPUs back in the early days only to have AMD stop supporting APIs and making their code useless. Their CUDA code on the other hand just kept working on generation after generation of new Nvidia cards.

DeathArrow• 3 years ago

Nothing stops AMD to give free cards to developers.

elAhmo• 3 years ago

It is important to have in mind that Nvidia started investing resources and time in this 10+ years ago, CUDA was introduced in 2007, where none of the current ML/AI flows even existed. And they continued waiting, betting the company a number of times that the market will "show up" for the products they have built.

In the past few years that has happened, as reflected on the stock price. Other players are basically a decade behind them and with all of the current hype and AI/ML workflows becoming mainstream, I think it is next to impossible for anyone to catch up.

qeternity• 3 years ago

Other players are basically a decade behind them

I disagree. NVDA aren't in the position they are because they had to work 10 years to get there. They're in that position because it took the market that long to develop, and they were there the whole time.

But the main factor is demand. You can't rush that, no matter how early you start. And second mover advantage is a real thing. CUDA is a double edged sword. Everyone has an incentive to break NVDA's dominance, and ultimately their moat will become commoditized especially has dominant architectures like Transformers drive the vast majority of demand. People will simply optimize for that.

Can AMD catch up to them quickly on the training / R&D side? Doubtful.

Can AMD catch up or even surpass them on the inference side? Absolutely.

PH95VuimJjqBqy• 3 years ago

They built it for bit-coin mining, not some super long term plan for ML/AI.

grubbs• 3 years ago

I'm not sure how accurate this is. I work at a university supporting researchers doing all the typical "AI" research. LLMs, CV, etc. All NVIDIA discounts is the A5000 card for EDU. Maybe one other card they aren't interested in (L40?). Most are buying A6000 and up at consumer prices from companies like Exxact and Supermicro.

I don't think I've seen a single researcher get a free GPU since the V100 era. (DGX-1) system.

paulmd6• 3 years ago

I think this is indeed more of a 2009-2014 thing, and even then it was more "we give you a dozen K40s for your cluster" than "we give you, an individual researcher, a GPU". The stories I've heard of individual researchers asking for free stuff always ended in "no".

I don't see anything unethical about seeding universities (as an organization) with hardware tbh. That's just... grants. Even grants often come in the form of "we fund someone who's doing research on something that's related to us/using our hardware+software".

AMD's CDNA 3 Compute Architecture

Related Stories