AMD MI300 performance – Faster than H100, but how much?

https://www.semianalysis.com/p/amd-mi300-performance-faster-than

by klelatti • 3 years ago

116 85 3 years ago

gigatexal• 3 years ago

But without CUDA it’s gotta be DOA no? CUDA is what makes Nvidia’s hardware valuable. AMD should finance a competitor. They’ve had strong hardware but it’s the software that’s hurting them from what I’ve seen.

wmf3• 3 years ago

AMD is now starting to have decent software support.

JonChesterfield4• 3 years ago

It's somewhat sad that AMD bet on open source over proprietary and then approximately zero people stepped up to help. Easier to use cuda and complain I suppose. Fortunately they've stuck with the open source plan anyway and just written it themselves.

deaddodo• 3 years ago

They have a competitor, it's just less popular than CUDA:

https://www.amd.com/en/products/software/rocm.html

Before that, they were pushing heavily for standard OpenCL, but that failed because the hardware wasn't as competitive and the ecosystem/tooling barren.

atemerev4• 3 years ago

ROCm is not just unpopular, it is so boilerplate-ridden, it is nearly unusable. It takes about 5x more code to do things there compared to CUDA.

Yes, “just use libraries”. But as Andrej Karpathy used to say, “I don't need some library holding my hand and providing abstractions. Real men command GPUs with their own raw kernel code”.

(And, incidentally, there are much less libraries for ROCm, for this reason. Somebody should write them, and why do that with something that takes 5x more code to do the same thing?)

screye2• 3 years ago

Rocm will only get adoption when Cuda code can be ported over with 1 click.

At the very least, pytorch has to work effortlessly across GPUs, down to the less popular and odd workarounds people build into their pyotrch code.

Even if RocM reaches parity, the first mover advantage for Cuda is too large. RocM porting has to be literally effortless. Everything else is DOA.

screye2• 3 years ago

Edit-

That being said, if you a cloud provider and need to scale up a bunch of basically similar transformers models, then it should be an easy sell for AMD.

wmf3• 3 years ago

The facts don't bear this out. Various companies have already ported to AMD even though it isn't effortless.

com2kid• 3 years ago

Counterpoint, at scale a month of training a GPU cost more than an engineer now days.

And the large companies (Meta, Microsoft) buy these GPUs by the thousands upon thousands.

Having a small team of engineers spend a couple months porting code over is well worth it for even modest cost reductions.

These may not be useful to smaller companies working in the AI space, but odds are to sellout, all AMD really needs are the sales contracts that they've already announced.

Der_Einzige2• 3 years ago

People here don't want to hear the truth, but this is the truth.

trynumber9• 3 years ago

Depends on your definition of DOA. For the big customers who can't get as many H100 as they want it'll be an option. They have the resources to make sure their workload does in fact work and then scale out hardware appropriately. And it might not even be a downgrade: MI300X has more bandwidth and larger shared memory space.

And there's MI300A which has a combined CPU+GPU with shared memory space which already has supercomputer customers. You can say it's DOA relative to Hopper but MI300 will be AMD's most successful GPGPU yet.

bratao• 3 years ago

I don´t think so. There are few algorithms that cover the majority of use. For example in NLP a fast LLM (such as LLAMA) training/prediction and support for Hugging face transformers (BERT and such) would cover 95%+ of NLP use-cases that need a GPU.

kemotep• 3 years ago

AMD should finance a competitor.

If they can’t afford to out R&D Nvidia today, how could they afford to fund a competitor? In the past 20 years the only serious new entrant to the discrete GPU market has been Intel and they are also far behind Nvidia and CUDA.

lanza• 3 years ago

PyTorch and Triton don't use CUDA. An extraordinary volume of big tech compute only cares about the PTX “isa” for NVIDIA.

nharada• 3 years ago

One interesting consequence of the model consolidation happening in the LLM world is that it's actually easier for AMD's hardware to be competitive. It used to be that you had to support a ton of ops since nobody is going to use your GPU for just a single model. Nowadays, most models are just transformers with only a few operations inside, albeit repeated many times over. On top of that, you can build a reasonable application or business using only one or two LLMs (i.e. LLaMA 2 + Whisper, etc). So as long as they can get those running fast, they can sell GPUs to people who don't need the flexibility and breadth of CUDA.

KRAKRISMOTT• 3 years ago

The mighty dot product and matrix multiplication strikes again!

oivey• 3 years ago

I’m not sure this is true. Nvidia ships hardware today that is specialized for transformers, and presumably AMD needs something similar on their side. An arms race of specific hardware like that is probably not to their advantage. It makes the interoperability story even harder.

They still need a usable GPGPU language to glue all that hardware together, too.

dhruvdh5• 3 years ago

There is nothing special about Nvidia hardware outside of them having significantly better lower precision FLOPs before anyone else. Their marketing chose to call this a "transformer engine" but there is no actual "engine" in silicon.

Like AMD announced today, and is shown in the article we're commenting on, AMD now is 1.3 times faster at FP8, INT8 flops with and without sparsity, meaning their "transformer engine" is 1.3 times faster.

The rest of your comment doesn't make sense. See the other comments on software in this thread.

ilaksh• 3 years ago

The idea of model architecture making fast hardware design easier is what makes https://github.com/tinygrad/tinygrad so interesting.

dharma14• 3 years ago

yep! I think geohot could singlehandedly turn AMD into a winner if (big if) they get tinygrad competitive perf wise on AMD hw. So far it's not

latchkey8• 3 years ago

Today's MI300x announcement. Worth a watch before you comment about how their software sucks.

https://www.youtube.com/watch?v=tfSZqjxsr0M

VHRanger• 3 years ago

Koduri left so they have a chance to make the software not suck anymore

dhruvdh5• 3 years ago

I'll add some highlights for those who don't have the time - Flash Attention 2 kernels, Paged Attention kernels, PyTorch, vLLM, transformers, ONNX runtime support, OpenAI triton support in v3, tensorflow, JAX, OpenXLA.

Also, while this wasn't in the presentation, CUDA can be translated to HIP using automated tools. and - CuPBoPAMD is a CUDA translator that translates CUDA programs at NVVM IR level to HIP-compatible IR that can run on AMD GPUs (https://dl.acm.org/doi/pdf/10.1145/3624062.3624185)

latchkey8• 3 years ago

CUDA can be translated to HIP using automated tools.

Not just automated, but also open source.

https://github.com/ROCm-Developer-Tools/HIPIFY

viewtransform4• 3 years ago

They've gone so far to be code drop in compatible that an <import cuda> statement is still <import cuda> when running Rocm on AMD!

AMD made a presentation on their AI software strategy at Microsoft Ignite two weeks ago. Worth a watch for the slides and live demo

https://youtu.be/7jqZBTduhAQ?t=61

dotnet00• 3 years ago

CuPBoPAMD

They really couldn't come up with a better name than whatever that is?

incrudible• 3 years ago

All that buzzword bingo does not mean the software is actually robust and performs well. If you have hardware that has much better specs on paper, but only delivers maybe 20% better performance in their own benchmarks, I am still going to presume that their software is not up to par.

daveguy• 3 years ago

20% is a huge amount in terms of high end performance. For example, 20% more transistors / compute units / etc does not give 20% more performance at the top end unless your execution is excellent.

latchkey8• 3 years ago

20% better performance, lower cost and most important... availability.

bastard_op• 3 years ago

Ok, so even if the AMD chips are equal to or faster, Nvidia still has a large advantage with their networking using their proprietary NVLink switching that was most certainly borne from the Mellanox Infiniband IP acquisition. AMD is going to have to rely on more traditional PCI-E based networking for all off-system traffic via Infiniband or Converged Ethernet vs the more optimized Nvidia GPU clustering that uses it's on proprietary super-low latency NVLink-based switching.

https://www.nvidia.com/en-us/data-center/nvlink/

I'm super curious to see how their networking results compare between transports when real testing results between large supercomputer-ish farms get out there.

latchkey8• 3 years ago

You obviously didn't watch the announcement. They are doubling down on ethernet and open standards.

dhruvdh5• 3 years ago

AMD uses Infinity Fabric, which is their NVLink. They announced today how they're opening up Infinity Fabric to select partners, including Broadcom. Meaning partners can now develop their own switches.

Also, as AMD and HPE power some of the most performant supercomputers in the world, they wouldn't have won those contracts if their networking was subpar. Those use slingshot.

You may also be interested in reading about https://ultraethernet.org/ .

saberience• 3 years ago

NVLink is used for linking chips together, InfiniBand is used for linking systems together, i.e. into big HPC clusters. Infinity Fabric is used for inter chip communication, it's not a replacement for Infiniband and cannot be used to create clusters of machines.

viewtransform4• 3 years ago

I queued for you the part where Forrest Norad explains all this in today's presentation including a discussion on ultrafast ethernet. https://youtu.be/tfSZqjxsr0M?t=5198

JonChesterfield4• 3 years ago

I think Pensando and Xilinx are the networking story, not pcie. The clusters use Cray's slingshot. Within a node, the x64 cores and GPU units are on a common fabric which is also not pcie.

xnx• 3 years ago

No mention of price anywhere that I've seen. I assume it will only be a slight $/flop discount vs. Nvidia.

latchkey8• 3 years ago

Pricing at this point is almost irrelevant given that it is available in Q1/Q2 2024 and H100's are sold out until who knows when and you're still going to be at the bottom of a long line unless you have a giant order to place. Advantages, like IB, are 50+ week lead times as well, if you're lucky.

zitterbewegung• 3 years ago

Is this a drop in replacement for clouds (which is the actual question to ask because if you are buying this you have to make new infrastructure) the answer is No. Will Microsoft and Oracle allow you to use this to rent ? Yes. Does Microsoft and Oracle need more accelerators to do internal business processes? Absolutely (Microsoft announced it would rent AI accelerators because they can’t keep up with their own internal (and maybe external) demand). So cost has to not just be per card but infrastructure build out .

Not sure when Microsoft will just have their own TPU accelerator but I wouldn’t be surprised if the announce it sometime.

Der_Einzige2• 3 years ago

Can you cite your claims about Oracle wanting to deploy these? Oracle has a deep partnership with Nvidia to the point that in many cases, they have joint booths at conferences.

I was not aware of any efforts to purchase or deploy MI300s at OCI.

dhruvdh5• 3 years ago

Oracle were present at the event today and said it themselves. Also, it has been widely reported for months now that this is happening.

viewtransform4• 3 years ago

At today's AMD presentation the SVP of Oracle Cloud Infrastructure reveals it. https://youtu.be/tfSZqjxsr0M?t=2006

IceHegel• 3 years ago

They announced the Maia 100 AI accelerator last month. TSMC N5, 105b transistors.

https://news.microsoft.com/source/features/ai/in-house-chips...

dhruvdh5• 3 years ago

The part is OCP compliant and can be drop in. Microsoft has already annouced their "TPU" at Ignite. The amount of misinformation in these threads is driving me insane.

latchkey8• 3 years ago

Me too, but you're doing a great job not only responding to things, but also giving more context.

TradingPlaces• 3 years ago

*if you ignore sparsity.

shrubble2• 3 years ago

So how much of the market does AMD need to be profitable and keep going? Supposedly the Nvidia H100 alone, has ~15 billion USD in order backlog.

If AMD sells say $3 billion USD into that market, is that a big net positive for them?

hajile• 3 years ago

AMD's use of chiplets matters here. There is overhead, but they also not only get higher yields, but also have many more chips that can hit their highest target clockspeed meaning they can ship far more high-end chips.

This is the same reason why they were so successful with EPYC. It's easier to find eight small 8-core chips with high clocks/low-power than to find one 56-core chip with that same high-clock/low-power set which is why Intel sold so many cut-down chips while AMD just sold their small amount of defective chips as 6 and 12-core consumer chips.

Nvidia will have a harder time shipping defective because everyone wants the best chips with the best performance/watt to save money and do things fast. Outside of that, there's not a huge market for defective 800mm2 chips and the massive GPU the come on. AMD can sell their defective units a couple chiplets at a time to non-AI customers in laptops and workstation cards.

shrubble2• 3 years ago

Thank you for that detailed info - had not thought about that...

shaklee34• 3 years ago

It also implies that it's harder to program efficiently and is slower on the interconnect.

latchkey8• 3 years ago

I ran 120k of the rx470-rx580 series and another 30k of the PS5 APU. All of them were run at the most efficient and tuned settings possible. Every single chip was a snowflake.

We're not just talking defective, we are talking about a silicon lottery on performance and it varies between every single one.

ElectricalUnion• 3 years ago

If the backlog gets too big, as a consumer you might get compelled to just abandon it and switch to the "available" alternative even if it's sightly inferior.

Suddenly potentially having a lot less backlog might screw up Nvidia faster that it "helps" AMD too.

brucethemoose23• 3 years ago

One very exciting thing is the vram. 192GB!!!

That seems like a minor bump compared to the new H100, but sometimes the quality of your training, or the speed of your inference, is hugely effected by small thresholds of VRAM capacity per card. The MI300X can do things the H100 cannot, and judging by the sheer size of the silicon, do them reasonably quickly.

dharma14• 3 years ago

I think vram is one thing they can beat Nvidia on. Both on the pro and consumer GPUs. Just put 48gb into a consumer GPU and everyone would flock to AMD and help build their software side

brucethemoose23• 3 years ago

They already have 48GB Pro GPUs.

If they were going to do it on the consumer side, they would have done it already. And I would own a 7900 48GB instead of a 3090, and probably have debugged ROCm on several projects by now :/

yukIttEft• 3 years ago

H100 GPU is a sibling of NVidia's customer RTX 40-series: https://en.wikipedia.org/wiki/Hopper_(microarchitecture)

Is MI300 also a variant of their customer GPUs?

edward28• 3 years ago

This is a different architecture, CDNA3

anticensor• 3 years ago

Which is a derivative of pre-RDNA customer GPU architecture, Vega.

fransje26• 3 years ago

Does the memory controller also bottom out at about half the bandwidth in practice, like on the MI250? Or is the full memory bandwidth finally available?

Theoretical numbers are nice, but they are just that, ethereal numbers on a piece of paper. And on a CUDA device, I know from practice that I can get at least 90% of the memory bandwidth.

phdelightful• 3 years ago

DOE HPC people are not having this problem with AMD GPUs: here’s a paper that reports 1.36 TB/s on one GCD of an MI250x (theoretical is 1.6TB/s).

https://dl.acm.org/doi/pdf/10.1145/3624062.3624203

Table 6

MichaelRazum• 3 years ago

It costs about 20k$ as far as I understand. Why is it so expensive?

AMD MI300 performance – Faster than H100, but how much?

Related Stories