Hot Chips 34: AMD’s Instinct MI200 Architecture

https://chipsandcheese.com/2022/09/18/hot-chips-34-amds-instinct-mi200-architecture/

by ingve • 4 years ago

102 30 4 years ago

I am very thankfull that the sites exists and does those architecture deep dives, ever since Ian and Andre? left anandtech it has lacked in that territory and this site is a good replacement.

atty5• 4 years ago

Very cool to see the rundowns like this. As someone in the ML space, I’m a little concerned that they may be ceding the AI accelerator market to Nvidia. The fact it’s exposed as two separate GPUs, the focus on fp64, etc, all point to them aiming more at traditional HPC workloads than at AI. To be clear, I don’t think that’s a terrible move. HPC is a large market, and these cards look like they’re going to be great there. My concern is that there is no one really trying to challenge the Nvidia A100/H100+Cuda ecosystem. I’d love to see some better competition, both to bring down prices as well as to spur more innovation.

From what I’ve seen of Biren’s new chip it’s nice but won’t be competitive with any of Nvidia’s chips. Maybe in a few generations it’ll be more compelling, and they can build out their software stack. AMD seems to be aiming directly for HPC. Maybe Intel and their OneAPI strategy will get there? I want to root for Intel, but it’s hard to have a ton of confidence after watching their performance through the late 2010s.

cpgxiii• 4 years ago

I think, quite frankly, AMD ceded the AI accelerator market to Nvidia the day they decided not to offer consistent compute-API support throughout their product range. ROCm support is limited to a very select number of flagship products, and even flagship products from recent generations are unsupported.

The future users of your highest-end HPC accelerators start with affordable hardware they can develop and test against on their desktop. No sane developer jumps headfirst into your most expensive product (with the highest power/cooling infrastructure requirements) just to test compatibility and get familiar with your APIs.

Nvidia understands this, and you are able to run the same basic algorithms across everything from mid-range desktop GPUs to high-end HPC accelerators (performance varies, of course). Intel somewhat understands this, although has had major missteps in this area (e.g. limited AVX-512 support on desktop processors, poor developer story for Xeon Phi).

dragontamer2• 4 years ago

Reports are that the 6800XT runs ROCm pretty good now. I don't got the hardware, but it seems like it took AMD a few years to get things sorted over to RDNA / RDNA2.

ece• 4 years ago

I wonder how hard it is to just sell a GPU and say it's CUDA compatible. Google built their own toolchain over PTX, AMD could do the same, and have CUDA compatibility if they wanted. I think the difference here just might be that Google's still buying A100s, while AMD wouldn't be.

HIP/ROCm support should absolutely be better supported on all AMD hardware for more adoption, instead it seems to barely register like OpenACC or Vulkan compute. Intel might have better luck with OpenAPI.

FeepingCreature3• 4 years ago

They could still participate in this space if they bring out a 32GB card first that has ROCm support.

sroussey• 4 years ago

I think neuromorphic stuff like Rain and others will speed past Nvidia so fast it won’t make a difference. AI screams for analog.

atty5• 4 years ago

I think for inference, analogue makes a ton of sense, and within a pretty short timeline (10 years maybe?) we’ll see it deployed for those workloads.

For training, I’m certainly interested but not at all convinced that it will dominate. I work on multiple projects right now where the reduced dynamic range of analogue signals would be a complete non-starter given the problem domain. I’m not sure how they get around that.

hajile• 4 years ago

Custom AI accelerators are in the works that seem destined to beat normal GPUs in those specific workloads by quite a lot (in performance per watt).

If they were to release a design with the same architecture, but with all compute larger than f16 completely gone, they could probably get identical performance from a chip that is significantly smaller. I think both AMD and Nvidia will be releasing this kind of design (with more tweaks I'd guess) in the future.

atty5• 4 years ago

I’ve had the chance to test a few of the up-and-coming dedicated AI accelerators, and except for the TPU, none of them gave a speed up or perf/watt improvement that was worth the additional complexity of using something that didn’t natively support Cuda. (And the reason the TPU is different is because XLA is a first class citizen for TF/JAX, and has pretty good support in pytorch)

And I definitely see some future for FP16/BF16 or INT8 devices for inference, but I don’t know how widespread such devices will become for training. For many types of models, they simply won’t converge if they don’t at least use a mixed precision scheme. And there are certain problems where even for inference you get a significant benefit from using FP32 - for instance, I work on building DL surrogate models for physical simulations. We get better results with the increased dynamic range and precision.

monkmartinez2• 4 years ago

Full disclosure; I am not a pro ML'er. I dabble as a hobby and I am trying to learn as much as possible.

Question; Does the innovation need to happen on the hardware side?

I ask because there is a concerted effort, at least for inference, to reduce memory requirements and provide more access to giant LLMs.[0] It seems like everyone for the past few years has just thrown more and more compute at the wall to see what sticks. Where does it stop? 3 or 4 trillion parameter models that cost a $Billion to train? Is there a law of diminishing returns? Have we reached the apogee or will it be the 15 trillion parameter that ate $100Billion?

Accessibility to all manner of models, for knuckle heads like myself, is relatively new. Maybe I should rephrase, accessibility to interesting and FUN models that mere mortals can use and hack on is relatively new. Stable Diffusion is nothing short of spectacular for just having some fun with ML/DL.

My naïve thought is that, similar to most natural systems, smaller and more specialized models that can be "glued" together as a way forward. In my mind this makes more sense than these ridiculously large language models. That is to say, purpose built or fine--tuned models that can communicate seems like a good idea. GPT-NeoX "communicates" with to Bloom and makes a gaggle of small models that can be fine-tuned further. If regular people with a consumer GPU can train/fine-tune a model at home, this happens a lot faster.

[0] https://arxiv.org/abs/2208.07339

Tuna-Fish• 4 years ago

Your brain has roughly a quadrillion synapses. These are (in a very rough manner) the closest thing there are to parameters in the ML sense. I think you can probably do better with smarter software, but if you want to match humans on general tasks, I doubt you can do 100x better.

So there is clearly still ways to go.

atty5• 4 years ago

Beyond the comparison to the brain that the other replier gave, you also have to consider that some problems are just big. Really big. Not every AI problem is NLP or CV. For instance, in physical simulations, the representation of the system you’re simulating could be on the order of 10s of gigabytes (a very large mesh, a fine 3D grid, etc.) These are the kinds of problems I work on, and we massively benefit every time a new chip comes out with more memory or increased memory bandwidth, because a single training example, plus it’s activations, the model, and optimizer state, can take about a half a TB of memory or more depending on model size. I’d also point out that as hardware devices get better, it’s good for everyone. It allows the large corporations to train even larger, more capable models, and it allows smaller players to train models they couldn’t have afforded before on smaller budgets. A rising tide lifts all boats, so to speak. Algorithmic and efficiency improvements are also important, but they’re additive, not a replacement. And while combining models is not novel and is certainly an avenue that should and is being explored, the models that have the most generalizability for recombination are these extremely large models! They have the capacity to learn very generalized patterns that are then useful for far more transfer/combination tasks.

dragontamer2• 4 years ago

The HSA-rebirth news is pretty good. CDNA3 + Zen (Zen3? Zen4??) with shared HBM packages means extremely fast GPU / CPU communications, the likes we've probably never seen before.

easygenes• 4 years ago

Curious about workloads for these, I came across a stunning video simulating airflow around the space shuttle using 4x AMD Instinct MI250 for the computation: https://www.youtube.com/watch?v=5AzxwQpng0M

Hot Chips 34: AMD’s Instinct MI200 Architecture

Related Stories