Amazon's Trainium2 AI Accelerator Features 96 GB of HBM, 4x Training Performance

https://www.anandtech.com/show/21173/amazons-trainium2-features-96-gb-hbm-quadruples-training-performance

by ksec • 3 years ago

45 31 3 years ago

dvdbloc• 3 years ago

As someone from outside the AI space, how useful are chips like this from Amazon? Meaning am I going to have to rewrite my entire stack to make use of this and then be locked into Amazon forever? Or are workloads at least portable in some capacity?

pram• 3 years ago

There’s semi lock-in. You can use torch, but you also have to use their framework. Not much different than using cuda I guess.

belval2• 3 years ago

It's a bit of a mixed bag. NeuronSDK is implemented with Torch XLA so there's not real lock-in there. Your code will usually just work on CUDA if it works in Trainium. However for larger models (ZeRO, Sharding, etc...) they have their own libraries that are more coupled to the actual accelerator topology because using something like DeepSpeed was not possible.

Overall the challenge is often more with making your code work on their accelerator, not porting it away from their accelerators so I wouldn't be too concerned about lock-in.

yeldarb• 3 years ago

We tried Inferentia & though they claim their compiler should "just work" on your existing models it was producing a bunch of NaNs for us & was opaque enough that we didn't really have a path forward so we gave up on it.

If we were operating at 100x or 1000x the scale it might make sense to spend the extra engineering time, but we just pay for the NVIDIA chips (especially because a big chunk of our customers run at the edge where NVIDIA compatibility is even more important so we'd be doing double the engineering on an ongoing basis).

bfeynman• 3 years ago

not very useful, but it's getting better with XLA. It does not just "work" by default for any given model, and if you're missing an HLO that has an optimized kernel written you lose benefits of accelerated computing thus people won't use it. I would not put any effort into it until they truly have it where you just cast `to_device(..)`.

latchkey6• 3 years ago

AMD is coming out with the MI300x on Dec 6th, it has 192 GB of HBM3 and you put 8 of them in a chassis all connected to each other with super fast connections (5.2 TB/s of memory bandwidth) and each chassis is networked with 800-900G connections.

We don't know what the tflops will be for the 300x yet, but it seems like the limitations here are more about memory than it is about tflops.

joshhart2• 3 years ago

Will be a long time before a large number of these are available in clouds though, not until summer at least.

latchkey6• 3 years ago

I predict sooner, but that's cause it is what I'm working on now. ;-)

buildbot3• 3 years ago

Not sure of the relationship between MI300x and MI300, but MI300 is claimed to be 2.2 PFlops FP8 (with structured sparsity, which I did not know AMD was implementing!!!): https://www.amd.com/en/claims/instinct

latchkey6• 3 years ago

There are 3 versions of MI300.

X is pure gpu, A is more like an APU (cpu+gpu), C is pure cpu.

belval2• 3 years ago

Those are all impressive numbers, but did they actually fix their software stack? Are there people here that have actually trained with those and if so how was the experience?

I'm genuinely curious because it seems like every other AI Accelerator/GPU thread on HN mentions MI300X, yet I haven't talked to/met anyone who trained anything with AMD.

latchkey6• 3 years ago

The MI300x aren't released yet, that's why it isn't popular. Azure is building a cloud [0]. I know someone else personally who's building a cloud too.

As for training, it has been happening a bit on MI250's [1]. There is also A LOT that happens that doesn't make the larger news [2] and [3] and [4].

[0] https://techcommunity.microsoft.com/t5/azure-high-performanc...

[1] https://www.databricks.com/blog/training-llms-scale-amd-mi25...

[2] https://www.amd.com/en/resources/case-studies/kt-cloud.html

[3] https://www.lamini.ai/blog/lamini-llm-finetuning-on-amd-rocm...

[4] https://twitter.com/cHHillee/status/1730293330213531844

latchkey6• 3 years ago

https://news.ycombinator.com/item?id=38479616

alfalfasprout• 3 years ago

The issue with AMD (and frankly AWS' custom accelerators) is more on the software side usually. ROCM is getting better but still lags far behind CUDA in adoption and often performance.

With things like inferentia, although the performance numbers they have reported are really impressive-- in practice, model compilation is super finnicky. I'd love to see how seamless they can make usage of trainium.

latchkey6• 3 years ago

What you said gets repeated many times.

Fact is that AMD was caught out of the initial AI race and is catching up. Lisa has said over and over again that she is committed to it.

Vt71fcAqt7• 3 years ago

I hope the development of all these accelerators drives developers and frameworks to move away from CUDA only and even better to move to a open standard of some sort (eg. ROCM, if it can atain parity with CUDA in terms of usability). It would be such a waste for each framework to need to have a seperate backend for each hardware vendor. Then we will end up in the same desperate situation as game engines with Vulkan, Metal and DX12 with a side of OpenGL and Intel OneAPI (for things like eg. Blender). That sadly seems to be the way it's going right now.

joshhart2• 3 years ago

Trainium and Inferentia sadly aren't going this way, they have their own approach https://github.com/aws-neuron/transformers-neuronx . I think the best case scenario is some middleware like Triton https://openai.com/research/triton becomes the standard and we can build adapters to all of these backends. It is disappointing though.

janalsncm2• 3 years ago

In my experience AWS Nvidia machines are extremely overpriced. $30/hour for 8 A100s, and you have to pay for all 8 at a time. Most competitors I have seen are around half that price.

At $30/hour I’m literally spending a dollar or two just installing dependencies.

Whether AWS custom hardware is reasonably priced, and how much of a pain in the butt it will be to port my code, and how training speed compares to Nvidia GPUs will be my main deciding factors. I’m not locked in to AWS so Trainium will need to be faster and cheaper (cheaper per epoch) to be competitive for me.

thrtythreeforty• 3 years ago

Can't you install dependencies on a potato instance then change the instance to the GPU when you're ready to roll?

dylan604• 3 years ago

I was wondering the same thing of why not have an image ready to go, but then realized why take away the easy complaint with rational logic?

buildbot3• 3 years ago

That only helps if your dependencies are slower moving targets than whatever research/product you are working on. Otherwise, the dependencies may be different each experiment or attempt.

Or whatever cloud provider you use does something funky and the setup on one machine is less portable to a GPU machine than you might expect.

janalsncm2• 3 years ago

The $2 dependency installation cost isn’t the problem, it’s the $30/hour training cost.

cbsmith• 3 years ago

A month with an a2-ultragpu-8g is going to cost you ~$6400/month or about $36.92. Where are you getting half price?

buildbot3• 3 years ago

650 TOPs Is surprisingly low for narrow precision datatypes - I assume they are quoting the fastest performing datatype (Int8 or maybe even lower!). That's on par with an A100, an H100 hits 2 PETA TOPs (4 PTOPs 2:4 sparse!) of Int8!

teruakohatu• 3 years ago

Everyone was trying to get their hands on A100s right until the H100 was released. If Amazon can replace A100s that’s a big win.

2OEH8eoCRo02• 3 years ago

What happens to all this data center hardware when Amazon or others get rid of it? Do they sell it or do they shred it?

sumtechguy2• 3 years ago

The answer is 'yes'. It probably depends on how they accounted for it on their taxes. Some sold, some destroyed. You can usually find bunches of it on ebay. Unless it is 'custom' then it will most certainly be destroyed as there is no market for it.

2OEH8eoCRo02• 3 years ago

How do they know there's no market for their outdated in-house silicon? It's a shame that all this compute is being manufactured only to go to the shredder.

Amazon's Trainium2 AI Accelerator Features 96 GB of HBM, 4x Training Performance

Related Stories