Amazon Unveils Graviton4: A 96-Core ARM CPU with 536.7 GBps Memory Bandwidth

https://www.anandtech.com/show/21172/amazon-unveils-graviton4-a-96core-cpu-with-5367-gbs-memory-bandwidth

by mikece • 3 years ago

165 87 3 years ago

snird• 3 years ago

One of the employees in the team developing Graviton4 is kidnapped by Hamas in Gaza: https://www-geektime-co-il.translate.goog/amazon-aws-employe...

wslh• 3 years ago

I hope he is released very soon.

Also, thank you for sharing the https://www.geektime.co.il/ site, it seems it has original content and you can follow it translated in English.

ChrisArchitect• 3 years ago

Earlier: https://news.ycombinator.com/item?id=38447705

tedunangst2• 3 years ago

Furthermore, these R8g instances provide up to three times more vCPUs and memory than Graviton 3-based R7g instances.

So, uh, 75% more bandwidth but 200% more cores is a net reduction in available bandwidth per core. If you rented 32 cores of a server before, you got a certain amount of bandwidth. If you now rent the same 32 cores, you'll have about half the realized bandwidth?

krasin• 3 years ago

Memory bandwidth is becoming the ultimate spec. In related news, Apple reduced memory bandwidth in their M3 chips by 25% compared to M1/M2: https://www.macrumors.com/2023/10/31/apple-m3-pro-less-memor...

fswd• 3 years ago

I'm so glad we're seeing memory specs next to the core count, awesome.

Aurornis• 3 years ago

Memory bandwidth is becoming the ultimate spec.

Only in applications that are constrained by memory bandwidth. Many applications are not constrained by the memory bandwidth on modern architectures with significant cache.

If you have an application constrained by memory bandwidth, carefully selecting the server would make sense.

In related news, Apple reduced memory bandwidth in their M3 chips by 25% compared to M1/M2:

Not exactly. M3 Max has the same 400GB/sec memory bandwidth as the top end M1 and M2 chips. It’s only certain lower tiers that have less memory bandwidth than their equivalent tiers in previous gens.

But it probably doesn’t matter for most applications.

ChuckMcM• 3 years ago

Narrators voice: It was always the ultimate spec, it is now becoming more obvious to more people.

One of the interesting things for me is how memory "operations" (which is to say transactions of the memory controller) can completely obliterate "bandwidth."

In the past (not sure how true this is on current microarchitectures) the controller between cache and DRAM worked by opening a "page" of dynamic memory. That was an artifact of how DRAMs use both a column address select and a row address select and then multiplex the address bits. So to "read" memory you needed to select the column, then select the row, and then read the memory. The good news was that if you needed the next word of memory in order, you could just read again. And again. Periodically if you were reading the chip would have to ask you to wait while it refreshed its contents.

Anyway, this memory operation of opening a new page was a lot slower than reading the next word in memory. So if you're requests were bouncing all around memory your effective bandwidth was limited by how fast your memory controller could open new pages. That could be one tenth the nominal serial access bandwidth. Controllers had multiple "page" registers so they could hold the state of two (or more) different DIMMS and try to interleave their access across DIMMs to hide the latency aspect of page mechanics.

Generally though, when you get to the point where your measuring memops and trying to layout your physical memory to minimize them you're in a different realm of system optimization.

jjtheblunt• 3 years ago

I think Seymour Cray significantly prioritized this in his designs.

The emphasis on processor bandwidth on memory even dates back to the CDC Star.

https://en.wikipedia.org/wiki/Cray-1

charleshn• 3 years ago

It's a shame more people aren't familiar with arithmetic intensity and the roofline model [1], which give a simple mental model for performance profiles on both CPU and GPU.

I always find this graph [2] covering decades of hardware evolution fascinating.

[1] https://en.m.wikipedia.org/wiki/Roofline_model

[2] http://www.nextplatform.com/wp-content/uploads/2022/12/donga...

fulafel• 3 years ago

"Becoming the ultimate spec" is one perspective, another is "the drive toward increasingly bandwidth-starved cores starts to hit limits in some applications". Bandwidth per core has been decreasing due to increasing core counts and slowly improving per socket mem BW, and cores have gotten faster at compute.

Anyone know some example bytes per flop machine balance numbers for current and eg 20 year old systems? For older ones one source is https://www.cs.virginia.edu/stream/peecee/Balance.html - the "machine balance" for 2003 boxes seems to be between 10 and 17 there. Sadly the "MW/s" unit for machine words is not self-explanatory, what is the word size used.

GeekyBear• 3 years ago

Given that the "reduced" bandwidth is still more than double that of an Intel or AMD HEDT chip, you're still doing much better than you would with x86.

Intel and AMD both tend to choke bandwidth (memory and PCI lanes) on anything below their server chips.

seuraughty• 3 years ago

The high memory bandwidth is impressive, but you’d actually have to contrive some set of artificial circumstances to saturate it. In almost all use cases the user won’t tell the difference.

https://www.anandtech.com/print/17024/apple-m1-max-performan...

athrun• 3 years ago

apparently the 75% improvement is per core: https://news.ycombinator.com/item?id=38451909

tedunangst2• 3 years ago

Well, somebody is lying. Amazon previously claimed graviton3 had 300GB/sec memory bandwidth.

https://www.mgt-commerce.com/blog/aws-announces-general-avai...

300 x 1.75 = 525, ~536. Per chip.

If the new chip has more cores, then where is the extra bandwidth hiding?

scq• 3 years ago

That comment reads more like they're talking about compute performance, not memory bandwidth.

CuriousCosmic• 3 years ago

~~Probably however Graviton3 instances will not be going away any time soon (Grav2 instances are still around). They'll likely fill a different niche where the Grav4 instances are going to be better for larger workloads that can make use of the whole CPU whole Grav3 instances will be available for people with smaller workloads.~~

Edit: Nevermind. A reply confirms the memory uplift is per core.

phkahler• 3 years ago

> So, uh, 75% more bandwidth but 200% more cores is a net reduction in available bandwidth per core.

My guess is that most cloud customers use fairly little compute and low average bandwidth. But everyone wants "dedicated" cores so this makes a ton of sense and Amazon is raking in billions due to stupidity. Remember the recent story how a couple clicks cut the company cloud bill by a shit-ton. That's probably the norm.

pants2• 3 years ago

Ever since the Spectre/Meltdown attacks came out, it became a big liability not to do dedicated cores.

diffeomorphism• 3 years ago

200% more cores

The article says R7g is 64 cores for a full CPU and R8g is 96. Where are you getting the 200% from?

ksec• 3 years ago

The key here is 3x more vCPU. I wasn't even aware ARM had SMT design, so it is highly likely to be the first dual socket / CPU design for ARM.

That is 192 CPU per instances. I assume AWS will continue to work with AMD and provide 256 vCPU. Would love to see them being compared. ( ARM with Physical CPU core and AMD are SMT thread. )

jrflowers2• 3 years ago

I want a desktop with Graviton4/Trainum2 chips

wmf2• 3 years ago

You'll have to settle for Grace Hopper.

deadmutex• 3 years ago

Just grab a beefy x86 CPU (e.g. one based on AMD zen4) and put it in SMT=1, and you'll probably have a much better experience. A lot of windows/Linux software is already optimized for x86, and you'll get good performance uplift per logical thread from SMT=1.

stingraycharles• 3 years ago

Is this true? You mean this implies single threaded applications will perform better?

pmw• 3 years ago

Me too. The closest we can get in consumer hardware is the Ampere Altra Dev Kit: https://www.ipi.wiki/products/com-hpc-ampere-altra

It’s a Neoverse N1 architecture, whereas the new Gravitron is Neoverse N2.

It is an E-ATX form factor, and I can’t tell whether the price makes it a good value for someone who simply wants a powerful desktop rather than ARM-specific testing and validation.

nixgeek• 3 years ago

Slight correction here, Graviton 4 is actually based on Neoverse V2.

thrtythreeforty3• 3 years ago

I work on Trainium (SoC design and firmware), the main thing you would lack is the Nitro management plane. The instance is not responsible for managing the Trainium chip at all. Additionally, the platform is most desirable when it's connected via side links to other Trainium chips (the 32xlarge instance type). Even more out of band management.

Of course this could all be redesigned to be a desktop PCIe card, but the design assumption that it lives in AWS is literally baked into the silicon.

Never mind the power and cooling requirements. You probably wouldn't appreciate it being next to you while you work.

jrflowers2• 3 years ago

Never mind the power and cooling requirements. You probably wouldn't appreciate it being next to you while you work.

I would use it to make jerky

buildbot3• 3 years ago

Were any power and cooling requirements shared? Or more design info? Super curious about the details of this chip!

jwestbury• 3 years ago

Have you done any show & tells for the hardware? I miss working in Blackfoot and going to those, back when all of AWS was there. Always fun looking at S3 and EBS chassis!

notjoemama2• 3 years ago

Just curious if anyone else had the thought, “Amazon is in the chip business?” and then subsequently wondered if the regulatory agencies should step in and force Amazon to start spinning their sub-entities off into independent companies. I mean, I’ve thought that about Walmart for years. Kinda weird…Amazon making chips.

TaylorAlexander• 3 years ago

I am sympathetic to this thinking but also Google and Apple make chips too, it would be perhaps hard to make a claim for why Amazon should be split without splitting Google and Apple too, but I think Apple in particular has such a good case for making their own chips that Amazon could prob use this as a defense of their own efforts. This is perhaps more a symptom of the lower cost of entry for chip fab today than we have had in the past.

anigbrowl3• 3 years ago

I would like to see both Google and Amazon broken up. Apple I'm not sure about, they are huge and have a lot of verticals, but when you get down it they sell personal computing devices above all else and so have a stronger defense for arguing that their custom hardware is the core product. They do have a walled garden, which is a Bad Thing, but OTOH you can use their gear perfectly well without ever creating an Apple ID, and likewise you can use the whole internet perfectly well without ever purchasing an Apple device. In contrast, I have difficulty wrapping my head around the sheer scope of Google's and Amazon's enterprises, they're like entire tribes of 800lb gorillas.

giantrobot• 3 years ago

Why in the world would Amazon need to spin out their chip design group? They are building things they use to run their business. There's plenty of potential anti-trust issues with Amazon. Having their own chip designs to use in their data centers is nowhere on that list.

rescbr• 3 years ago

They should spin-off the whole of AWS, not specifically the chip group.

candiddevmike2• 3 years ago

They acquired a company that does this, Annapurna Labs

reissbaker2• 3 years ago

I don't think vertical integration should be made illegal, and "Amazon Web Services offer exclusive Amazon-designed chips" doesn't to me seem like an abuse of monopoly power any more than "Apple iPhones use exclusive Apple-designed chips" does. What is the abuse of power that forcibly splitting apart the company would solve?

wmf2• 3 years ago

I'd be worried if they had some wonder-chip with no competition that they rented out at exploitative prices but that's not really the situation.

bob10292• 3 years ago

Vertical integration can take us to dark places, but it can also provide incredible product experiences.

I think with Apple, Amazon, Microsoft and Google, you have a good competitive landscape to work with right now. All of these players already making their own chips. ~4 solid verticals that are somewhat compatible and highly competitive with each other in large, mostly-overlapping regions.

Honestly, things feel pretty OK to me when you factor it all in. If it was just Microsoft or Apple doing the vertical chip thing and they were going absolutely hockey stick over it, then perhaps. But there are several competent players doing this now.

rmccue• 3 years ago

Very annoyingly, a lot of the Graviton processors are only available on larger sizes (minimum of a .medium, but often much larger), and they’re still not filtering down to the smaller sizes. Hopefully AWS will finally start making them available some time soon.

cherioo• 3 years ago

T4g might be what you are looking for

ericpauley• 3 years ago

Yup. M7g.medium is one vCPU so by definition to go smaller you can’t get a dedicated core anymore.

anigbrowl3• 3 years ago

What happens the older chips? Do they just keeping delivering services to a lesser AWS tier, or do they get junked? I'm not much into cloud stuff and didn't even realize Amazon made its own silicon these days.

candiddevmike2• 3 years ago

Would be incredible to buy them on eBay!!

gtirloni• 3 years ago

The A1 (graviton1) instance type was introduced in 2018 and it's not available anymore for purchase, but I don't think they've forced anyone to migrate.

Customers will want to migrate anyway because the newest types offer much better performance and pricing. At which point AWS may decide there's no demand for new instances of that type and just remove it from the API.

LeifCarrotson• 3 years ago

Well sure, you can remove the digital asset from the API and it's completely gone, but what are they doing with the physical asset?

Selling them? Running other services on them? Just keeping them in reserve? Or all turned into e-waste?

sesuximo• 3 years ago

I don’t know for sure, but many companies use old hardware for less intense tasks like being a cpu attached to a massive disk for cold storage

pwarner• 3 years ago

In the past they used to become cheaper at least via spot pricing

SoftTalker• 3 years ago

Lifecycle replacement on a server is 5-6 years. At that age they aren't competitive for the power consumed, and failures become more common.

Now, I manage many servers at my job that are much older than that, but only because the operations budget and the capital expenditures budget do not take this into account.

iszomer• 3 years ago

They get recycled and put back into inventory as spare parts for existing SKUs they currently have deployed. And just fyi, Annapurna Labs (creator of the Graviton family) was acquired by Amazon AWS back 2015 iirc.

bee_rider• 3 years ago

I wonder how they’ll compare to Ampere’s chips. I hope Ampere holds on, it is nice to have chip designer that primarily, like, designs and sells chips as a business model.

rawrawrawrr• 3 years ago

Fun fact, a Nvidia RTX 4090 has ~1TBps memory bandwidth, for an apples to oranges comparison.

LispSporks22• 3 years ago

Is that up there with the z16? I'm a mainframe n00b still, but it seems like some z16 configuration at least would be.

bob10292• 3 years ago

I think we are talking about entirely different classes of hardware.

The z16 is designed to run applications that cannot fail. In some cases, the datacenter is expressly built around these systems to accommodate their unique capabilities.

The Graviton line is designed to run as many applications per unit of volume and power possible for a multi-tenant, hyperscale ecosystem.

If a CPU core goes bad on a Graviton chip, you would likely need to redeploy your EC2 instance (or it would be done for you automatically). This would almost certainly have some downtime for your application. In the z16, if an entire CPU goes out, you hypothetically won't drop a single transaction if you followed IBM's guidance.

joebocop• 3 years ago

Norton Anti-virus going to grind this thing to a halt

Amazon Unveils Graviton4: A 96-Core ARM CPU with 536.7 GBps Memory Bandwidth

Related Stories