AMD EPYC 9004 Genoa Under-the-Lid – ServeTheHome

https://www.servethehome.com/amd-epyc-9004-genoa-under-the-lid/

by rbanffy • 3 years ago

46 23 3 years ago

servethehome.com

tpmx2• 3 years ago

Under the lids is.. another layer of lids?

rbanffy• 3 years ago

It’s lids all the way down. Still, quite an impressive number of them.

actionfromafar• 3 years ago

Each pair of lids make up a NAND gate.

pclmulqdq3• 3 years ago

Those "lids" are the backs of silicon wafers, so not actually "lids." The silicon chips are flipped over to attach to the surface of the blue PCB there.

tpmx2• 3 years ago

Thanks for the explanation (that the article writer clearly didn't know about)!

nullc• 3 years ago

Where is Genoa-x? moooarr cache plz.

Patrick-STH2• 3 years ago

Later in 2023. We will cover it like we did with Milan-X.

Idiot_in_Vain• 3 years ago

Hi Patrick :)

rwmj• 3 years ago

Does anyone know how they place the chiplets on to the blue substrate? It must be incredibly precise, presumably?

bfrog• 3 years ago

Maybe, I wonder if this can work like ic placement on pcbs where surface tension can help align things (solder will pull parts into place)

I too would love to know as well!

XzAeRosho• 3 years ago

Most of those techniques and machinery is behind company secrecy, but yes, you need highly precise and specialized machinery to build that kind of things.

Linus Tech Tips did a tour of the Intel fab factory a few months ago: https://www.youtube.com/watch?v=2ehSCWoaOqQ

tyingq• 3 years ago

This doc has some info about current pitch size for the "bump map" of where the chiplets connect to the substrate.

https://www.opencompute.org/documents/odsa-openhbi-l-v1-0-rc...

Page 38 says "OpenHBI-L specifies the minimum bump pitch (min) to range from 110um to 130um for manufacturability considerations".

Edit: Also related and interesting...

"One of the key points AMD spoke about...is how each of the silicon dies are attached to the package. In order to enable a pin-grid array desktop processor, the silicon has to be affixed to the processor in a BGA fashion. AMD stated that due to the 7nm process, the bump pitch ...reduced from 150 microns on 12nm to 130 microns on 7nm. This doesn’t sound like much, however AMD stated that there are only two vendors in the world with technology sufficient to do this."

https://www.anandtech.com/show/14525/amd-zen-2-microarchitec...

lizknope• 3 years ago

I'm a digital physical design engineer.

Even though the 7nm number is kind of meaningless these days it is still useful to compare the scale.

The flip chip bumps on the top metal layer interface to the outside world. Then they get distributed out to the pins of the package which connect to the socket on the motherboard.

Socket AM4 has 1331 pins. Socket SP5 for the Epyc has 6096 pins. I've worked on some chips with over 10,000 bump/pins.

A modern chip has over 16 metal layers, with multiple via layers in between and probably another 40-60 base layers with the transistors. The technology to align the masks for each layer is on the nanometer level. In contrast the bumps for the chip to package are in the hundreds of micron level.

If we just go with 130 microns / 7 nanometers = 18,571 less precise for the alignment of the package vs alignment of the next mask layer while making the chip.

So it is certainly impressive to line up chips on the package but nowhere close to the precision for the mask layers

pclmulqdq3• 3 years ago

The technology to do this is pretty old, and it's basically just a very precise pick-and-place machine (disclaimer - this is for 3D stacking, not necessarily AMD's thing). The rub is that it doesn't need to be that precise: surface tension from the soldering process will actually correct a small mis-placement as long as you aren't off by too much, so the tolerances are bigger than you think.

bob10293• 3 years ago

Looks like ~$12k per unit. Not even half a year of our cloud spend. The economics around this are pretty hilarious to me now.

If we were to consolidate all of the processing power across our Azure/AWS VMs, we would not be at even 30% of what this part can handle. You could even throw in the IT stack of our entire parent org with room to spare.

The only reason we are still in the cloud is the deep compliance packages that our customers really enjoy. Even so, management is starting to consider alternatives.

I know a lot of us can't stand the idea, but the notion of putting the entire company on 1 socket is super compelling to me. On-prem turns into a dragon for me when I have to manage 10+ machines and related vendors. 2-3 machines and a handful of vendors is more manageable.

One thing is certain - The various cloud providers will take every ounce of AMD's performance innovation and make it feel like a negligible improvement to their end customers.

unethical_ban• 3 years ago

Accounting: Hardware and software acquisition is very different at an accounting level in large orgs, and come from different budgets. Licensing and support can be easier to procure and plan for than hardware.

APIs/RBAC: Cloud infrastructure-as-code for block storage, object storage, DB, network, VM and serverless with the same security policy language accessible via HTTP is really incredible. I wish there were an open-source cloud equivalent, but I don't know of one that can do 20% of what AWS can do.

bob10293• 3 years ago

I don't know of one that can do 20% of what AWS can do.

We only use a few AWS products: Route53, EC2, VPC, EBS, and S3/Glacier. This feels like <20% of what AWS can do today.

Are these use cases not yet covered? I'd be OK with Route53 not being in the box, but surely we have VMs, storage and backups sorted by now, right?

pclmulqdq3• 3 years ago

Keep in mind that you need to put this in a server with DIMMs, power supplies, and disks. It will likely come out to $20-30k all in for one server depending on how much margin your system integrator wants. You probably want two in case the first one fails, and you may want as many as 3-4 for true 24/7 availability, plus a backup service.

The math still doesn't look too good for the cloud, though.

gary_0• 3 years ago

putting the entire company on 1 socket

What about availability? Do you mean 1 socket, plus a hot backup in a separate location in case of hardware failure or some catastrophe?

IIRC, Hacker News runs on 1 machine plus a hot backup.

justsomehnguy• 3 years ago

Most of the time 'availability' means network connectivity. Most of the time you can throw resources in the processing.

AMD EPYC 9004 Genoa Under-the-Lid – ServeTheHome

Related Stories