JEDEC publishes GDDR7 graphics memory standard

https://www.jedec.org/news/pressreleases/jedec-publishes-gddr7-graphics-memory-standard

by ksec • 2 years ago

152 105 2 years ago

paddy_m• 2 years ago

How much relevance does JEDEC still have?

I would think NVIDIA in particular and other chip makers/integrators like apple make up their own standards now. It also seems less relevant because memory is rarely interchangeable anymore

sliken4• 2 years ago

Do you have any evidence that Nvidia or Apple are using non-standard memory chips? Link? From what I can tell they are both using standard chips, but very wide memory interfaces. Apple's lowest end is 128 bit, but offer 256, 512, and 1024 bit wide memory interfaces for more bandwidth, which is mostly a benefit for the iGPU in all of Apple's m series CPUs. This is part of why Apple are pretty good at LLMs, especially those needing more ram than is in even the most expensive GPUs.

Sad that the vast majority of x86-64 laptops and desktops have the same bus width of decades ago, while the core counts are ever increasing.

jsheard• 2 years ago

Nvidia and Micron came up with GDDR6X, which isn't a JEDEC standard. JEDEC did standardize GDDR5X before that, but only Micron ever made it and only Nvidia ever used it AFAIK.

zeusk• 2 years ago

JEDEC never "standardized" GDDR6x that Nvidia does use; Micron and Nvidia worked closely on both GDDR6x and GDDR5x

kllrnohj• 2 years ago

Apple's lowest end is 128 bit, but offer 256, 512, and 1024 bit wide memory interfaces for more bandwidth

Apple doesn't have any 1024-bit offerings. The Ultras are 2x 512-bit which is different from a true 1024-bit since it's a NUMA configuration (two different memory controllers).

Sad that the vast majority of x86-64 laptops and desktops have the same bus width of decades ago, while the core counts are ever increasing.

The consistent trend is CPU performance on consumer workloads is still much more sensitive to memory latency than it is bandwidth. This is probably also why those 512-bit M1/2 Max's don't even bother to let the CPU make use of it, they top out at "just" 200GB/s memory bandwidth even though there's 400GB/s available to the SoC as a whole.

Being able to pair a lot of DRAM with the GPU of an M* Max/Ultra is definitely a currently unique perk, but the bandwidth numbers available to those GPUs are not actually all that special. Desktop GPUs passed 400GB/s mark way back in 2016, and are currently pushing 1TB/s in modern flagship offerings.

hedgehog• 2 years ago

Apple and NVIDIA are both members of JEDEC...

monocasa• 2 years ago

HBM is a JEDEC spec these days. Apple's on package memory is still LPDDR.

ajross• 2 years ago

JEDEC isn't a memory technology monopoly or anything, they're just a standards organization. You have a situation where lots of companies need to make products that interoperate, but interoperation is complicated in electrical engineering. There are a *lot* of ways to get a memory interconnect wrong.

So the solution is to pick (or create) a separate, notionally independent body staffed and supported by representatives of all the relevant stakeholders, and have them write "standards" that everyone agrees to adhere to. The body doesn't invent the technology, that happens at the individual chip companies. They then present their proposals to JEDEC[1] and everyone argues and agrees on what will go into GDDR19 or whatnot. And JEDEC then publishes the standards for all to see.

[1] Or whoever, JEDEC does DRAM, but there's a USB Consortium, Bluetooth SIG, WiFi is under IEEE, etc...

pillusmany• 2 years ago

Standardized memory chips allows economy of scale to work.

Dalewyn• 2 years ago

How much relevance does JEDEC still have?

Ah, a FOSSy dev insistent that his bits are gibi- and mebi- and kibi-bytes.

ryukoposting• 2 years ago

The physical memory modules might not be interchangeable, but JEDEC is still extremely important to hardware and firmware engineers. It gives us a baseline - we know anything that says "JEDEC" will play nice with our SoC's QSPI/DDR peripherals.

paulmd• 2 years ago

NVIDIA has basically been doing that with GDDR5X and GDDR6X already - they are "official standards" but nobody besides NVIDIA actually uses them.

sliken4• 2 years ago

Doubling the channels from GDDR6 sounds good, the speed of light isn't changing, so at least we can handle more parallelism with the same latency.

oorza2• 2 years ago

The ratio of light speed to the area of the universe is so stupidly small that I'm convinced our simulation is determining how low the speed of light can get before interstellar travel is outright impossible.

adtac4• 2 years ago

With sufficiently advanced technology, travelling between stars will be more of a transfer of your consciousness using interstellar WiFi (remember to set TCP_NODELAY!) rather than transporting slow and heavy atoms. You just get transferred from one biological substrate to another. None of the time dilation, all of the space exploration.

hnuser123456• 2 years ago

Or is it the ratio of your lifespan to the age of the universe? The universe is only about 3x bigger instantaneously than it is when transversed at lightspeed. The ratio of the age of the universe to your expected lifespan is about 8 orders of magnitude.

ko27• 2 years ago

"Ratio of light speed to the area of the universe" does not determine how far you can travel in a set amount of time, because time dilation exists.

tutfbhuf• 2 years ago

"speed of light can get before interstellar travel is outright impossible"

Time is relative to your frame of reference. If you travel to Proxima Centauri at 99.99% c, it will take you 22 days from your point of reference sitting in the spaceship, which is quite acceptable. On Earth, 4.24 years have passed. So, your family and friends grow old quite fast when you do interstellar travel without them; hence, it's better to take them with you.

znpy2• 2 years ago

Just checking my intuition: could we still get a speedup via pipelined execution and branch prediction?

From what i see reading https://en.wikipedia.org/wiki/Multi-channel_memory_architect... different channels could, in theory, be used "autonomously of each other".

Lramseyer5• 2 years ago

Controllers kind of do that. At the end of the day, it's what makes designing a memory controller so difficult (and I'm not even talking about the Phy, those things are straight up cursed!) We see these eye popping numbers for maximum potential bandwidths, but the reality is a bit more complicated. There's a lot that goes on behind the scenes with opening and closing memory banks, refreshes, and general read and write latencies. Unoptimized prediction algorithms (as they are programmable) can result in losing _half_ of your performance.

sliken4• 2 years ago

Pipelining helps, usually. You can have 10 stages run at 3 GHz instead of 1 stage at 300 MHz. Up until you have a branch misprediction and have to dump the pipeline.

Memory channels are independent, however generally all cores use all channels. There are two common designs. Intel has 4 dies, 2 memory channels per core, so 2 channels are closer/lower latency than the other 6. AMD has multiple chiplets, but a single memory controller with 12 channels. So All cores have the same latency to all channels.

Generally Intel has lower latencies to 25% of the channels, but AMD has more throughput (bandwidth or random IOPs).

One thing that surprised me is that for maximum throughput you want at cache misses queued to the memory controller, at least twice the number of memory channels. These days missing in L1/L2/L3 is often approximately half the total memory latency. So on an Intel Xeon at least 16 misses (per socket), AMD at least 24 misses (per socket.).

So on Intel you could tune things (and the NUMA support helps) to prefer the local channels. Most OSs help, and C calls like numa_alloc_local() allows local control.

For memory intensive codes I have found the best scaling when there's 2 cores per channel. Of course most codes are pretty friendly.

colechristensen• 2 years ago

the speed of light isn't changing

Oh but the speed of the signal does depend quite a lot on the transmission medium. In Cat-6 signals travel 2/3c. Can't find a quick reference for on-die or motherboard kinds of interconnects. If you had optical interconnects traveling through vacuum in a silicon chip, that's a full 50% faster (as in lower travel time for one bit over a distance) than Most copper ethernet.

https://en.wikipedia.org/wiki/Velocity_factor

lloeki2• 2 years ago

Oh but the speed of the signal does depend quite a lot on the transmission medium

And this (actually, phase velocity) is what makes refraction a thing.

I knew that we could slow down light to subsonic speeds, but TIL we can put it at a complete standstill! Amazing.

https://en.wikipedia.org/wiki/Speed_of_light#In_a_medium

Lramseyer5• 2 years ago

with the same latency

I highly doubt that. With on-die ECC and the ridiculously complicated PAM3 encoding/decoding, I would bet that latency is going to increase over GDDR6.

eigen• 2 years ago

as the press release says PAM3 sends 3 bits in 2 bit times whereas NRZ would require 3 bit times to send 3 bits. that, coupled with the increased WCK for shorter bit times, suggests latency shouldn't necessarily increase.

I assume by "the ridiculously complicated PAM3 encoding/decoding" you are referring to section 2.9.3,

"The total burst transfer payload per channel is encoded using 23 x 11b7S and 1 x 3b2S for the data, 6 x 3b2S for the CRC and 1 x 2b1S for the SEV/PSN, it adds up to the 176 PAM3 symbols that can be allocated for a 16 burst over 11 data lines."

that does seem complicated using 3 (11b7s, 3b2s, 2b1s) different modulation schemes in 1 burst.

Scene_Cast2• 2 years ago

One interesting thing to note is that all the high speed interconnect standards (GDDR, PCIE, USB, Ethernet) are moving (or have already moved) to RF-style non-binary communication (as opposed to bumping up clock speeds or increasing pin count). I wonder what the next steps will be for interconnects - full-blown baseband style transceivers with QAM perhaps?

wmf3• 2 years ago

Optical communication is using QAM so that's probably the next step.

klysm• 2 years ago

Sharp signal edges are hard to get at higher frequencies - it seems quite natural

SmellTheGlove• 2 years ago

This is probably a stupid question but I don’t claim any knowledge here. Even though interconnect standards are moving to non-binary communication, doesn’t the signal eventually need to be converted back to a bitstream at its destination? Does this just push the bottleneck around, or do I just not understand the problem being solved? It’s almost certainly the latter and I’d love to understand more.

kevvok• 2 years ago

You’re right that the signals have to be converted back into bits at the destination. Basically, this solves the problem of pumping those bits at high speed across traces on a circuit board vs within a chip. The longer a signal has to go, the harder it is to maintain its integrity.

wmf3• 2 years ago

A serializer/deserializer (serdes) is used to convert between high-speed serial I/O outside the chip (e.g. 100 Gbps) and lower-clocked parallel signals inside the chip (e.g. 64 bits at 1.5 GHz). Using serial protocols reduces the cost and thickness of cables while parallel wires are cheaper inside the chip.

ak217• 2 years ago

Not a stupid question - you can think of the problem by analogy with RF engineering. You have very high performance digital logic and precise clocks on the chip that you can use to encode/decode (convolve/deconvolve) bits into waveform signals and time those signals before they leave the chip at minimal latency/power expense. Once the bits are off the chip, you have no such resource and are dealing with all kinds of impedance and noise issues, which is why there are separate circuits/logic dedicated to training and calibration of the encoding parameters of the signals sent over the wire in DRAM chips.

This more complex encoding scheme is just the next level in that process, indeed moving it closer to techniques used in RF engineering.

lpmay2• 2 years ago

The important distinction is between on-silicon interconnects and off-silicon. On silicon, the interconnect density is much higher so it's feasible to get high bandwidth by having lots of "channels" (wide busses). This becomes less desirable off chip (on the PCB) for many reasons. Some of them are: 1) The interconnect density is much lower, and having such a large bus becomes physically large 2) The bus is larger and less precise, which makes maintaining skew between channels more and more challenging 3) The parasitic capacitance/inductance (i.e. energy storage) of the bus becomes larger as it gets physically larger, meaning each channel needs a relatively large driver circuit (which costs expensive silicon area) and dissipates more power to drive correctly, and even more power if the speed is increased.

Increasing the symbol complexity of each channel does more than just move the bottleneck around, because it allows fewer chip to chip interconnects to carry more data.

I don't work in this regime, but as a layman I'm not convinced using full QAM for on-board chip to chip interconnects makes sense. One major advantage you natively have over the RF case is you can be easily coherent (shared clock). Throwing this away to do carrier recovery introduces a lot of complexity and potentially reduces the available bandwidth. Assuming you transmit without a carrier, can you have "baseband" QAM without a separate I and a Q signal? If you transmit an I and Q signal separately, does that not just become the same thing as two PAM-32 signals?

2OEH8eoCRo0• 2 years ago

At the physical layer, digital/binary doesn't exist. It's all signals.

kromem• 2 years ago

Technically, it's the opposite - all measurable physical things are only measurably digital (i.e. measurable in discrete units). But for all practical purposes, that threshold is low enough that it doesn't really matter for current signals transport standards.

bunnie• 2 years ago

Generally the limiting factor for approaching the asymptote you observe is power. There is a figure of merit with units of Joules per bit per second that gets worse - much worse - as you start to get fancier with your modulation.

Most of the fancier schemes take advantage of the fact that traditional binary signalling has excess noise margin, ie, they were throwing away energy to start with, and they are encoding extra bits in the energy budget. But to maintain noise margins as you cram more bits in, you have to up the amount of energy per bit.

The other half is that the physical layer implementation that does the encoding and decoding consumes more energy because it's doing more computation. This also figures into the energy per bit metric if you're being honest about your comparisons (and because it is not always clear if this is included in a metric you find papers where people cherry pick numbers to make their case). This number can become quite big because the baseline of a binary tx/rx is so low compared to doing effectively a DAC/ADC and phase recovery system.

What you find is that QAM or more schemes are certainly possible, but they can consume more power than the CPU just to keep the link idle and trained. The real art is picking the implementation and developing new circuit tricks that we hadn't thought of before to wring a little more bandwidth without killing the power budget.

redox99• 2 years ago

Would power actually be an issue, for the 200w+ GPUs these memories will likely be used with?

bgnn2• 2 years ago

10G-baseT ethernet uses 128-DSQ modulation (LDPC coded). It's 128 symbols sent over 2 cycles.

These complex modulations have a huge drawback though: latency. You need to pack, ramdomize, encode and modulate at the transceiver, and undo all these + equalization at the receiver. Especially feed-forward equalization is a huge latency source.

silvestrov• 2 years ago

Some numbers here: https://wavesplitter.com/technical-column/latency-and-power-...

Cable latency is about 5ns/m, so 2.6 µs is equivalent to latency of a 520m cable. This latency might become a big issue for the HPC-based applications.

crotchfire• 2 years ago

I wonder what the next steps will be for interconnects

PCB manufacturers start offering FR-4 with strands of single-mode fiber embedded in it?

imtringued• 2 years ago

Unless Nvidia can somehow massively increase memory capacity, it is looking bleak for them in the consumer AI inference space. From the left Apple has a fully integrated SoC with insane memory bandwidth and capacity, from the right AMD is tackling the FLOPs advantage using their XDNA AI Engines that they got from their Xilinx acquisition and they are going to open source the compiler for their AI Engines. The only competitive advantage that NVidia has left is its high memory bandwidth, but even that is being threatened by Strix Point so they will need to adopt GDDR7 with 32 GB to 64GB VRAM fast or they will become irrelevant except for training. Oh and by the way AMD GPUs will stay completely irrelevant for AI so that explains why they didn't want to waste so much time on ROCm for consumer GPUs. Nobody is going to buy those for AI anymore by the end of the year.

rubatuga• 2 years ago

AMD was the first to introduce consumer HBM cards.

account42• 2 years ago

Well, on "consumer" cards that were AMDs desperate attempt to stay relevant in that space by rebadging their enterprise compute-focused design for consumers. They haven't used HBM in any of their actual consumer designs unfortunately.

IshKebab• 2 years ago

I mean... CUDA. nVidia is fine.

lloeki2• 2 years ago

Too big to fail? Sure, they have an edge in being the current de facto go to solution, but that can be changing very rapidly if they rest on their laurels.

wmf3• 2 years ago

It's not clear that the "consumer (local) AI inference space" is a real market. Ultimately Nvidia has access to all the same technologies as their competitors and better software so anything they can do Nvidia can do better... if they want to.

mschuster91• 2 years ago

Unless Nvidia can somehow massively increase memory capacity, it is looking bleak for them in the consumer AI inference space.

Who cares about consumer inference? The money is in training because that's where utterly insane amounts of compute capacity are needed, and as long as no competition comes even close to CUDA, NVIDIA has a cash cow to milk.

As for potential competition, Apple doesn't sell their silicon to anyone else, and AMD lacks the available manufacture capacity (keep in mind, they also supply the game console market), the driver/tooling quality and most importantly developer trust/ecosystem quality. Everyone is using CUDA.

coffeebeqn• 2 years ago

Is there any reason they couldn’t make a card with lots of GDDR5 for example? Isn’t it “just” memory chips that you could put a lot of on a board ?

adgjlsfhk1• 2 years ago

they do. the reason Nvidia doesn't do it for consumer cards is that they do it for their server cards and raise the price 40x.

Lramseyer5• 2 years ago

Just got my hands on the spec and read through it for a bit. A couple interesting things I noticed that were notable changes from GDDR6.

Obviously, the big news is PAM3 signaling, and on die ECC. These aren't all that new, as NVIDIA's GDDR6X used PAM4 signaling (at lower frequencies than traditional GDDR6) and an unnamed DRAM vendor had GDDR6 DRAMs with on die ECC, though at the cost of having annoyingly high read and write latencies. Thankfully, only the DQ (data) pins use PAM3 signaling.

I'm going to do my best to explain how this works in GDDR7, since I need to understand this for my work:

If you don't know what PAM3 is, it stands for "Pulse Amplitude Modulation, 3 levels" Traditional communication could be thought of as PAM2, since there's a level for 0 and 1. But we usually call it NRZ for "Non-return to Zero" There's a bit of nuance, since not all binary data communication is NRZ. But that's a different discussion. Now you might ask, why PAM3 and not PAM4? With 4 levels you can transfer 2 bits, and that seems much easier to work with. Well, it's because we hate ourselves, and we hate you. That's why.

For context, a GDDR6 channel uses 16 DQ (data) pins + 2 EDC (Error Detection/Correction) pins and 16 transfers for 256 data bits and 32 CRC bits transaction. GDDR6X (from my understanding) does the same thing, but since it's PAM4, it sends 2 bits per transfer with (I think) half the number of transfers.

GDDR7 on the other hand has 11 DQ pins of PAM3 signaling. Since PAM3 is 3 state {-1, 0, 1} these are defined as "symbols" or "trits" (I really hate the word "trit".) These symbols have are 3 separate encoding methodologies (yikes) that are used in this protocol: 11b7S, 3b2S, 2b1S. These essentially determine how many bits of data you can encode in a given number of symbols. 11b7S means 11 bits of data encoded in 7 symbols. 11 bits of data has 2048 unique combinations, 7 symbols have 2187 unique combinations (3^7). 3b2S is 3 bits (8 combinations) encoded in 2 symbols (9 combinations). And 2b1S is a misnomer but it's application specific to the Poison and Severity flag bits and the Severity flag takes precedent, so the unrepresented combination is invalid.

Like GDDR6, there are still 16 transfers, but the DQ bus width is reduced to 11 DQ pins, and a parity error pin. With this, we get a total of 176 symbols per transaction. When decoded, this allows us to send 256 bits of data, 2 bits for Poison/Severity flags (this is the 2b1S symbol), and 18 bits of CRC. Encoded though is a bit of a doozy: 163 symbols for the data, 1 symbol (2b1S) for the Poison/Severity flags, and 12 symbols (3b2s) for the CRC. If that's not confusing enough, The 163 data symbols don't even all use the same encoding. The first 161 symbols encode 253 bits in 23 sets of 7 symbol to 11 bit (11b7S) sets, for 23 sets of 11 bits. The remaining 3 bits a are encoded in the remaining 2 symbols as a 3b2S set.

How this is mapped to each pin is outlined in the spec. I'll take a look at that part later, as I have had enough unfriendly math for the day.

On a different note, there were some other notable changes that I noticed:

They also shrunk the CA (Command Address) bus width from 10 pins down to 5 and run it twice as fast relative to the CK and WCK clocks. Instead of 2 cycles of 10 bits, it's 4 cycles of 5 bits. The 5 bits are split into "Row" [0:2] and "Column" [3:4] bits. It was a bit weird at first glance, but it's actually kind of nice. The commands make a lot more sense now. Interestingly enough, it looks like the CABI (Command Address Bus Inversion) and CAPAR (Command Address Parity) are part of the 20 bit CA command now. Though I'm not sure how useful a CABI is if it's once every 4 cycles. Not sure how much power you're really saving.

There are 64 mode registers instead of 16. Still 12 bit wide though. Wait WTF? This (kind of) made sense in GDDR6, since the address and the data fit nicely into 16 bits (4 address, 12 data) and you could use the other 4 bits as the command ID for MRS. Now we have 12 bit wide registers, 6 bit wide addresses, and the MRS command is a double length command because it doesn't use the column bits. This is really weird. Registers 0-31 are defined by the spec, 32-47 are reserved for future use, and 48-63 are Vendor Specific.

Found this funny - There's a feature for addressing clock drift between CA and WCK clocks due to varying Voltage and Temperature. It's called the "Command Address Oscillator" I wonder why they picked that name... "There is a CAOSC associated with each channel and operates fully independent of any channel’s operating frequency or state" Someone had fun with that one!

Other Quick blurbs:

- No mention of Quad Data Rate (QDR)

- CTLE looks like it's supported from the DRAM side.

- They added an RCK (read clock) pair, and is generated on the DRAM side.

- They added a static data scrambler! You don't know how happy I am to see this!

Dylan16807• 2 years ago

Now you might ask, why PAM3 and not PAM4? With 4 levels you can transfer 2 bits, and that seems much easier to work with. Well, it's because we hate ourselves, and we hate you. That's why.
For context, a GDDR6 channel uses 16 DQ (data) pins + 2 EDC (Error Detection/Correction) pins and 16 transfers for 256 data bits and 32 CRC bits transaction. GDDR6X (from my understanding) does the same thing, but since it's PAM4, it sends 2 bits per transfer with (I think) half the number of transfers.

GDDR6X did not transfer 2 bits at a time despite being PAM4, because they disallowed transitions between the lowest voltage and the highest voltage.

Because of that each group of 4 symbols has 139 easily-stackable sequences, so they went with 7b4S.

https://research.nvidia.com/publication/2022-04_saving-pam4-...

I couldn't tell you how each transaction is laid out.

Lramseyer5• 2 years ago

Oh, interesting! I remember hearing rumblings that PAM4 was kind of a pain in GDDR6X. This makes sense. I still stand by my claim about PAM3 though.

Sparkyte• 2 years ago

I just want memory that can dynamically change on the basis of workload so it can be great for graphics and great for system performance.

mistyvales• 2 years ago

Still interested to see if a newer gen of HBM will ever be on retail GPUs again

JEDEC publishes GDDR7 graphics memory standard

Related Stories