Ten Lessons from Three Generations Shaped Google’s TPUv4i [pdf]

https://www.gwern.net/docs/ai/2021-jouppi.pdf

by djoldman • 5 years ago

108 15 5 years ago

Explain it like I'm five? :(

bsder• 5 years ago

Power is a problem and is scaling far less than the node scaling. Raw single threaded performance basically isn't scaling at all.

Going from 45nm to 7nm we only gained about a factor of two in terms of power consumption. Most of that is due to the fact that interconnect and SRAM aren't scaling.

The upshot is that it is worth doing almost infinite amounts of calculation in order to avoid moving anything from point A to point B in a compute unit. If you must use memory, DRAM is death so put big amounts of SRAM on chip and make the buses as wide as possible.

cokernel_hacker• 5 years ago

The conclusions section of the paper is a good summary:

"In the process, we learned ten lessons about DSAs and DNNs in general and about DNN DSAs specifically that shaped the design of TPUv4i:

1. Logic improves more quickly than wires and SRAM ⇒ TPUv4i has 4 MXUs per core vs 2 for TPUv3 and 1 for TPUv1/v2.

2. Leverage existing compiler optimizations ⇒ TPUv4i evolved from TPUv3 instead of being a brand new ISA.

3. Design for perf/TCO instead of perf/CapEx ⇒ TDP is low, CMEM/HBM are fast, and the die is not big.

4. Backwards ML compatibility enables rapid deployment of trained DNNs ⇒TPUv4i supports bf16 and avoids arithmetic problems by looking like TPUv3 from the XLA compiler’s perspective.

5. Inference DSAs need air cooling for global scale ⇒ Its design and 1.0 GHz clock lowers its TDP to 175W.

6. Some inference apps need floating point arithmetic ⇒ It supports bf16 and int8, so quantization is optional.

7. Production inference normally needs multi-tenancy ⇒ TPUv4i’s HBM capacity can support multiple tenants.

8. DNNs grow ~1.5x annually in memory and compute ⇒ To support DNN growth, TPUv4i has 4 MXUs, fast onand off-chip memory, and ICI to link 4 adjacent TPUs.

9. DNN workloads evolve with DNN breakthroughs ⇒ Its programmability and software stack help pace DNNs.

10. The inference SLO is P99 latency, not batch size ⇒ Backwards ML compatible training tailors DNNs to TPUv4i, yielding batch sizes of 8–128 that raise throughput and meet SLOs. Applications do not restrict batch size."

ArtWomb• 5 years ago

>> 8. DNNs grow ~1.5x annually in memory and compute

Wow! That's a massive growth rate for ML. TPUv3 was already faster than A100 in MLPerf. But this suggests a real breakthrough is needed to keep pace with future requirements. Each MXU already handles 16k ops per tick. And with the additional constraint of optimizing per watt rather than dollar, its quite the challange ;)

codemac2• 5 years ago

How does George Kurian get listed as an author on these TPU papers? I'm seriously confused.

ralph84• 5 years ago

The same way professors get named on papers where their students did all the work?

Marat_Dukhan• 5 years ago

He's a Software Engineer on the TPU team. Are you confusing him for Thomas Kurian, GCloud SVP?

Note: I work for Google, but speak for myself.

ignoramous• 5 years ago

To add to the confusion, NetApp CEO George Kurian is Thomas' twin brother: https://archive.is/jxtu6

taktoa• 5 years ago

different guy

sillysaurusx2• 5 years ago

Fun fact: this pdf was paywalled, but thanks to some generous members of my Twitter, it was able to be freed. I look forward to a day when all such work is available to anyone who wants it.

Normally I’d have something else to add, but I feel down lately, and it’s nice to see that I’m able to help with one or two things that matter.

I was mostly surprised that the paywall model still persisted in 2021.

throway453sde• 5 years ago

To be clear visiting the posted link to "gwern.net" does not ask for pay to view the pdf.

sillysaurusx2• 5 years ago

Yeah, sorry for the confusion. I asked gwern to mirror the liberated pdf, since his /docs/ folder will probably be around a lot longer than me. https://www.gwern.net/docs/index There's a lot of neat stuff in there.

By the way, if any of you happen to run into paywalled pdfs, try posting the paywall link to https://reddit.com/r/scholar. It's surprising how often (and how quickly) that gets you the pdf.

londons_explore• 5 years ago

Parts of this document seem like they came from meeting notes copied to latex and not really re-edited...

Ten Lessons from Three Generations Shaped Google’s TPUv4i [pdf]

Related Stories