Cloud TPU Pods Break AI Training Records

https://cloud.google.com/blog/products/ai-machine-learning/cloud-tpu-pods-break-ai-training-records

by jonbaer • 7 years ago

42 46 7 years ago

cloud.google.com

Is it me or are those results somewhat underwhelming if anything? Dedicated hardware for a 2x speedup at best, tossup for most results, and only competes in some categories. Not to be just a NVIDIA fan here, surely there is value in dedicated training hardware, but just surprising that benefit isn't bigger!

totoglazer• 7 years ago

I think it’s probably because the benchmark isn’t optimized for TPU Pods. Check out the BERT in 76 minutes paper for how you need to rethink the training regime to take advantage of pods.

zak9• 7 years ago

Yes, Cloud TPU Pods are designed to train much larger models on much larger datasets. And, as you mention, if you are willing to adjust your model architectures and training algorithms to take full advantage of the hardware, you can sometimes achieve substantial gains.

boulos3• 7 years ago

Disclosure: I work on Google Cloud (even with Zak sometimes).

The DGX-2h is a beast! Don’t parse this as “huh, TPU Pods are about the same as just a few V100s”. The data sheet [1] is probably the easiest to follow, but their writeup is more informative [2].

These are souped up V100s, with awesome networking, which is pretty similar in style to a TPU Pod. So I’d say that they’re both purpose built systems for distributed ML training. The name for the NVIDIA system is even “DGX SuperPOD” :).

[1] https://www.nvidia.com/content/dam/en-zz/es_em/Solutions/Dat...

[2] https://devblogs.nvidia.com/dgx-superpod-world-record-superc...

zak9• 7 years ago

Author of the blog post here.

As mentioned in other comments, I'd recommend doing a performance-per-dollar comparison in addition to looking at this pure performance comparison at maximum scale.

daxfohl• 7 years ago

What's in the picture? Can't figure out the scale. Are those like server racks or breadboards or ...?

smartinspereira• 7 years ago

Server racks. The colored things are all cables.

Klinky• 7 years ago

Here's a higher resolution version: https://storage.googleapis.com/gweb-cloudblog-publish/origin...

morphle2• 7 years ago

All you see is 8 server racks with colored network cables, switches and power supplies. The TPU ASICs themselves make up just a tiny part of this datacenter, you also have the printed circuit boards, cooling fins, 8x48 metal boxes, power and network cables, DC/DC or AC/DC converters at the bottom, fans or water tubes for cooling and airgaps.

My startup is trying to develop wafer scale integration where you collapse 2 racks of the network, metal boxes, power and cooling into a 300mm wafer immersed a 100 mm x 400mm box with a few fibers and three power cables coming out. That can save around 900% of the capital cost and orders of magnitude of power (especially if you put the box in a building where the waste heat is not wasted but used to heat water for showering and space heating).

As it is now, these datacenter customers and hyperscalers don't seem to care about the enormous waste and cost of paying for inefficient hardware. Considering the enormous cost and carbon emmission savings a wafer scale integration would bring (and the competitive advantage), you would be suprised how hard it is to get funding from them to develop it.

I suspect is also the main reason they don't care to publish normalized benchmarks for $/performance/joule, as it would demonstrate how wasteful it all is.

rrss9• 7 years ago

would be suprised how hard it is to get funding from them to develop it.

That's probably because there has been no evidence that wafer scale integration can actually work.

rrss9• 7 years ago

It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark).

Seriously.

Transformer: 1024 TPUs are twice as fast as 480 GPUs.

Resnet50: 1536 GPUs are about as fast as 1024 TPUs.

SSD: 1024 TPUs are twice as fast as 240 GPUs.

Great.

aey4• 7 years ago

Results are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.

anonuser1234562• 7 years ago

When Moores law is finally dead and buried (e.g. 5nm), new architectures will be all that's left. Seems like a great time to start down the new architecture path.

zak9• 7 years ago

Author of the blog post here.

I'd recommend doing a performance-per-dollar comparison before drawing this conclusion.

solidasparagus3• 7 years ago

For actual practitioners, TPUs are incredible. The cost/performance combo is unmatched.

Now the real problem is, can you actually get a TPU pod in practice?

m3at• 7 years ago

You might want to check the DAWN benchmark, it does just that:

[1] https://dawn.cs.stanford.edu/benchmark/index.html#imagenet-t...

rrss9• 7 years ago

Thanks, I've seen dawnbench, and it's definitely better than this presentation.

Unfortunately, their cost numbers aren't actual cost, as they don't allow using AWS spot instance pricing (which is what fast.ai does).

https://www.fast.ai/2018/04/30/dawnbench-fastai/

boulos3• 7 years ago

Disclosure: I work on Google Cloud.

I know. Within MLperf, to reach consensus, it was agreed that “dollars are great, but also game-able via price cuts, lets normalize to a reference system”. It’s not the largest box though, it’s one that everyone was able to provide, so that it lets you compare through one level of indirection.

I don’t agree with it, because it makes it so hard to compare to funky hardware (“Should I do inference on this chip from < startup >? I dunno, what’s it going to cost?”). Having to keep going back to some reference box is undesirable, but there’s nothing stopping someone from making a spreadsheet equivalent that translates these to dollars on the cloud providers (harder for on-prem submissions which reopens the “rent vs buy” debate).

rrss9• 7 years ago

I'm not (currently) objecting to the actual MLPerf results.

I'm objecting to the fact that your marketing team (at least, I hope it was your marketing team) chose to represent 5 different GPU systems as the same color and 3 different TPU systems as another color in one bar chart.

There's a book with a chapter or two on how to reasonably compare the performance of computer systems. Given that the author (Patterson) works on TPUs, I would expect better.

zak9• 7 years ago

Author of the blog post here.

Cloud TPUs are designed to maximize performance-per-dollar, so you are right that pure performance comparisons at maximum scale don't tell the whole story.

The most straightforward performance-per-dollar comparisons would be among several different hardware configurations across the major public clouds. However, we haven't yet seen any other public cloud MLPerf submissions at scales comparable to Cloud TPU Pods, so there isn't currently a strong baseline available for comparison. It's also not clear whether public cloud networking will ultimately be able to match the performance of the network hardware that was used to produce the largest-scale on-premise MLPerf submissions.

rrss9• 7 years ago

Why performance per dollar over performance per watt?

Performance per watt doesn't care where the systems are (on-prem vs cloud) and should track perf/$ (unless perf/$ is mostly determined by subsidies).

yaroslavvb2• 7 years ago

Can you elaborate on the performance-per-dollar part? IE, I'm seeing GCP providing V100 at $2.48/hour and a TPUv3 at $8.00/hour

justicezyx• 7 years ago

I cannot stand Google people's tendency to explain or sometimes rebuttal comments.

You are talking to your customers who is paying or considering paying, or in search of products.

Take the feedback, if it can be done, and it's beneficial, do it and report so.

Or stop explaining... That's simply not professional for a cloud provider...

gok3• 7 years ago

I'm curious how (absolutely) efficient the Transformer training is even on the TPUs. The results from self-attention models are really impressive but unfortunately their topology makes them very difficult to implement efficiently in silicon. It often becomes purely a question of memory bandwidth, because you're not doing much math per weight on each iteration. I wonder if the speedup is from the use of on-chip HBM in the TPUs.

rrss9• 7 years ago

You mean this speedup?

1024 TPUs are twice as fast as 480 GPUs

Might it be because there are twice as many?

zak9• 7 years ago

Author of the blog post here.

We submitted multiple results using various Cloud TPU v3 Pod slice sizes to show the current achievable Transformer training efficiency at several scales:

https://mlperf.org/training-results-0-6

We're actively improving the whole TPU software stack, so training efficiency is likely to continue to increase over time.

Cloud TPU Pods Break AI Training Records

Related Stories