Google announces a new generation for its TPU machine learning hardware

https://techcrunch.com/2018/05/08/google-announces-a-new-generation-for-its-tpu-machine-learning-hardware/

by deesix • 8 years ago

141 114 8 years ago

polskibus• 8 years ago

Is this going to be available off-the-shelf like NVidia GPUs? That's the only way to get wider and faster developer buy-in.

dejv• 8 years ago

They did denied selling older generation, so I would expect TPU 3 is not going to be available as well. The only way to use those is to use Google Cloud.

aseipp• 8 years ago

No, they're going to be used by Google and nobody else, so they can establish dominance in the cloud AI/AI SaaS space using their immense resources. This will be used by internal Google projects to rapidly develop their large scale models and, if you're lucky, available on Google Cloud one day (TPUv2 already is, at least).

Google doesn't need "developer buy in" for these to make sense -- they need better hardware for training and deploying deep learning models for their products, which is the overwhelming motivation. And, if they offer these to you, it's only because you're willing to pay for faster TensorFlow iteration.

option_greek• 8 years ago

Models are the new bitcoins and no one is willing to sell the hardware directly. Weirdly enough exact same thing happened in mining (specialized hardware passed nvidia gpus in performance at some point).

jacksmith2100629• 8 years ago

Would doubt it and not really make much sense in 2018.

shaklee311• 8 years ago

No sense at all. Nvidia's stock increased 14x since 2014 by running a closed system in a public cloud. Oh wait...

thwd• 8 years ago

It's 'Tensor Processing Unit' as far as I know.

tlb• 8 years ago

Title corrected from "Tensorflow Processing Unit"

frisco• 8 years ago

100 petaflops per what? All of google’s deployment? Per rack? 100 Pflop / chip can’t be right.

blattimwind• 8 years ago

Per pod (256 TPUs)

deepnotderp5• 8 years ago

They increased the pod size by 4X, also each "TPU" has 4 tpu chips on it IIRC.

raverbashing• 8 years ago

Can you allocate a whole pod to do your computations? How much does that cost?

mlazos• 8 years ago

Yeah I love this, 8x PFlop improvement per pod, but the pod size has increased 4x. That 8x speed up is now mystically halved.

sailingparrot5• 8 years ago

*mystically quartered.

sabalaba3• 8 years ago

This is great news. It’s extremely important to the research community that large companies enter into the DL silicon space to contend with NVIDIA’s monopoly.

NVIDIA is now exerting pricing power to the point where they’ve decided to train their sales people to disregard the metric that is most important to customers: cost of training. Talk with one of their enterprise sales people and you’ll find they’ll say things like “FLOPS / $ doesn’t matter” to justify a 10x increase in price for their TESLA line. As history has shown, a monopolist sows the seeds of their own destruction and, by disregarding the metrics that matter, they alienate their customers.

Here are open source projects that you can contribute to to break the monopoly:

ROCm: https://rocm.github.io/

MIOpen: https://github.com/ROCmSoftwarePlatform/MIOpen

TensorFlow: https://github.com/tensorflow/tensorflow

zitterbewegung2• 8 years ago

Well until that actually happens I will be buying more NVIDIA stock.

I have done deep learning also and I don't see NVIDIA being an incumbent that is easy to be knocked out. I think we would need some distruption such as a new technique that is more performant or different design goals. Intel is still leading the way and I don't see NVIDIA being unseated with its head start anytime soon.

sabalaba3• 8 years ago

I'm not as pessimistic. 3dfx and Matrox didn't look like easy to knock out opponents in the 90s and yet here we are. Markets are powerful incentive mechanisms.

"Slowly grinds the mill of the gods, but it grinds fine."

hedgehog2• 8 years ago

We're working to provide more choice with PlaidML as well. You can use it to run Keras, ONNX, and TF models on AMD, Intel, and mobile/embedded GPUs, on most operating systems and OpenCL, LLVM, OpenGL, and soon CUDA and Metal, etc. We would welcome contributions for TensorFlow integration, ROCm support, and anything else that widens support to more people.

https://github.com/plaidml/plaidml

floatboth• 8 years ago

Cool! Are you considering Vulkan?

twtw2• 8 years ago

Instead of being able to buy your own nvidia gpu and run any cuda or opencl on it, now you can run only tensor flow on a tpu only in Google cloud.

How fantastic.

chewxy2• 8 years ago

And only using Tensorflow.

sabalaba3• 8 years ago

Choice in the marketplace is still choice. And, Google might eventually decide to sell this to outside companies. Eventually the desire to build competitive advantages for Google Cloud might be surpassed by the desire to compete with NVIDIA. Commoditizing the complement is a powerful strategy.

throwaway20482• 8 years ago

Yeah seems like a way more abusive position to me frankly, a Google monopoly would be way worse than a NVIDIA one.

jacksmith2100629• 8 years ago

It is about choice and driving down cost. Love google keeps pushing the envelope which is good for everyone

tehsauce• 8 years ago

TPUs seem like great hardware. Unfortunately google's insistence on keeping them in a walled garden is a major deterrent to building something that relies on their platform.

jacksmith2100629• 8 years ago

What makes sense. Would guess each cloud provider will do their own.

beagle3• 8 years ago

An up-and-coming project in an up-and-coming language:

ArrayMancer: https://mratsim.github.io/Arraymancer/

Written in Nim, which looks and feels a lot like Python, but runs a lot like C++ or Rust, especially in the speed and FFI departments, and deploys like Go as a small single fast executable. Already supports CPU, CUDA and OpenCL. Still early, but with a lot of promise. Use is modeled on some mixture of PyTorch and Keras.

chewxy2• 8 years ago

Self-plug: Gorgonia[0] can support OpenCL as well (but time and effort is required to write support libs). Would appreciate contributions. I recently gave a talk on how to interface with foreign libs[1]. Writeup soon to come.

[0] https://github.com/gorgonia

[1] https://www.youtube.com/watch?v=7R39CMW4MJQ

punnerud• 8 years ago

Water cooling on their TPU is probably because they cost way more to buy than the extra cost of special cooling. It is also probably to increase “spec” on their new version without the best improvements in hardware/silicon.

bitL• 8 years ago

I think NVidia was milking the cow in anticipation of moves from Google, AMD and MS in their space, leading to a pricing war.

jamesblonde2• 8 years ago

What will be interesting to see is if they are going the hardware specialization route, like Nvidia with their support for efficient 4x4 fused FP16 matmuls with FP32 matrix output (they call them 'tensor cores' - hah). I suspect with the liquid cooling, they are just dialing up the matmul speed, which is probably the right way to go, IMO.

We are doing data-center experiments - using oil cooling. It's suprising to see circuitboards dumped in ordinary oil and working away, no short circuits.

deepnotderp5• 8 years ago

You mean immersion cooling? As long as the fluid isn't conductive you're good to go.

If you're interested, Alibaba has been doing some work along the lines of production immersion cooling.

jamesblonde2• 8 years ago

Yes, it's not my research - a colleagues. He has some big vats of oil. It's a sureal experience to see servers just being oil-boarded (can i say that? :) ) and coming out ok!

zeusk2• 8 years ago

mineral oil based cooling is well known in enthusiast PC builds but it leaves weird gunk on boards that is harmless but a big PITA to clean up.

Also - some oils while won't hurt your circuits, they might start dissolving the various plastic connectors on cables and board.

jacquesm• 8 years ago

It's suprising to see circuitboards dumped in ordinary oil and working away, no short circuits.

That will make a datacenter fire into something else entirely.

zeusk2• 8 years ago

Flashpoint of some of the oils is higher than 350F.

100F and below is considered flammable.

sailingparrot5• 8 years ago

Google CEO Sundar Pichai said the new TPU is eight times more powerful than last year

Are we sure about this? He specifically said that a "pod" would be 8 times faster than last year, not the TPU itself.

And the picture in the background showed what looked like to be 8 racks of 64 TPUs (or maybe 32?). Until now a "pod" was a single rack of 64 TPUs. So if the new definition for a "pod" is 8 times as many TPUs as it was before, the result is less impressive...

Is there any actual spec released?

twtw2• 8 years ago

Google started playing this game with the tpuv2, wherein they define a "TPU" as whatever they want to make it sound suitably impressive (I.e. Cloud TPU is four chips). This in turn led to nvidia calling the DGX-2 "the world's largest GPU

jacksmith2100629• 8 years ago

What does it matter? What we care about is the cost. The TPU gen 2 are about half as much as using Nvidia in the Amazon cloud for the same work. To me that is what matters. If the TPU 3 further drops the cost fantastic.

How many chips sounds a lot more like a pissing contest. What if it was a giant chip with actually a bunch of chips inside? Who cares?

sanxiyn3• 8 years ago

While DGX-2 includes 16 V100, you can't replicate DGX-2 performance with 16 V100. Much of its performance is due to NVSwitch/NVLink. I think it's same with TPU.

sanxiyn3• 8 years ago

This is about scaling. Since you usually can't achieve 8x speed with 8 chips (there will be issues with interconnect, algorithm, etc.), 8x speed with 8 chips is noteworthy. Although I wouldn't describe that scaling work as "8 times more powerful" myself.

sailingparrot5• 8 years ago

~8x speed with 8 chips of the same generation would obviously be very impressive. But 8x speed with 8 chips compared to a last year's single chip is definitely not the same thing.

Pretty sure you can get more than (or around) 8x speed up when comparing 8 V100 vs a single P100 for example.

whazor• 8 years ago

More chips on a board requires more cooling or less power consumption.

forgot-my-pw• 8 years ago

I haven't seen much info yet on TPU 3, but TPU 2 was:

- 64 TPUs per pod (11.5 petaflops)

- 4 chips per TPU (180 teraflops)

shaklee311• 8 years ago

It's 2x faster per area:

https://twitter.com/soumithchintala/status/99391656047246131...

ai23237• 8 years ago

Welcome to ASIC land....training going the route of crypto mining

Maybestring2• 8 years ago

It's just an arithmetic logic unit for tensors. It's not at all like a crypto miner that implements a single algorithm.

jacksmith2100629• 8 years ago

Exactly. Drives me crazy when people compare to a single alogrithm chip. It is not.

ai23237• 8 years ago

You kind of summed it up right there...'it's just an arithmetic logic unit for tensors'...whether its a matrix multiply engine or implmenting sha256 still a custom rapidly iterated chip for narrow very specific use case. Google accomplishment here clearly the software, but doubt they only ones going to crack asic systolic arrays. At some point china inc figures this out in mass...maybe bitmain themselves.

esmi• 8 years ago

Does the Google Cloud API for TensorFlow expose any hardware details? I am just curious why Google announces these hardware details at all.

jacksmith2100629• 8 years ago

Look at this thread. Because people want to know as just what "techies" do.

But what I wanted to know is performance in terms of joules compared to TPU 2s.

My hope is we get a paper on the TPU 2 now the 3 is released. We got the TPU 1 paper as they were releasing the TPU 2 I suspect.

ai23237• 8 years ago

This very typical google. V100 comes out, they don't deploy it in their cloud immediately. Then they launch their tpu cloud. Spend 2 months touting cost/speed in benchmarks. Then a week before io they make v100 available in gce, nearly six months after aws. Then at io they announce tpu3. This is supposed to be nvidia's most captive customer, and now looks like going to be their biggest problem. Would love to see google spend with them and how its changed since they ramped tpu2.

oh-kumudo• 8 years ago

Lower GPU price for all, I am for it.

ai23237• 8 years ago

For all the google hate, nvidia is too blame as well. Their ceo running around saying 'we saving you money every time you spend 10k on a gpu' is getting old. The should have been far more aggressive price wise with this biz considering the resources of their target customers. Almost every 'choose not to use a gpu' use case being touted by msft/goog etc is about cost. Meanwhile nvda busy tweeting stuff like msft seeing ai app for blind powerd by nvda gpu. Soon micron will be tweeting they powering all ai as well, and then arista...all the way down to the power companies. Watching goog today its not looking good for the competition, by the time all these ml/dl accelerators are commoditized google will have one hell of a moat.

jacksmith2100629• 8 years ago

Spend with them? Who?

ai23237• 8 years ago

Their spend on nvidia gpu's...how it's shifted as they have progressed tpu wise. I think you go back 18 months google was their biggest here.

boulos• 8 years ago

Disclosure: I work on Google Cloud (and even on the GCE parts).

No conspiracy here, we were simply late to market on the V100 Beta. As many have noted, we also hold a really high reliability bar before going to Beta (“Google’s Beta is like AWS’s GA” though I wouldn’t claim that in all cases) and we’ve be in Alpha for months. If you’d like to be included in Alpha offerings drop us a line.

As I’ve said in previous threads: Compute Engine intends to be the best place for computing. That includes the kind of workloads TPUs are tuned for, but it also includes all the stuff that GPUs excel at. That’s not going away, and GCE isn’t favoring one over the other. We sell infrastructure, and we don’t try to (overtly) pick favorites, we let our customers do that.

lostmsu• 8 years ago

“Google’s Beta is like AWS’s GA” though I wouldn’t claim that in all cases

Wow, that's so ... I don't even have a word for it.

ai23237• 8 years ago

Apologies, stand corrected. Conspiracy theories are always more fun:) Guess was just coincidence that after grabbing all this press on GCE v100 finally being available, tpu3 is announced few days later.

jacksmith2100629• 8 years ago

You do not want to be dependent on Nvidia for your own stuff is why Google has to do the TPUs.

It is like how they ported to Power for their stuff if needed.

Look at the shortage of late with Nvidia. Google does not have to deal with it.

But also I suspect they could not do WaveNet without their own silicon as the cost would be prohibitive. Using a NN for speech and competing against traditional approaches and have a competitive price is going to require really efficient silicon.

shaklee311• 8 years ago

What shortage? You can't make statements like that with no basis in reality. The V100 is not suffering from any shortage.

zackmorris• 8 years ago

Just to play devil's advocate for a moment: I'm excited that there will finally be some viable competition for GPUs, but am disappointed that this isn't a general-purpose multiprocessor.

We're long overdue for a general-purpose CPU with say 1024 cores, that avoids a central main memory, where each core can be independently programmed just like any other CPU. Google's may count as a somewhat general-purpose DSP, which is definitely a step forward. But no matter how mature or mainstream a framework like TensorFlow gets, it can never replace full programmability.

Without seeing the internals, I'm going to have to give this a nay vote for now. There are many other rather exciting problems that need to be opened up to a new generation of tinkerers. Off that top of my head, it's things like: content-addressable memory to provide high data locality (evolving various interconnects instead of hardwiring them), exploring other types of general vector processing like the kind MATLAB/Octave uses, and exploring other hill-climbing algorithms than backpropagation/neural nets.

I picture something more like network topology-agnostic Docker containers programmed in Elixer/Erlang/Go that can act as semi-autonomous agents and switch into various modes in order to solve the problem at hand. I just find that a much simpler metaphor to work with than OpenCL/CUDA/TensorFlow. Yes it would take more silicon and would probably violate YAGNI, but only full programmability gives us the freedom to explore the problem space at the level that's going to be required to implement artificial general intelligence.

tim333• 8 years ago

The announcement bit on video https://youtu.be/ogfYd705cRs?t=1h36m

Google announces a new generation for its TPU machine learning hardware

Related Stories