Trillium TPU Is GA

https://cloud.google.com/blog/products/compute/trillium-tpu-is-ga

by gok • 2 years ago

178 80 2 years ago

cloud.google.com

xnx4• 2 years ago

We used Trillium TPUs to train the new Gemini 2.0,

Wow. I knew custom Google silicon was used for inference, but I didn't realize it was used for training too. Does this mean Google is free of dependence on Nvidia GPUs? That would be a huge advantage over AI competitors.

m3kw9• 2 years ago

Maybe only for their own models

walterbell• 2 years ago

Now any Google customer can use Trillium for training any model?

xnx4• 2 years ago

Google trained Llama-2-70B on Trillium chips

Permit• 2 years ago

My understanding is that the Trillium TPU was primarily targeted at inference (so it’s surprising to see it was used to train Gemini 2.0) but other generations of TPUs have targeted training. For example the chip prior to this one is called TPU v5p and was targeted toward training.

dekhn5• 2 years ago

Google silicon TPUs have been used for training for at least 5 years, probably more (I think it's 10 years). They do not depend on nvidia gpus for the majority of their projects. Took TPUs a while to catch up on some details, like sparsity.

summerlight• 2 years ago

This aligns with my knowledge. I don't know much about LLM, but TPU has been used for training deep prediction models in ads at least from 2018, though there were some gap filled by CPU/GPU for a while. Nowaday, TPU capacity is probably more than the combination of CPU/GPU.

rajnathani• 2 years ago

Is sparsity used for training though? I thought it was an inference-only thing for both GPUs and TPUs. Furthermore, I think TPUs have not been able to support training models like GANs, etc. I'm not sure if that has changed now.

lern_too_spel2• 2 years ago

Since TPUv2, announced in 2017: https://arstechnica.com/information-technology/2017/05/googl...

The superscalers are all working on this. https://aws.amazon.com/ai/machine-learning/trainium/

drusepth• 2 years ago

Why is that a huge advantage over AI competitors? Just not having to fight for limited Nvidia supply?

aseipp• 2 years ago

That is one factor, but another is total cost of ownership. At large scales something that's 1/2 the overall speed but 1/3rd the total cost is still a net win by a large margin. This is one of the reasons why every major hyperscaler is, to some extent, developing their own hardware e.g. Meta, who famously have an insane amount of Nvidia GPUs.

Of course this does not mean their models will necessarily be proportionally better, nor does it mean Google won't buy GPUs for other reasons (like providing them to customers on Google Cloud.)

xnx4• 2 years ago

Yes. And cheaper operating cost per TFLOP.

badlucklottery• 2 years ago

Vertical integration.

Nvidia is making big bucks "selling shovels in a gold rush". Google has made their own shovel factory and they can avoid paying Nvidia's margins.

bufferoverflow• 2 years ago

TPUs are cheaper and faster than GPUs. But it's custom silicon. Which means barrier to entry is very very high.

felarof6• 2 years ago

TPUs have been used for training since a long time.

(PS: we are startup trying to make TPUs more accessible, if you wanna fine-tune Llama3 on TPU check out https://github.com/felafax/felafax)

ec109685• 2 years ago

Apple trained their LLM on Google TPU’s: https://www.theregister.com/2024/07/30/apple_google_tpu_ai/

randomcatuser• 2 years ago

How good is Trillium/TPU compared to Nvidia? It seems the stats are: tpu v6e achieves 900 TFLOPS per chip (fp16) while Nvidia H100 achieves 1800 TFLOPS per gpu? (fp16)?

Would be neat if anyone has benchmarks!!

chessgecko• 2 years ago

1800 on the h100s is with 2/4 sparsity, it’s half of that without. Not sure if the tpu number is doing that too, but I don’t think 2/4 is used that heavily so I probably would compare without it.

amelius2• 2 years ago

Are the Gemini models open?

stefan_• 2 years ago

Not even to Google customers most days, it seems.

jnwatson• 2 years ago

Just the Gemma models are open.

blackeyeblitzar• 2 years ago

So Google has Trillium, Amazon has Trainium, Apple is working on a custom chip with Broadcom, etc. Nvidia’s moat doesn’t seem that big.

Plus big tech companies have the data and customers and will probably be the only surviving big AI training companies. I doubt startups can survive this game - they can’t afford the chips, can’t build their own, don’t have existing products to leech data off of, and don’t have control over distribution channels like OS or app stores

talldayo3• 2 years ago

Nvidia’s moat doesn’t seem that big.

Well, look at it this way. Nvidia played their cards so well that their competitors had to invent entirely new product categories to supplant their demand for Nvidia hardware. This new hardware isn't even reprising the role of CUDA, just the subset of tensor operations that are used for training and AI inference. If demand for training and inference wanes, these hardware investments will be almost entirely wasted.

Nvidia's core competencies - scaling hardware up and down, providing good software interfaces and selling direct to consumer are not really assailed at all. The big lesson Nvidia is giving to the industry is that you should invest in complex GPU architectures and write the software to support it. Currently the industry is trying it's hardest to reject that philosophy, and only time will tell if they're correct.

felarof6• 2 years ago

CUDA, just the subset of tensor operations that are used for training and AI inference. If demand for training and inference wanes

Interesting take, but why would demand for training and inference wade? This seems like a very contrarian take.

tada131• 2 years ago

If demand for training and inference wanes, these hardware investments will be almost entirely wasted

Nvidia also need to invent smth then, as pumping mining (or giving good to gamers) again is not sexy. What's next? Will we finally compute for drug development and achieve just as great results as with chatbots?

llm_nerd3• 2 years ago

It seems this way, but we've been saying this for years and years. And somehow nvidia keeps making more and more.

Isn't it telling when Google's release of an "AI" chip doesn't include a single reference to nvidia or its products? They're releasing it for general availability, for people to build solutions on, so it's pretty weird that there isn't comparisons to H100s et al. All of their comparisons are to their own prior generations, which you do if you're the leader (e.g. Apple does it with their chips), but it's a notable gap when you're a contender.

jeffbee• 2 years ago

Google posted TPUv6 results for a few things on MLCommons in August. You can compare them to H100 over there, at least for inference on stable diffusion xl.

Suspiciously there is a filter for "TPU-trillium" in the training results table, but no result using such an accelerator. Maybe results were there and later redacted, or have been embargoed.

mlboss• 2 years ago

The biggest barrier for any Nvidia competitor is that hackers can run the models on their desktop. You don't need a cloud provider specific model to do stuff locally.

r3trohack3r2• 2 years ago

This. I suspect consumer brands focusing on consumer hardware are going to make a bigger dent in this space than cloud vendors.

The future of AI is local, not remote.

melodyogonna• 2 years ago

If Modular (modular.com) delivers on their MAX product the moat may get smaller as they may lose the software advantage.

Hilift• 2 years ago

"we constantly strive to enhance the performance and efficiency of our Mamba and Jamba language models."

... "The growing importance of multi-step reasoning at inference time necessitates accelerators that can efficiently handle the increased computational demands."

Unlike others, my main concern with AI is any savings we got from converting petroleum generating plants to wind/solar, it was blasted away by AI power consumption months or even years ago. Maybe Microsoft is on to something with the TMI revival.

beepbooptheory• 2 years ago

This has been a constant thought for me as well. Like, the plan from what I can tell is that we are going to start to spinning all this stuff up every single time someone searches something on google, or perhaps, when someone would otherwise search something on there.

Just that alone feels like an absolutely massive load to bear! But its only a drop in the bucket compared to everything else around this stuff.

But while I may be thirsty and hungry in the future, at least I will (maybe) be able to know how many rs are in "strawberry".

r3trohack3r2• 2 years ago

Energy availability at this point appears (to me at least) to be practically infinite. In the sense that it is technically finite but not for any definition of the word that is meaningful for Earth or humans at this stage.

I don't see our current energy production scaling to meet the demands of AI. I see a lot of signals that most AI players feel the same. From where I'm sitting, AI is already accelerating energy generation to meet demand.

If your goal is to convert the planet to clean energy, AI seems like one of the most effective engines for doing that. It's going to drive the development of new technologies (like small modular nuclear reactors) pushing down the cost of construction and ownership. Strongly suspect that, in 50 years, the new energy tech that AI drives development of will have rendered most of our current energy infrastructure worthless.

We will abandon current forms of energy production not because they were "bad" but because they were rendered inefficient.

WanderPanda• 2 years ago

Crazy conglomerate discount on Alphabet if you can see TPUs as the only Nvidia competitor for training. Breaking up Alphabet seems more profitable than ever

wrsh073• 2 years ago

TPU is available to people outside of Google, but people prefer Nvidia

It's not obvious to me that a hardware business manufacturing tpus for the general public is necessarily more valuable than one that benefits from tight integration with Google's internal software stack and datacenter tech

It's orthogonal: tpu doesn't do much marketing and doesn't have to. Engineers at Google will use it because they have to and because of the huge cost advantage they get for using it. TPU is probably lacking the libraries that Nvidia has developed to be accessible to a broad array of use cases

lanthissa3• 2 years ago

This is the thing that makes no sense to me, if TPU's are even close to nvidia its a business worth as much as search. They could spend billions on just dx and get 100x that back on investment.

The whole TPU line makes no sense to me, if its as good as it says (which is does seem to be) sell it publicly and add a trillion to your market cap.

The only way this makes any sense is if inside google people legitimately think google cloud is going to be bigger than nvidia+ a lot of azure&aws, which seems crazy

lanthissa3• 2 years ago

Okay I really dont understand this, nvidia has a 3.4T market cap google has a 2.4T post run up, and its PE is like 38 vs 25 so its a higher multiple on the business too. It appears making the best AI chip is a better business than googles entire conglomerate.

If TPU's are really that good why on earth would google not sell them. People say its better to rent, but how can that be true when you look at the value of nvidia.

LittleTimothy2• 2 years ago

I don't know why you'd be confused by the valuations, they're just 2 different businesses. Google is an ad sales business with an extremely strong revenue driver (youtube and Search) so it churns out money but it's pretty well understood revenue stream and is facing some possible disruption. Nvidia is a chip sales business and whether through luck or through skill (or both) they've essentially ended up as the monopoly provider of the worlds best chips. It's not unreasonable to expect Nvidia can squeeze a tonne of cash out of their customers for the next few years in an undisruptable way. So it's easy to understand the valuations.

On the GPU side - One way to look at it is lets just ignore whether you're selling chips. Let's just look at where compute is done - because Google let's you rent time on the TPUs so if they're really great people will just choose to rent time on them. How much time is being rented on TPUs in the cloud vs. Nvidia GPUs? Google's TPUs are not a market leader, they're probably not even significant market share. Pretend Google decided to sell TPUs tomorrow - are many people who weren't choosing to rent TPUs going to decide to buy them? No. There's an element of interoperability and supply chain etc. etc. but I think the core of it is a full ML product the TPU isn't anywhere close to the Nvidia GPUs.

lanthissa3• 2 years ago

I'd be confused by them choosing to not enter hte more lucrative business if they had the product to do so.

The only way the current situation makes sense is if your last sentence is true, which is what i'm getting at. Either TPUs are significantly behind Nvidia chips, or google is choosing not to add like a trillion dollars of market cap to their business its really one or the other.

VirusNewbie• 2 years ago

From my understanding TPUs are in so much demand it’s hard to even get access to to the best ones on GCP. It’s a capacity issue more than a popularity one.

shaism• 2 years ago

Maybe the market hasn’t recognized the value yet.

Hence, buy $GOOG.

theptip• 2 years ago

As a modeling exercise: this isn’t necessarily inconsistent.

Right now if Google can earn more from using their TPUs than they would selling them (ie profitably utilize them) they would be crazy to sell them.

Companies are valued not on their present value, but the net present value of their future earnings. So the big multiple implies that there is much more revenue growth potential here. For example, NVidia has Cuda, Google doesn’t have anything as good for third party users. Maybe the market is pricing in that Google doesn’t have a good path to manufacture and sell these chips externally, while NVidia has room to grow into every 1GW datacenter that’s built over the next few decades.

evanjrowley• 2 years ago

Right now if Google can earn more from using their TPUs than they would selling them (ie profitably utilize them) they would be crazy to sell them.

Reminds me of what happened with Bitcoin mining ASICs. Everyone capable of making them realized they could make more money using them directly vs. selling them to others.

hn3er1q• 2 years ago

If TPU's are really that good why on earth would google not sell them.

nVidia sells nearly 4M GPUs per year. Google claims like 100K TPUs. Scaling a production line by 10x is very difficult and Google has not shown aptitude in this area of expertise.

Even if Google wanted to scale 10x I'm not sure they could. nVidia is believed to be taking like half of TSMC's new capacity (existing capacity is not idle). I suppose technically that means the other half could be consumed by Google but it's likely TSMCs other customers wouldn't appreciate that.

LittleTimothy2• 2 years ago

The question I'd like to know the answer to is "What was the total cost of training Gemini 2.0 and how does it compare to the total cost to train equivalent capability models on Nvidia GPUs?". I'd be fascinated to know, and there must be someone at Google who has the data to actually answer that question. I suspect it's politically savvy for everyone at Google to pretend that question doesn't exist or can't be answered (because it would be an existential threat to the huge TPU project), but it would be absolutely fascinating. In the same way that Amazon eventually had to answer the "Soo.... how much money is this Alexa division actually making" question.

teleforce• 2 years ago

It's beyond me why processor with dataflow architecture is not being used for ML/AI workloads, not even in minority [1]. Native dataflow processor will hands down beats Von Neumann based architecture in term of performance and efficiency for ML/AI workloads, and GPU will be left redundant for graphics processing instead of being the default co-processor or accelerator for ML/AI [2].

[1] Dataflow architecture:

https://en.wikipedia.org/wiki/Dataflow_architecture

[2] The GPU is not always faster:

https://news.ycombinator.com/item?id=42388009

almostgotcaught• 2 years ago

let me ask you a very serious question and please answer honestly: have you ever tried to program a "dataflow processor"? if the answer is no then I invite you to try and then you will understand intimately why they're not being used for anything.

germanjoey• 2 years ago

Sambanova's RDU is a dataflow processor being used for ML/AI workloads! It's amazing and actually works.

Trillium TPU Is GA

Related Stories