Cerebras’ new monster AI chip adds 1.4T transistors

https://spectrum.ieee.org/tech-talk/semiconductors/processors/cerebras-giant-ai-chip-now-has-a-trillions-more-transistors

by Anon84 • 5 years ago

273 165 5 years ago

spectrum.ieee.org

fredfoobar3• 5 years ago

Time for an AI winter I guess.

streetcat1• 5 years ago

You do realize that AI crossed human expert performance in NLP / Vision tasks ?

semi-extrinsic• 5 years ago

Exactly how does one outperform a human expert in natural language processing?

firebaze• 5 years ago

I'm not sure if you're assuming best intentions. If you do, it'd be nice to provide sources - to my knowledge, vision under non-optimal conditions (rain, snow, sunlight ahead) is only partially solved by resorting to sensors resistant to the disturbance.

I'd be glad to learn I'm wrong.

king_magic• 5 years ago

No, it hasn't. Not even close.

AI has "crossed human expert performance" on extremely narrow NLP/CV tasks.

AI is still light years away from human-level performance.

anthk• 5 years ago

GPT-3 is not even close to a minimal and coherent text adventure made with Inform6 from a novice, even if it's written by a non-native English speaker.

Those networks didn't match "Detective", a crappy story written by a 12yo.

trhway• 5 years ago

Time for an AI winter

with AI chips burning 15KW? No chance for the winter in sight. Some chances for AI hell though.

fredfoobar3• 5 years ago

This statement was made for experimental purposes, to understand HN biases.

systemvoltage3• 5 years ago

Most interesting aspect of wafer-scale manufacturing is yield. Even if we have 95% chip yield, as the chip size approaches the wafer-level dimensions, I don't know off top of my head what the math would be but it is going to plummet drastically. My guess is that they're handling this in the chip logic. Building resiliency by turning off cells in the wafer that didn't yield. That begs the question, how are they probing them? A probe card of the size of the wafer is unheard of. How are they running validation? Pretty mindblowing to say the least!

belval• 5 years ago

Not an expert in chip manufacturing but my guess is that they just disable the parts that don't work and their big numbers represent ~80% of the actual number of transistors in the wafer because they account for that manufacturing loss.

KETpXDDzR2• 5 years ago

Correct. You already have the same with modern CPUs and GPUs. You just disable the defect parts. Obviously, that sounds way easier than it is.

vkazanov• 5 years ago

Disabling parts of the chip? The secret sauce then is to make it defect-proof

gsnedders• 5 years ago

Yup. This. From https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...:

Cerebras achieves 100% yield by designing a system in which any manufacturing defect can be bypassed – initially Cerebras had 1.5% extra cores to allow for defects, but we’ve since been told this was way too much as TSMC's process is so mature.

baybal22• 5 years ago

Yep, not all semiconductor defects can be repaired, and at such scale repair circuits themselves will need to be redundant.

imtringued3• 5 years ago

The question is, how do you detect defective cores?

bob1029• 5 years ago

They probably aren't bothering to. The extreme economics for producing this type of chip are likely acceptable to the stakeholders.

Also, there is no reason they cant have some redundancy throughout the design so you can fuse off bad parts. It all really depends on the nature of the anticipated vs actual defects, which is an extraordinarily deep rabbit hole to climb into.

systemvoltage3• 5 years ago

I wonder how much this "chip" costs!

Pet_Ant• 5 years ago

How does this fusing work? I assume there are a bunch of wires than are either hot or ground and that determines if part of a chip gets run?

baybal22• 5 years ago

I bet they just probe the chip:

1. piece by piece

2. on die test circuits

seniorivn• 5 years ago

they exclude/disable cores with issues so an long as infrastructure parts of the chip are not affected the chip is functional

pradn• 5 years ago

The first Cerebras Wafer Scale Engine used "... breakthrough techniques in cross-reticle patterning, but with the level of redundancy built into the design, ensured a yield of 100%, every time." I'm unsure what to think of this.

"When we spoke to Cerebras last year, the company stated that they already had orders in the ‘strong double digits’."

And they cost 2- 2.5 million each!

https://www.anandtech.com/show/15838/cerebras-wafer-scale-en...

robocat• 5 years ago

"the price has risen from ~$2-3 million to ‘several’ million".

"The CEO Andrew Feldman tells me that as a company they are already profitable, with dozens of customers already with CS-1 deployed and a number more already trialling CS-2 remotely as they bring up the commercial systems".

Quotes from https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...

AnimalMuppet• 5 years ago

So, less than 100 orders? That doesn't blow me away. Though at 2 million or more each, that adds up to real money...

Zardoz84• 5 years ago

It's the old Wafer Scale Integration again. But now shows a successful product : https://en.wikipedia.org/wiki/Wafer-scale_integration

tandr3• 5 years ago

How do they do heat management and dissipation on such a big wafer? I can imagine some different parts heats up differently, putting a mechanical strain on wafer, and leading to cracks.

punnerud• 5 years ago

A large portion of the article is answering most of this

tandr3• 5 years ago

No, it does not. There is a paragraph _talking_ about it, but not much info still.

NortySpock• 5 years ago

"Carefully."

The article mentions a year of engineering went into dealing with the entire wafer thermally expanding under load.

addaon• 5 years ago

If you look at the die shots (wafer shots?), you'll notice small holes a few millimeters across spaced roughly at reticle spacing. Those are drilled holes to allow through-wafer liquid cooling. With liquid cooling not just around but through the wafer, the temperature differential is minimized.

tandr3• 5 years ago

thank you

etaioinshrdlu2• 5 years ago

How much memory is on the chip, and what kind is it?

Under what circumstances does the chip need to access external memory?

What type of communication interfaces does this chip have?

Also, if the chip is the size of a wafer, is it appropriate to call it a Chip?

verdverm• 5 years ago

Tom's Hardware has some nice tables comparing the specs: https://www.tomshardware.com/news/cerebras-wafer-scale-engin...

(more than the IEEE)

This thing pulls 15-20kW of juice!

teruakohatu• 5 years ago

I did a double that when I realised that was Kilowatts not Watts. This chip uses more energy in an hour than the average household (in my country at least) does in a day.

It may be a very large wafer but dissipating that heat is still very impressive.

meepmorp2• 5 years ago

This thing pulls 15-20kW of juice!

If you look at the wafer she’s holding at the top, it’s seemingly segmented into a 12x7 grid of roughly chip-sized rectangles. That’s 84 “CPUs” at 200-240 watts each, which is pretty well in line with discrete server CPUs.

The amount of heat coming off this thing must be amazing, though.

typon• 5 years ago

40GB of SRAM. Not quite big enough to fit the big models like GPT3

MuffinFlavored• 5 years ago

SRAM (static RAM) vs DRAM (dynamic RAM) for anybody else curious: https://computer.howstuffworks.com/question452.htm

etaioinshrdlu2• 5 years ago

It seems like you typically want a balance of memory size/bandwidth to compute ratio for typical deep learning applications.

The 40GB of SRAM probably has tremendous bandwidth (it could all be updated every few cycles!), but the memory size is very small compared to the amount of compute available.

However, maybe a different way of looking at it is that this chip will allow the training steps on deep learning models to take a fraction of the time as a GPU. Perhaps what takes 1s on a GPU could take 10ms on this chip.

So, this product may be effective at making training happen very fast, but without substantial model size or efficiency gains.

That's still ground breaking -- you can't acheive this result on GPUs. You can't achieve this result by any parallelization or distributed training, either. The large batch sizes in distributed training do not result in the same model or one that generalizes as well.

claytonius• 5 years ago

I don’t think it’s straightforward to do a head to head comparison.

from: https://www.youtube.com/watch?v=yso2S2Svdlg

@ 25:14

James Wang: "If a model doesn’t fit into a GPU’s HBM, is it smaller when it’s laid out in the Cerebras way relative to your 18 gigabytes?"

Andrew Feldman: "It is — it’s smaller in that we hold different things in memory than they do. One can imagine a model that has more parameters than we can hold — one can posit one, but remember our memory is doing different things. Our memory is basically holding parameters. That’s not what their memory is doing. Their memory is holding the shape of the model, their model is holding the results of the batches. We use memory rather differently. We haven’t found models that we can’t place and train on a chip. We expect them to emerge, that’s why we support clustering of chips and systems, that’s why we do that in whats called a “model parallel” way, where If you put two chips together you get twice the memory capacity. That’s not what you get when you put multiple GPUs together. When you put multiple GPUs together you get two versions of the same amount of memory, you actually don’t get twice the memory. I see you smiling here because you know that’s a problem… …With us if we support 4 billion parameters and you add a second wafer scale engine, now you support 8 billion parameters ,and if you add a third you can support 12 billion. That’s not the way it works with GPUs. With GPUs you just support two chips, each with a few million - tens of millions of parameters."

IshKebab2• 5 years ago

It's 40 GB of SRAM. I doubt it supports external memory.

Also, if the chip is the size of a wafer, is it appropriate to call it a Chip?

Good question. I think it is. I mean the word "chip" isn't really that well defined (is HBM one chip?), but given that they sell it as a single unit and you can't really cut it in half I think it's one chip.

NaturalPhallacy• 5 years ago

Looking at the picture, the thing is a platter. Not a chip.

It's a really cool picture too.

bhewes• 5 years ago

This thing reminds me of Daniel Hillis's "The Connection Machine". Just 35 years later.

xhrpost• 5 years ago

How can the chip itself consume that kind of power? Or is the 15kw value for the entire unit? That's like 10 residential space heaters all turned to max. I'm surprised that much heat could be dissipated over such a small surface area. Does it use refrigerant for cooling? If my math is correct, if you had a 6500BTU window air conditioner, you'd need 8 of them to move the heat from this chip.

philjohn• 5 years ago

Well, a single 7nm CPU (AMD Ryzen) can pull down 95 odd watts at peak, granted, the IO die is a fair bit of that as it's on an older process, but if you extrapolate that out across a giant wafer it's "only" 157 Ryzen 7's.

YetAnotherNick• 5 years ago

It would melt if you just do air cooling. I did some math for liquid nitrogen which has heat of vaporization as 200 kJ / kg. It would boil 3600*15/200 = 270 kg of nitrogen every hour. Just insane.

virtuallynathan• 5 years ago

It’s water cooled for sure, perhaps with chilled water, but all integrated into the 15U chassis. There are cutaway images of the v1 system online.

KETpXDDzR2• 5 years ago

I think I once saw one of the founders with a wafer in an In n' out with a potential investor. Looking at what Apple achieved with their M1A and the demand for "AI" - or training neutral networks, what it really is - they have a lot of potential. At least as long as the AI bubble doesn't burst.

arisAlexis3• 5 years ago

How can the future burst? It's like saying medicine will burst or physics

lainga• 5 years ago

I don't know, expert systems haven't been so hot lately...

mattkrause• 5 years ago

Same way as the last time: rampant over-promising and under-delivering, maybe catalyzed by some high profile mishaps.

Bubbles popping aren’t always evaluations of the objective quality of something; it could just be about its assumed value relative to other plausible options. Homes mostly don’t become uninhabitable when a real estate bubble pops.

foobiekr2• 5 years ago

Personalized medicine, as an example, absolutely burst.

imtringued3• 5 years ago

Right now everyone is betting that you can solve all problems just by throwing enough hardware at the problem. It's attractive for investors because they get to use their fat stacks of cash and corner the market but if this bet doesn't work out then we will have to go back to the drawing board and figure out how to build NNs efficiently and investors are not interested in this at all. They just want to launch a product as quickly as possible.

zitterbewegung2• 5 years ago

Much better article from anandtech at https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...

wyxuan• 5 years ago

Here's a video made by the author, Ian Cuttress, which goes into more detail as well.

https://www.youtube.com/watch?v=FNd94_XaVlY

smithza• 5 years ago

This is very cool. The explanation of how yield/defects was interesting: they can bypass cores with defects due to channeling and account for the statistical defects, allowing them to have a 100% yield.

XCSme• 5 years ago

I laughed at the "arm+leg" official cost of the CPU.

EDIT: Or GPU, or whatever it is.

cpr• 5 years ago

Interesting--the placement of code-to-processor so that things will be done roughly at the same time sounds a lot like the VLIW compiler problem of scheduling execution units so things are available at exactly the right time, without hardware interlocks.

durst4• 5 years ago

Does anyone here have firsthand experience using the compiler? Can you give a rough approximation of performance tuning with the compiler compared to performance tuning with compilers targeting the tensor cores on an A100 or a TPU?

terafo• 5 years ago

Is there any information on how this thing performs on actual AI workloads? It is priced similarly to a dozen of DGX A100s, but is it faster on something like training big transformer models(such as CLIP or GPT-3)?

01100011• 5 years ago

This thing is awesome and, as someone working for a competitor, kind of scary. I applaud their approach though. I think we're a couple years off from it, but we'll probably see wider adoption of larger silicon, with more specialized functional units, which are used with a lower duty cycle to manage heat. If nothing else, they're probably developing some good IP and techniques to handle other sorts of ultra-mega-insane-scale-integration.

I wonder what their software stack looks like. Can they support the sort of virtualization and sharing you'd want to keep this expensive beast fully utilized 24/7?

ohazi• 5 years ago

LSI, VLSI, UMISI

I like it!

In previous articles they've gone into some detail about how they deal with reticle limits, jumping over the scribe line area, and other n stuff. Between that, chiplets, HBM-style die stacks, etc... the developments here have been more interesting than I expected.

zetazzed2• 5 years ago

I wonder what the dev story looks like for these? I know they say "just use TF/Pytorch" but surely developers need to actually test stuff out on silicon and run CI on code... do they offer a baby version for testing?

AlexCoventry• 5 years ago

Imagine a beowulf cluster of these.

faichai• 5 years ago

This made me chuckle!

michelpp2• 5 years ago

This thing needs a GraphBLAS[1] implementation yesterday. 100 billion edge graphs and up are the new norm. This monster could smoke the competition if the implementation was tuned right!

[1] http://graphblas.org

blueyes• 5 years ago

Creating the ecosystem of both software and adjacent hardware for wafers this size is the real challenge for a company like Cerebras (which is doing amazing work). At first, they thought they just needed to make a chip 56x the size of its predecessor, and somehow get around the issue of defects and yield. After they solved those problems (which blocked Gene Amdahl, among others), they found they needed to bring an entire ecosystem into being to work with their hardware.

michelpp2• 5 years ago

Agreed, that's why I think the GraphBLAS would be such a great fit for this hardware. The ecosystem is growing pretty fast. There are, for example Python bindings, you can do sparse 'A @ B' on millions of elements in parallel with this wafer-chip. MATLAB 2021a now has GraphBLAS built in, you could drive this thing directly from your notebooks.

I'm sure there's a compiler and low level primitives to really get the maximum performance out of it, but the trade-off maybe worth it in many cases to do it using an abstraction like the linear algebra approach.

lastrajput• 5 years ago

Who said Moore’s law is dead?

miohtama• 5 years ago

It’s not dead, but melting.

MangoCoffee• 5 years ago

its not a law but an observation.

sillysaurusx8• 5 years ago

I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way.

The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples.

And that’s just for gpt — you lose all the other libraries people have written. This matters more in GAN training, since for example you can find someone else’s FID implementation and drop it in without too much hassle. But with this specialized chip, you’d have to write it from scratch.

We had a similar situation in gamedev circa 2003-2009. Practically every year there was a new GPU, which boasted similar architectural improvements. But, for all its flaws, GL made these improvements “drop-in” —- just opt in to the new extension, and keep writing your gl code as you have been.

Ditto for direct3d, except they took the attitude of “limit to a specific API, not arbitrary extensions.” (Pixel shader 2.0 was an awesome upgrade from 1.1.)

AI has no such standards, and it hurts. The M1 GPU in my new Air is supposedly ready to do AI training. Imagine my surprise when I loaded up tensorflow and saw that it doesn’t support any GPU devices whatsoever. They seem to transparently rewrite the cpu ops to run on the gpu automatically, which isn’t the expected behavior.

So I dig into Apple’s actual api for doing training, and holy cow, that looks miserable to write in swift. I like how much control it gives you over allocation patterns, but I can’t imagine trying to do serious work in it on a daily basis.

What we need is a unified API that can easily support multiple backends — something like “pytorch, but just enough pytorch to trick everybody” since supporting the full api seems to be beyond hardware vendors’ capabilities at the moment. (Lookin’ at you, google. Love ya though.)

whimsicalism• 5 years ago

I'm on board with you that there should be a "drop-in" cross support of these chips, but pytorch is at a way higher abstraction level than what should be commonly supported.

socialdemocrat• 5 years ago

Maybe people will get to their senses and switch to Julia instead of having to waste all these time on Python bindings.

habibur2• 5 years ago

Python here basically works as a binder for C, which is what everything is written in.

dnautics2• 5 years ago

Honestly I think julia missed the boat. There was an opportunity for julia to treat the gpu as an abstract distributed node (which is a first class concept in julia) and pay respect to the reality of data gravity. But they chose to instead basically treat the gpu as a synchronous entity.

I'm now bullish on elixir-nx because I think there's an outside shot they will get it right.

Python is a lost cause.

zucker42• 5 years ago

The philosophy here seems to be “if we build it, they’ll buy it.”

Supposedly Cerebras is already profitable, so it's hardly a situation where they are building something and hoping people buy it eventually.

That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples.

Again, based on the companies representations, Cebebras transparently supports Pytorch and Tensorflow, only requiring a few lines of changed code.

Source: https://www.anandtech.com/show/16626/cerebras-unveils-wafer-... (Dr. Cutress's video on TechTechPotato is also good).

sillysaurusx8• 5 years ago

I’ve learned to be skeptical of such claims. TPUs made that same claim, but it’s hard to use it for real work, since you don’t have access to the 300GB TPU host memory / cpu, where you infeed training examples.

There is a very specific test for “supports pytorch / tendorflow”: show an MLPerf imagenet resnet benchmark. It’s impossible to fake that. If you come anywhere close to TPUs in tensorflow, people (like me) will leap: https://mlcommons.org/en/training-normal-06/

Till then, it’s a “proof, please” type of situation.

habibur2• 5 years ago

So we are looking for OpenCUDA for AI. Like OpenGL in the 90s.

DesiLurker• 5 years ago

So basically OpenCL?

zitterbewegung2• 5 years ago

I think apple expects you to use pretrained models and convert them to CoreML.

There is also webGPU which is experimental on Safari and Chrome. But that doesn’t touch the ML accelerators on the M1.

systemvoltage3• 5 years ago

You're bearish on new hardware for AI training because it takes 2 months to setup? That's like saying you're bearish on the invention of transistor because there are no computers yet. This chip is for research organizations such as LLNL and ANL. Tooling and commoditization will happen.

“if we build it, they’ll buy it.”

This is literally how any new thing is invented and commercialized.

sillysaurusx8• 5 years ago

Well, two months has a way of turning into two years. June 2019 was right around when I fell in love with TPUs, and we’re still happily TPUing today.

Sometimes, though, it would be nice to simply focus on doing interesting ML work rather than wrestling with the intricacies of tensor slicing and memory access pattern optimization (to say nothing of the cursed inability to manually manage memory, resulting in explosions of allocations in the forward pass for unclear reasons).

Let’s put it this way. If you contract me to implement a full gpt model on this new hardware, two months full time work would be my minimum estimate. That’s 40 hours a week of focused effort, with no breaks and no other projects. I don’t know about you, but two months with nothing to show can demoralize most teams I’ve worked with.

The point was, we’re early in ML’s lifecycle. And, like React for webdev, people are slow to change their habits — for better or worse, pytorch and tensorflow are the APIs people think in. So if you want your hardware to be widely adopted, you need tooling that supports the workflows people have spent months learning.

Jax is on the horizon too. Supposedly they’re launching something soon that might tip the scale in their favor. Perhaps an ambitious hardware vendor could capture the future market by preemptively implementing Jax support. But perhaps not: at that point you’d be competing toe to toe with Google’s TPU offering, since Jax is Google’s dogfood.

Right now my money is on TPUs, partly because of their fantastic support staff. But maybe some other company will come along and offer a better integrated cloud experience.

oivey3• 5 years ago

Graphics APIs and deep learning frameworks are really nothing alike. In graphics, save for a few new features, the goal every year is the same: draw more and faster. Deep learning is evolving much faster than that. The answer isn’t an API; it’s powerful and mostly transparent automatic differentiation. Anything that is just an API will forever be behind and is much less useful for active research. There are a few projects around improving AD support, but it’s a relatively difficult problem because to do it best you need deep compiler integration.

sanxiyn4• 5 years ago

The answer is an API, like NNAPI. AD is a frontend concern and doesn't really matter to accelerators.

For AD, I am bullish for Enzyme, which does AD on LLVM IR, avoiding deep compiler integration: https://enzyme.mit.edu/

justaguy88• 5 years ago

You might like this talk that describes parts of the compiler ecosystem that are lacking standardization for accelerators

https://www.youtube.com/watch?v=4HgShra-KnY

sanxiyn4• 5 years ago

What we need is a unified API that can easily support multiple backends

Android standardized NNAPI and on-device inference is in pretty good shape. As you said, training is different matter.

de6u99er• 5 years ago

Thr Anandtech article from a other comment here indicates that both PyTorch and Tensorflow are already running on it.

A key to the design is the custom graph compiler, that takes pyTorch or TensorFlow and maps each layer to a physical part of the chip, allowing for asynchronous compute as the data flows through.

https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...

jquery• 5 years ago

We had a similar situation in gamedev circa 2003-2009. Practically every year there was a new GPU, which boasted similar architectural improvements.

You mean one of the most rapid periods of graphical improvement in history? Growth was just too breakneck to standardize for a bit.

MaxBarraclough• 5 years ago

sillysaurusx's comment goes on to acknowledge that the major graphics libraries (OpenGL and Direct3D) did a pretty good job of abstracting away/indirecting away the architectural changes. As you indicate, the architectural changes were made for good reason, and translated to real improvements in performance/capabilities.

If AI acceleration chips aren't able to offer similar API standardisation/stability then it's not the same as what happened with GPUs, which was sillysaurusx's point.

TFortunato• 5 years ago

The article mentions that developers use standard PyTorch and Tensorflow.

docfort• 5 years ago

You might be interested in this for your M1 MBA: https://github.com/apple/tensorflow_macos

sillysaurusx8• 5 years ago

That was actually what I was referring to! It's super weird. I like it, and it's what I use primarily, but only because no other version of tensorflow will install.

Their compiler is closed source, so it's almost impossible to tell what's going on. But, when I connect using tf.Session(), then .list_devices() shows only CPU available.

However, when I enable the TF_MLC_LOGGING=1 magic variable, it does seem to be printing out messages that indicates it's doing some kind of graph substitution under the hood. Therefore, I assume that this is the intended usage mode.

In other words, there seems to be zero difference between the "CPU" and the "GPU". Normally you can say "Do this on the CPU" while "do that on the GPU." But not with this.

Hopefully they'll open source the code sometime this century so that it's clearer what the heck it's doing. For now, though, it's reasonably fast in whatever this "CPU" mode is -- I only need to run unit tests on my laptop anyway, since all training happens on TPUs. So I ended up happy.

(For the first day or so, I was panicking that I was going to have no working tensorflow whatsoever on my M1 laptop, which would've necessitated a swift return + substitution.)

blihp• 5 years ago

The philosophy is working as it often does in the early days/years of a technology. The customers for this have problems at an entirely different scale. It's worth it for them to drop millions on hardware and potentially millions more to (re)write the software as needed. Sure, they'd prefer to be buying commodity hardware with standardized APIs. But since that doesn't currently exist for this class of performance, they'll pay up. People did the same back in the days of Cray/SGI/Lisp Machines and so on. These custom solutions can be quite profitable until the competition gets to the 'good enough' stage and then orders start drying up.

If you're looking for standardized hardware and software for AI, Apple is the wrong platform to be on. I'm fairly confident it won't be happening there. (I don't say that to be a hater: see the direction they've gone re: GPU APIs... it's just Apple being Apple)

dmingod666• 5 years ago

If you look at it historically and put it in scale with the AI winter, 2 months is next to nothing..

solidasparagus2• 5 years ago

I don't really agree, at least from a longer term perspective. It's early days yet, but XLA seems to be a promising intermediate representation for allowing the DL frameworks to run on a wider array of hardware without user-facing software changes. It has traction with Google, NVIDIA, IIRC Intel and maybe more (others are definitely using the same approach of compute graph splitting and subgraph scheduling, but I'm not certain if they are using XLA specifically - I know some aren't like Mindspore).

XLA has already proven its value by allowing PyTorch to run on TPUs (shittily, but that appears to be more of a VM/GCP infra problem than an XLA problem). The work done for TPUs (and to a lesser extent for GPU optimization) has started to expose some of the major issues and so work can start on addressing them (the cost of dynamic XLA compilation as tensor shapes change and how lots of important code assumes that accelerator-to-CPU communication isn't tooo expensive, but it's is a huge issue when trying to compile the graph into machine-specific code with XLA or similar to because it forces you to only be able to compile small subgraphs).

It's early, but the rise of a really effective IR in XLA combined with the huge amount of resources that Google/NVIDIA can pour into XLA makes me very bullish on purpose-built hardware for AI training. It will take a while I admit.

sillysaurusx8• 5 years ago

Mr/mrs anonymous HN person, please put some info in your profile. You clearly have some deep knowledge of TPUs that I didn’t expect to pop up offhandedly on HN. You’re correct on all counts: dynamic tensor shapes are more or less impossible with XLA, making it more or less impossible to train a model with arbitrary image size inputs, even though the math would allow for that; the pytorch XLA work on TPUs is indeed kind of shitty, and I’m surprised as heck that literally anyone said this except me; and XLA as an IR is promising for portability. Now I’m curious what you’ve been doing to have experienced these things, since there didn’t seem to be many others who have (or at least, who are vocal about it).

I agree with you, but I think we differ on our timetables. I am bearish for the next two years, at which point I’ll awaken from my slumber and become a flaming bull. (It helps to remember that “we overestimate the impact of years, but underestimate the impact of decades.” I try to plan accordingly.)

In other words, if you’re bullish that two years from now we’ll start seeing portability implemented in the field across various HPC chips, then we fully agree. But that’s also a glacial pace; GPT-2 changed the world almost two years ago now, and DALL-E seems to be the next frontier for doing interesting generative work. So, we’ll split the difference and say that the bears and bulls will meet in two years for a deep learning hackathon. As a bonus, the pandemic will be over by then, so it can be an in-person meetup.

cs702• 5 years ago

from a longer term perspective

Yes.

Longer term, new hardware will also make it practical to train large models in a fully parallelized, fully distributed manner -- i.e., without having to backpropagate gradients, which requires a lot of complex bookkeeping and plumbing for distributed training.

Recent progress suggests this will happen. See, for example:

https://arxiv.org/abs/2006.04182

https://arxiv.org/abs/2103.03725

https://arxiv.org/abs/2010.01047

I for one am excited to see what happens over the next decade as it becomes trivial to train/use models with 1K, 1M, or 1B times more dense connections than present state-of-the-art models.

fromaj• 5 years ago

But they do have PyTorch support??? They ship a 'cerebras graph compiler' extension for pytorch that allows you to replace the default optimizer with theirs.

londons_explore• 5 years ago

That means you’re looking at two months of R&D minimum to get everything rewritten,

For a hobbiest, sure, that's a problem.

But for a big company with a big ML research team already, that isn't an issue - they just assign a few people to work on it, and in a few months it's done. If you're running any model at scale anyway you probably want to rewrite everything to make it run efficiently on your hardware.

noobydoobydoo• 5 years ago

What does the programming model look like for one of these (like at the assembly level)? I’m not even sure what to google.

durst4• 5 years ago

The article states that the programming model is TensorFlow or PyTorch. However, I'm not clear on the details of customizing TensorFlow or PyTorch code to run efficiently on this chip.

userbinator• 5 years ago

I suspect it's more like an FPGA - massively parallel structures that don't fit traditional "programming" well.

Incidentally, FPGAs are also insanely expensive at the high end.

syntaxing• 5 years ago

Super curious, how much do one of these cost? Like 10K,100K, or 1M range?

thunkshift1• 5 years ago

First gen was ~ 2.5M usd per machine as per the video link

LegitShady2• 5 years ago

Any article or headline talking about transistor count is just a puff piece. The performance matters, the number of transistors never does, and I have never purchases anything based on the number of transistors.

mousepilot• 5 years ago

I'm not going to buy one, but its more because I don't have several million dollars to spare.

LegitShady2• 5 years ago

I get that, but it's just a thing I've noticed about tech puff pieces. If the transistor count is front and center it's probably not worth reading.

v8dev123• 5 years ago

Why it's called as "AI"?

macksd• 5 years ago

Designed for deep learning

intrasight2• 5 years ago

Interesting consider that if this "chip" was manufactured with the feature size of the MC68000, the chip would be about 1/4 mile square.

intrasight2• 5 years ago

Or another way to grok the process shrinkage is to consider that the MC68000, if manufactured with current process dimensions, would fit within the space of a single gate of the original process.

TedDoesntTalk• 5 years ago

the size ... [is] ... 46,225 square millimeters

According to google calculations, this is just under 6 square feet.

Phil-bitplex• 5 years ago

I think it might be closer to 215mm x 215mm or 0.49 square feet.

Though that is unimaginably large to me for a silicon chip. I can't comprehend it - the engineering behind it would be incredible.

deathanatos• 5 years ago

No…? https://www.google.com/search?q=46%2C225+square+millimeters+...

0.49756176 square foot

jzer0cool• 5 years ago

This is the first I'm coming across Cerebras and interested here to understand more. Are these already in use integrated in some pc/servers or is it some dedicated server which you need to integrate into some motherboard, ...

kumarski• 5 years ago

Polyhedral LLVM compiler optimization > Quantum Computing Technology

Well at least on relevance* to active problems.

lxe• 5 years ago

That chip needed some 20,000 amps of current fed to it from one million copper connections to a fiberglass circuit board atop the wafer.

Is the 20,000 amp number not an error?

Karliss• 5 years ago

20000 amps seems plausible. A regular desktop CPU operates at 1-2 Volts and power consumption of 100-200 watts which means 50-200 Amps.

Comparison table shows nvidia A100 which has max power consumption of 400W. This has roughly 50 times the transistors and 50 surface area so having 50 times the power consumption 50*400=20000W and same order of magnitude of current seems possible.

magicalhippo• 5 years ago

They also say the peak power draw is 23kW. Now 23kW/20kA = 1.1V which is about what my NVIDIA 2080 GPU uses at load. So that does indeed seem to work out.

Of course 20kA is pretty insane, would be interesting to see just how they feed the power to this thing. Bond wires going to the middle of the chip? And the local regulator must be a beast, or rather lots of beasts.

ksec• 5 years ago

For those interested, here is an article [1] on glimpse of how it is cooled. Although I still dont understand how this is possible though. 20kW.

[1] https://www.eetimes.com/powering-and-cooling-a-wafer-scale-d...

Bancakes• 5 years ago

My canvas fingerprint addon went berserk soon as I visited that link. 45 scripts and 14 fonts as well. Thanks for the link, though.

Cerebras’ new monster AI chip adds 1.4T transistors

Related Stories