Jim Keller criticizes Nvidia's CUDA, x86

https://www.tomshardware.com/tech-industry/artificial-intelligence/jim-keller-criticizes-nvidias-cuda-and-x86-cudas-a-swamp-not-a-moat-x86-was-a-swamp-too

by flykespice • 2 years ago

197 134 2 years ago

tomshardware.com

rerx• 2 years ago

"Basically nobody writes CUDA," wrote Keller in a follow-up post. "If you do write CUDA, it is probably not fast. […] There is a good reason there is Triton, Tensor RT, Neon, and Mojo."
Even Nvidia itself has tools that do not exclusively rely on CUDA. For example, Triton Inference Server is an open-source tool by Nvidia that simplifies deploying AI models at scale, supporting frameworks like TensorFlow, PyTorch, and ONNX. Triton also provides features like model versioning, multi-model serving, and concurrent model execution to optimize the utilization of GPU and CPU resources.
Nvidia's TensorRT is a high-performance deep learning inference optimizer and runtime library that accelerates deep learning inference on Nvidia GPUs. [...]

Keller was speaking of OpenAI's Triton (https://openai.com/research/triton), a Python-like language that is compiled to code for Nvidia GPUs, but Tom's Hardware mixed this up with Nvidia's Triton Inference Server, a higher level tool that's really not a replacement for CUDA and not directly related to the Triton language. Easy to confuse these if you are a writer in a hurry.

londons_explore• 2 years ago

Indeed - Keller is a low level hardware guy, and isn't going to have much interest in model versioning...

95014_refugee2• 2 years ago

Keller is a manager. He was a gateware engineer.

chrisjc• 2 years ago

Isn't low-level hardware really at the heart of a lot of this? Hasn't a lot of the criticism of Cuda been that it's incredibly difficult for others to implement on other hardware bc of the low-level interactions and Nvidia's usage of dark-APIs (can't recall the term i've heard used).

Wasn't this one of the reasons AMD abandoned/deprioritized their efforts on such a project?

p1esk• 2 years ago

Jim Keller works for Tenstorrent - direct Nvidia competitor.

diggan2• 2 years ago

Wow, that's some omission in the article. Mentioned in the very bottom, but with no disclaimer that it might influence his opinion as they're a competitor:

His statements also imply that even though he has worked stints at some of the largest chipmakers in the world, including the likes of Apple, Intel, AMD, Broadcom (and now Tenstorrent), we might not see his name on the Nvidia roster any time soon.

shash3• 2 years ago

"Works for" is one way to put it - he's CEO and (I think?) co-founder..

RcouF1uZ4gsC• 2 years ago

Most developers would rather write code for a “swamp” that has long term backwards compatibility, than for a perfect ivory tower that will get replaced in 2 years by another more perfect ivory tower.

frfl2• 2 years ago

Interesting that the web ecosystem basically works on the ivory tower model. Roughly every 2 years there's some new ivory tower and a lot of greenfield projects move to that. Brownfield projects become legacy and go into maintenance mode or get rewritten after some number of years.

I've seen something like this:

- Delphi desktop app (some time in the 2000s)

- Adobe Flash web app (in late 2000s to mid 2010s)

- 2015-era JS framework web app (mid 2010s to present, considered legacy codebase now, no feature development)

- 2020-era JS framework web app (late 2010s to present)

I expect another major rewrite or move to a new project in the next ~5 years

spenczar5• 2 years ago

The web frontend works that way because little depends on the frontend. But the infrastructure one layer down is obsessive about backwards compatibility, both in the browser and in HTTP servers. JavaScript frameworks change all the time, but nginx and apache and friends have lasted for many decades, for good reason.

pjmlp5• 2 years ago

Depends where you look into, I have been pretty much Java and .NET since 2005 in what concerns Web development and backend infrastructure in distributed systems.

Vvector2• 2 years ago

Swamps keep users trapped in, by making it difficult for a competitor to make a compatible alternative. Had Nvidia (or Intel or Microsoft) kept their product lean by aggressively removing backwards compatibility, they would have lost users AND made it easier for another company to compete.

blackoil• 2 years ago

Yeah for some reason people deny to understand that this messy swamp is the reason things are successful. Be it x86, Windows, PDF...

mschuster91• 2 years ago

PDF hasn't been a vendor-locked swamp for many years now. It got successful because it was better than TIFF for scanned images and better than raw Word/Postscript for print products.

rob74• 2 years ago

So a swamp is even better than a moat? And here I was thinking that Keller is insulting Nvidia, while he's actually giving them compliments...

mobilio• 2 years ago

ROCm isn't alternative too: https://geohot.github.io/blog/jekyll/update/2023/06/07/a-div...

kamikaz1k• 2 years ago

I think that's a misunderstanding of the analogy. The moat isn't about trapping users.

Moat as its typically used is to say that a business is protected by the moat, the prize being the castle/business. The residents/business are free to build and prosper, be productive in other words.

When Keller calls it a swamp, he's trying to say there's no castle to be productive in. The swamp makes it a mess, and thus hard for the business to actually maintain productivity. It is still difficult for the invaders to break in (?) but they don't need to...because it's a swamp. They should be busy making a better castle -- which is what Keller is trying to do.

Disagree with him or not, but I think that's the analogy.

robocat• 2 years ago

Or maybe just humorously alluding to swamp castle in Monty Python and the Holy Grail. Not sure where you are getting your analogy from, but my guess is Jim's not really making an analogy. Mixing a batch of metaphors is not cooking chips.

https://www.youtube.com/watch?v=aNaXdLWt17A

https://montypython.fandom.com/wiki/Swamp_Castle

roenxi• 2 years ago

CUDA seems to have a lot of mind-share as a strategic asset, but all the evidence I've seen is that AMD defeated themselves. I've never seen them fail due to lacking features or capability, and I haven't heard stories of it either. The complaint seems to be, consistently, that the drivers crash.

If CUDA is so good, it'd be great to know what it does that AMD cards can't. I've never gotten that far because I hit what seems to be some sort of kernel panic or driver lockup.

Vvector2• 2 years ago

If CUDA is so good, it'd be great to know what it does that AMD cards can't.

You are comparing Nvidia software with AMD hardware. It's the AMD software that has been lacking historically.

EDIT: here is a discussion from 2 months ago on CUDA versus ROCm. https://news.ycombinator.com/item?id=38700060

declaredapple• 2 years ago

I've never seen them fail due to lacking features or capability, and I haven't heard stories of it either.

I've seen a mix of both for years. For the last 6 years the story has usually been some combination of "X program technically supports it, but you'll get 20% of the performance you'd expect given the hardware and/or you'll need to go through a bunch of hoops, and even then you'll have to troubleshoot tons of random errors and/or it'll crash the program or system"

That's on top of the "nobody supports ROCm, or if they do it's because of a single person who amd the PR - you're on your own though because none of the core contributers have amd hardware" which I'll admit is a chick-egg problem.

But given these two factors, it's always meant "if you want buy hardware to do data science, you have to go nvidia if you don't want to write the support yourself, and/or an insane headache that often resulting in switching to nvidia hardware anyway"

slices• 2 years ago

Supporting anecdote: I built a PC a few years ago, and first tried multiple AMD cards. All of them worked fine while using the default windows drivers, but as soon as I installed AMD's drivers, they stopped working. Spent weeks troubleshooting, no luck.

Gave up, bought an Nvidia card, zero problems.

belval• 2 years ago

"Basically nobody writes CUDA," wrote Keller in a follow-up post. "If you do write CUDA, it is probably not fast.

I am a big fan of Jim Keller, but this is semantics that argues CUDA the language is not a moat, when most people refer to CUDA the libraries.

CUDA has first party support in most of the libraries in use today, if you were to stray from that happy path you must have deep pocket to work through the hurdles of working with something like XLA.

There's a reason why Microsoft, Google, Amazon and Meta are still buying Nvidia accelerators even when they have their own in-house accelerators.

dist-epoch2• 2 years ago

The impossibility of buying NVIDIA GPUs in quantity today is a giant incentive to port the most important libraries to AMD/....

photonbeam• 2 years ago

Im surprised we dont hear more about this, it seems like a huge motivator

cjbgkagh3• 2 years ago

I think you’re assuming these big companies are serious about AI itself and not just joining in on a trend. They would much rather spend huge amounts of money on chips than hire a competent researcher who might tell them that they didn’t really need to waste so much money.

hackerlight2• 2 years ago

Andrew Ng says ROCm is a lot better today than early 2023

pjmlp5• 2 years ago

The computing shops on my city shopping malls have enough laptops with NVIDIA GPUs, while they aren't at the same level as desktop or server GPUs, there are available and usable for people to learn CUDA, quite the contrary with most AMD offerings.

westhanover2• 2 years ago

Well he is obviously just talking his book. He is selling a Nvidia competitor. If CUDA isn’t beautiful and the future deserves to be beautiful, why don’t you make it and show us, Jim? All we are getting is more hot air about CUDA. That is all AMD and Intel have as well, bold statements about the future and no code.

neural_thing• 2 years ago

He isn't though. He hasn't been at AMD or Intel for quite some time.

HarHarVeryFunny2• 2 years ago

CUDA/etc is certainly a pain.

A good number of years ago I wrote my own Torch-like C++ NN framework using CUDA (cuDNN, cuBLAS) for the GPU, and one of the annoyances is that cuDNN has incomplete coverage of even the basic operators needed. Add/Sub/Min/Max/Sqrt/Negate are all provided as part of cuDNN, but if you want other common NN building blocks like Div/Exp/Log/Pow/Inv/InvSqrt then you have to write them yourself in CUDA and either forgo cuDNNs tensor-descriptor layout flexibility or re-implement that yourself too.

Of course frameworks like PyTorch support all the operators you'd expect, since they've written their own kernels in CUDA where the functionality is missing from cuDNN.

It's not clear exactly what Keller is referring to there. When people say CUDA they might be referring to the entire ecosystem (nvcc CUDA C/C++ compiler, CUDA API's, higher level cuBLAS, cuDNN, etc), or maybe just the base compiler (which lets you write your own kernels) and API for allocating memory, queueing kernels, etc.

The cuDNN kernels (convolution, etc) are highly optimized, as is cuBLAS (e.g. matmul), and I doubt anyone is going to do better writing these themselves. Does Keller consider using cuDNN as "writing CUDA" ?

As far as I'm aware the higher level, performant, CUDA libraries, as well as specialized components like TensorRT are written in CUDA, although that could mean a combination of C/C++ & PTX pseudo-assembler (ptxas is really a compiler, not an assembler). The alternative would be they they were written in hand optimized SASS assembler which afaik is only available outside of NVDIA via the Open Source CuAssembler.

I believe Mojo support for NVIDA is based on PTX. I'm not sure if that would be really be considered as "CUDA" or not if they are not using nvcc at all.

Most people, outside of framework vendors, would have no reason to use CUDA anyway, since it's just too low level. The only sane use case would be where writing a custom kernel in (e.g.) PyTorch or Mojo doesn't get the performance you want and you write than one kernel in CUDA. The hope would be that the Mojo compiler is good enough that this would not be necessary.

m463• 2 years ago

"Basically nobody writes CUDA,"

Can't help but think of that old yogi berra quote:

"Nobody goes there anymore, it's too crowded"

(but yeah, I suspect a lot of people use cuda via accelerated libraries/apps like opencv)

throw0101c• 2 years ago

Given the rise in ARM in many areas of computing, I'm not sure how much of a moat x86 is / will be long-term.

Is there anything 'inherit' to the instruction set that cannot be done anywhere else? I know that many Intel/AMD systems now have a bazillion cores, which is very handy for some things. Also, AMD seems to have a large numbers PCIe lanes, which is great for I/O between (e.g.) the network and on-system stuff.

blinkingled• 2 years ago

OTOH I have been hearing about ARM's rise since a long time now. It's not just performance per watt and even that's just Apple beating everyone. See Windows on ARM.

x64 is here to stay with its incredible ecosystem and OSS friendliness - Coreboot, Intel and AMD GPU drivers, great choice of operating systems and apps with a fairly long usable lifespan even with Windows updates.

frfl2• 2 years ago

I came across an interesting fact about X86,

currently there are actually no modern x86 CPUs on the market. Both Intel and AMD don't actually use x86 cores, but instead proprietary RISC cores, with microcode that translates the x86 code to RISC code on the fly at execution time.

https://cs.stackexchange.com/questions/132211/what-are-the-a... This cs.stackexchange link is a good read.

Wikipedia also states something similar,

In the P6 and later microarchitectures, x86 instructions are internally converted into simpler RISC-style micro-operations that are specific to a particular processor and stepping level

https://en.wikipedia.org/wiki/Intel_Microcode

jerf• 2 years ago

Also, the more transistors we stuff on to a chip, a process that is still continuing, the less are proportionally dedicated to these decoding pipelines. My impression is that they are already relatively insignificant parts of the chips, though corrections welcome.

I wonder if there's any ISA that could be written against the x86 cores that would be more efficient. In theory the chip could use that as a mode, so the chip itself wouldn't require an entire OS to shift before things could use it. I don't know anywhere near enough about the inside of an x86 core to have even a clue if such a thing would be possible. But it would be an interesting escape hatch from x86. Anyone who can flesh this idea out with knowledge of the x86 core internals is welcome to explain to me why my idea is bad and I should feel bad.

kryptiskt• 2 years ago

By that account, there never has been any x86 CPU, since the OG 8086 had a microcode ROM and translated most instructions into micro-ops.

sjsdaiuasgdia• 2 years ago

currently there are actually no modern x86 CPUs on the market.

This feels like one of the stranger "no true Scotsman" arguments I've ever run across.

Even the 8086 had microcode that translated the instructions as generated by the compiler/programmer into the instructions that would be processed: http://www.righto.com/2022/11/how-8086-processors-microcode-...

I would love to know what the person who wrote that stackexchange answer would say in response to the question, "so which x86 processors used 'real' x86 cores?" Because from the very beginning of x86, there's been translation of the front-door opcodes into internal opcodes via microcode.

QuadmasterXLII• 2 years ago

Note that you couldn't directly simplify the architecture by just writing an assembler for the RISC microcode. The reason is that the RISC microcode is much larger than the x86 it is derived from. This on the fly decompression does wonders for cache hit percentage.

gpderetta2• 2 years ago

It seems I make this comment every 6 months or so, but... x86 uops are not at all RISC-style. They are very large (classic RISC uses 32bit encoding, uops are over 100 bits), variable size (classic RISC is fixed style, uops can use multiple slots to encode constants) and can encode load-op (classic RISC separates load and stores from operations).

There is also the fact that RISC-ness is a property of the ISA, so applying it to describe a microarchitecture doesn't make much sense.

edit: reference: https://www.quora.com/Why-are-RISC-processors-considered-fas... (sorry for the quora link)

samsartor• 2 years ago

Intel and AMD don't use x86 cores but [...] instead translate the x86 code to RISC code on the fly

I used to say this all the time and I've been informed that it's something of a misunderstanding. For example, most RISC-V processors also decompose instructions into multiple μops: https://docs.boom-core.org/en/latest/sections/execution-stag...

So it isn't like there is a literal RISC processor inside the x86 processor with a tiny little compiler sitting in the middle. It's just that the out-of-order execution model requires instructions to be broken up into subtasks which can separately queue at the core's various execution units. Even pipelining instructions still wastes a lot of silicon (while you're doing an integer add the floating point ALU is just sitting there, bored) so breaking things up this way greatly improves parallelism. As I understand it, modern μop-based processor cores can actually have dozens of ALUs, multiple load/store units, virtual->physical address translation units, etc all working together asynchronously to chug through the incoming instructions.

jcranmer• 2 years ago

currently there are actually no modern x86 CPUs on the market. Both Intel and AMD don't actually use x86 cores, but instead proprietary RISC cores, with microcode that translates the x86 code to RISC code on the fly at execution time.

This kind of factoid does more to obscure the truth than it does to illuminate it.

The truth of the matter is that all high-end CPUs do a µop translation, whether or not their frontend is a CISC or RISC ISA. Indeed, the very notion of CISC versus RISC is way overwrought in architecture textbooks, and this probably produces the garbled thinking: since everyone "knows" that CISC can't be superscalar, this means that the Pentium (in making superscalar x86) has to somehow be RISC.

Another thing to note is that there's not really anything called CISC. RISC is the overall term for a family of computer architecture design methodologies arising the 80's that argued for compiler-centric rather than assembler-centric design and simpler instructions, sometimes to the point that you omit hardware and call it a feature (e.g., delay slots). CISC is... everything else; it's a strawman constructed for RISC to compete against rather than a coherent design methodology.

In actual practice, though, RISC v CISC hasn't been relevant for decades. Some of the RISC design ideas have won out: there's generally a high emphasis on instructions that can be selected by the compiler over hand-tuned assembly, for example. But things like delay slots have been generally considered a failure. The architectures that are the most successful--x86 and ARM--are the ones that blur the line between RISC and CISC the most.

Actually, if you scrubbed the x86 assembly away and came up with some new assembly syntax (including new mnemonics of course), you could probably sell the x86 ISA as a "compressed RISC" ISA and get many people to believe you that it was designed as a RISC. x86 doesn't have many instructions that have crazy interrupt rules or multiple memory references (the string instructions are the main exceptions here), and it's this property which turns out to be really key to making something high-performance or not. It would be better for us to be honest about what enables or doesn't enable superscalar architectures rather than trying to argue that somehow x86 cheated its way to success.

imtringued4• 2 years ago

There are two ways to interpret this statement. First is that ISA is completely irrelevant since you can just convert to whatever is best. This is bad for RISC since there is literally no advantage gained by losing backwards compatibility.

The other interpretation is that micro ops are actually a microarchitectural optimization and that RISC processors should use it too. The compressed instruction set of RISC-V is leaning towards that direction. RISC-V processors internally convert RISC-V instructions to RISC-V.

It is kind of hollow to talk about it, since it is hardly clear to say that this is a unique disadvantage or advantage.

King1st• 2 years ago

ARM has abysmal SIMD support. Not even supporting AVX 256. for reference that is about a 10 year lag behind x86. Neon is not an adequate substitute. Additionally when AVX is used, the power draw of a ARM skyrockets to x86 level defeating the advantage ARM has over x86 while offering worse performance. ARM is good for things that require many threads and are not heavily dependent on incredibly high integer performance per thread. x86 is the dominate in high power, when individual thread power is more important due to application requirements or size limits or if they need to do anything with SIMD. ARM has a lot of push behind it. But it contrary to techbro hype., it is not a drop in replacement for x86 and I dont think it ever will without shooting themselves in the foot making them less efficient.

Earw0rm• 2 years ago

ARM has, optionally,the very capable SVE instruction set, but no consumer chips currently implement it.

The M1 and later big cores can dispatch four NEON FMA instructions per clock, so 512 bits worth of vector math, which compares OK with most Intel or AMD chips (Zen 4 can do two 256-bit MUL and two ADD, and Intel "client" bigcores since Sunny Cove typically do three 256-bit FMA).

owlbite• 2 years ago

Focussing on vector length is an error. You should care about throughput and number of different instructions you can retire.

If I have a core with four 128-bit neon vector units, I have the same throughput as an x86 with two AVX2 units or one AVX512 unit. However that 4x128-bit core is actually more flexible than the other two as I can do 4 different things at once, or 4 scalar operations per cycle. (Of course the downside is you spend more frontend resources on decode).

Given that most code isn't vector code, the multiple short vector length approach is actually superior on many common real-world workloads that aren't machine-learning (and CPU is unit-of-last-resort for large ML workloads anyway).

devnullbrain• 2 years ago

when AVX is used

spoiler: it isn't

jeroenhd• 2 years ago

Before Apple's M1, ARM was slow and near unusable for x86-class machines. There were some cheap ARM Chromebooks with a much shorter lifespan than their x86-based brethren, and an ARM tablet that Microsoft abandoned, and that's about it.

After the M1, the ARM companies needed to up their game, and Qualcom is rumoured to finally release something competitive to a real x64 chip in the near future. This isn't the first time they've made claims like these, though, and I very much doubt they'll live up to their promise.

Meanwhile, x64 has caught up to Apple in terms of compute power (especially per dollar, which is the reason x64 is so popular), is getting closer and closer to Apple's power consumption levels, and unless the M4 will have dramatically more performance, the ARM overtake will just have been an outlier.

I'm not so sure how long non-Apple ARM will be able to stay competitive given the turnaround x64 has managed to make in just a few short years. It looks like x64 still has plenty of room for improvement, and this proves that the reason compute performance has plateaued was that there was no real competition.

Kon-Peki• 2 years ago

This isn't the first time they've made claims like these, though, and I very much doubt they'll live up to their promise.

The difference now is that they actually have a set of customers that would buy a desktop-class chip if they produce one.

zokier2• 2 years ago

Before Apple's M1, ARM was slow and near unusable for x86-class machines

You ignore one huge class of x86 systems: servers. AWS Graviton launched 2 years before M1 and had competitive perf.

gpderetta2• 2 years ago

There is nothing inherent in the ISA. The moat of x86 is purely backward compatibility and decades of experience of ARM and Intel in building large systems.

boringuser22• 2 years ago

Out of curiosity:

Is it legal for China to hire Jim with an obscene pay package?

Why wouldn't they?

fsmv• 2 years ago

China is a country not a company

ThrowawayTestr• 2 years ago

Hard to tell sometimes

azinman22• 2 years ago

It’s blurred when the state owns so many companies

espadrine• 2 years ago

It is disallowed by the executive order “on Addressing United States Investments in Certain National Security Technologies and Products in Countries of Concern”: https://www.whitehouse.gov/briefing-room/presidential-action...

dfsgdfg• 2 years ago

I think Tenstorrent is based in Canada. At least Jim and his team are all from Canada, and their headquarters seems to be there. Probably more complicated than that but it isn't so clear if this would apply.

boringuser22• 2 years ago

That's a 2023 executive order.

It's hard to imagine that this gaping hole was left wide open until the deft hand of Joseph Biden took the wheel.

reubenmorais• 2 years ago

Every successful platform is a swamp. Human lives are short, so we do things hastily; communication and coordination is hard; zero sum views of competition are widespread; we don't know how to do sustainable fast iteration of ecosystems (or even slow iteration); the swampiness helps concentrate wealth, which is even more highly rewarded.

Cities are a swamp. The Web is a swamp. Linux is a swamp. Capitalism is a swamp. Democracy is a swamp.

the_panopticon• 2 years ago

well said. Reminds me of a comment I received from a manager years ago: "This component is great, I haven't heard of any bugs." My retort was: "It's because no one is using it." Fast forward 10 years. Many bugs, CVE's, complaints of back-compatibility, challenges in porting/maintenance. Show me a technology w/o complains or bugs or technical debut & I suspect you'll find a similarly rarely or not used one. Swampiness is an emergent property of 'success at scale' in usage in a sense.

brucethemoose22• 2 years ago

"Basically nobody writes CUDA,"

Quite a thing to say as head of an AI hardware startup.

Most of the machine learning libraries we use/test in production (and that I use personally on my desktop) have hand written, extensively optimized CUDA kernels. It would be problem #1 switching to Tenstorrent hardware.

Not that I like that one bit, or that I really understand the optimization of these libraries. I do see triton code as well, but it still seems to be the suboptimal path.

qwertox3• 2 years ago

But what kind of CUDA kernels have you written? Kernels for AI training and inferencing? Because I think that's what he's trying to say.

I doubt he's saying that in the context of CFD or other similar ones.

brucethemoose22• 2 years ago

If you are implying we are at the mercy of a few CUDA experts because we can't efficiently write it ourselves, you would be correct.

Triton does seem way more approachable as something I'd like to pick up.

PeterisP• 2 years ago

But why do I need to write CUDA kernels? The CUDA kernels I need already exist, and I don't particularly care if writing them was so horrible that it required scratching your eyes out and sacrificing kittens to demons - I would care about the (re)writing effort only if I was considering other hardware that doesn't yet have the support that CUDA has as then I'd actually have to do that myself.

slashdev• 2 years ago

Hot take: x86 was a swamp, but overtime we stopped using legacy features, and the instructions got better, and the amount of chip resources needed to implement the CISC architecture decreased to a small percentage of the total, such that it sort of just turned into a wierd compression algorithm for encoding RISC instructions and didn't matter much anymore.

qwertox3• 2 years ago

I think it's more of a subtle publicity stunt for Tenstorrent's Grayskull [0] cards, which are highly optimized for inference.

With CUDA you get a lot of flexibility which you don't get with Grayskull, at the expense of having a more layered/patched API which has been evolving for many years while R&D came up with new solutions to new problems.

I wonder what Nvidia's capabilities are in creating new, highly optimized cards, similar to Grayskull, but I wouldn't be surprised if they don't have any interest in creating them since their current products are already consuming all their resources.

It's great to see that companies like Tenstorrent are offering more optimized and pricewise more accessible products. It would also ease the situation with how hard it is to get Nvidia cards, because even companies doing only inferencing are buying chips which are developed to be able to do much more than that.

[0] https://tenstorrent.com/cards/

camel-cdr• 2 years ago

which are highly optimized for inference.

My understanding is that grayskull is a devboard for inference, Wormhole is supposed to do both, because you can scale it out with multiple cards.

With CUDA you get a lot of flexibility which you don't get with Grayskull

Do you? From what I can tel tt-metalium gives you quite low level access.

Grayskull/Wormhole/... basically have a bunch of tensix cores, which each have 5 small rv32 cores, where some are used for data movement and some for computation. IIRC two (could be one) drive a SIMT compute unit.

From what I can tell there is no "matrix in matrix out" style fixed accelerator, the SIMT unit looks decently flexible, the grayskull one is a bit limited isa wise, wormhole is supposed to improve on that.

tt-buda is supposed to be the high level API with pytorch, tensorflow, ... support, but idk how much compatibility there is at the moment. I assume there is a lot of software work left.

qwertox3• 2 years ago

Wormhole is an accelerated NIC, like what Nvidia got access to when it bought Mellanox. It's meant for high-speed routing.

Yet you got me to reading the following:

t-Series Workstations
Our t-series workstations are turnkey solutions for running training and inference on our processors, from a single-user desktop workstation in the t1000 up to the t7000 rack-mounted system designed specifically to function as a host with our Galaxy 4U Server
8 Grayskull Cards
16 Grayskull Chips

So these cards can also do training.

https://tenstorrent.com/systems/aicloud/

mnky9800n• 2 years ago

What does it mean by "inference" in this context?

syntaxing• 2 years ago

You’re trying to get an output from giving an input to a bunch of matrices (tensors if you will). You do not touch anything in the matrices themselves which happens during training. Essentially, they’re great from deploying models but not making one.

hangsi• 2 years ago

Neural networks have two different compute costs: training and inference.

These are roughly analogous to compile time vs runtime for compiled programming languages.

Training is in general a more intensive task. However, in an ideal scenario training is run once and inference is run millions of times, so the lifetime cost of inference is bigger - this is why it might make sense to optimize for intense.

thoi324o23434• 2 years ago

Isn't RoCm just AMD's version of "CUDA" (whatever that means at this point) ? IIRC, it was tied to HIP which was AMDs IR compiler from ptx.

HarHarVeryFunny2• 2 years ago

AMD vs NVIDIA

RoCm = CUDA+cuBLAS+cuRAND

MiOpen = cuDNN

HIP is an AMD portability abstraction layer over the NVIDIA APIs that is pass-thru to cuDNN/cuBLAS/cuRAND on NVIDIA hardware, and pass-thru to AMD's equivalent APIs (that are basically drop-in replacements for NVIDIA's) on AMD hardware.

zokier2• 2 years ago

AMD doesn't have any equivalent of CUDA, i.e. something you can run on commodity GPUs. ROCm is only for specific "pro" devices.

hackerlight2• 2 years ago

ROCm officialy supports 7900 series retail cards and works fine on unsupported older AMD retail cards

atq2119• 2 years ago

I don't think this is a deliberate strategy but rather the result of focusing resources in where there are currently the largest margins, which is in the big server chips.

It is somewhat short-sighted though, since having support on all their GPUs would surely do wonders for adoption via more organic channels.

raggi• 2 years ago

Invest in webgpu compute. It's not just for web, but it is portable.

cjbgkagh3• 2 years ago

I write fast CUDA so I guess I'm one of the lucky few - which I already knew. I spent a lot of time benchmarking, instruction counting the PTX, function approximations, interleaving etc. I don't think it would cost that much money to write new ops for a new hardware for existing frameworks but for some reason very few do it - I figure it must be cheaper than making the hardware. I assume it's "penny wise, pound foolish" or maybe that it's very hard for hardware companies to do software as they try to pay their software guys like hardware guys.

More and more I'm doing more branchy algorithms so the CPU is now my new bottleneck - thank you fast GPUs - but I don't think the market will move to this way doing things before the current AI wave has crested. So it is probably not good business sense to target my use cases. I have quite a lot of concurrency so I think my ideal hardware is a whole lot of little CPU cores with decent cache and Half Matrix Multiply Accumulate (HMMA) instructions. Once my CPU bottleneck becomes too painful I'll look at what options are on the market. I think, but am not sure, the early Tesla AI chips focused too much on optimizing ResNet and then later they went with CPUs with the HMMAs. Instead of using a variety of instructions I can instead approximate most functions with more HMMAs.

EDIT: changed my reference of matmul intrinsics to the more precise HMMA instruction

blopp99• 2 years ago

"or maybe that it's very hard for hardware companies to do software as they try to pay their software guys like hardware guys."

Very interesting point.

abstractcontrol2• 2 years ago

I have quite a lot of concurrency so I think my ideal hardware is a whole lot of little CPU cores with decent cache and matmul intrinsics

Back in 2015 I thought this would be the dominant model in 2022. I thought that the AI startups challenging Nvidia would be about that. Instead, they all targetted inference instead of programmability. I thought that a Tenstorrent hardware would be about what you are talking about - lots of tiny cores, local memory, message passing between them, AI/matmult intrinsics.

I've been hyped about Tenstorrent for a long time, but now that it is finally coming out with something, I can see that the Grayskulls are very overpriced. And if you look at the docs for their low-level kernel programming, you will see that Tensix cores can only have four registers, have no register spilling, and also don't support function calls. What would one be able to program with that?

It would have been interesting had the Grayskull cards been released in 2018. But in 2024 I have no idea what the company wants to do with them. It's over five years behind what I was expecting.

My expectations for how the AI hardware wave would unfold were fit for another world entirely. If this is the best the challengers can do, the most we can hope for is that they depress Nvidia's margins somewhat so we can buy its cards cheaper in the future. As we go towards the Singularity, I've gone from expecting revolutionary new hardware from AI startups to hoping Nvidia can keep making GPUs faster and more programmable.

Ironically, that latter thing is one trend that I missed, and going from Maxwell cards to the last generation, the GPUs have gained a lot in terms of how general purpose they are. The range of domains they can be used for is definitely going up as time goes on. I thought that AI chips would be necessary for this, and that GPUs would remain as toys, but it has been the other way around.

cjbgkagh3• 2 years ago

I wasn't as optimistic that there would be a broad adoption of some of the more advanced techniques I was working on so I did figure back in 2013 that most people would stick to the GEMMs and Convs with rather simple loss functions - I had a hard enough time explaining BPR triplet loss to people. Now with LLMs people will be doubling down on GEMMs for the foreseeable future.

My customers won't touch non-commodity hardware as they see it as a potential vector for vendors to screw them over, and they're not wrong about that. In a post apocalyptic they could just pull a graphics card out of a gaming computer to get things working again which gives them a strong feeling of security. Having very capable GPU cards as a commodity means I can re-use the same ops for my training and inference which roughly halves my workload.

My approach to hardware companies is that I'll believe it when I see it, I'll wait until something is publically available that I can buy off the shelf before looking too closely at it's architecture. NVidia with their Tensor Cores got so good so quickly that I never really looked too closely at alternatives. I'm kind of hopeful that AMD SoC would provide a good edge compute option so I might give that a go.

I had a look at tenstorrent given this article and the Grendel architecture seems interesting.

imtringued4• 2 years ago

Grayskull shipped in 2020 and each tensix cores has five RISC-V cores. Get your basic facts right before you complain. The dev kit is just that, a dev kit. Groq sells their dev kit for $20k even though a single LPU is useless.

danielmarkbruce• 2 years ago

It's a swamp and a moat. Lots of moats are swamps. Look at a lot of large enterprise software products. Awful to work with, sticky as hell, swamp as it gets. And very much a moat.

parentheses• 2 years ago

JK is always going to be critical of any winners in HPC chips. They are competitors of his.

StressedDev3• 2 years ago

I think we could summarize this article as Jim Keller calls CUDA and x85 names. Name calling is not an argument and whenever I see someone resort to name calling, it tells me they probably don't have better arguments.

In engineering and software development, we have to accept trade-offs and imperfection. Every decision has pluses and minuses. Instead of name calling, Jim Keller should explain what his approach is and why he thinks it is an improvement. He should also explain its limitations and weaknesses. Finally, he should also explain how he will handle the inevitable compromises and imperfections which arise in successful systems.

Jim Keller criticizes Nvidia's CUDA, x86

Related Stories