The race for "AI Supremacy" is over – at least for now

https://garymarcus.substack.com/p/the-race-for-ai-supremacy-is-over

by signa11 • 1 year ago

46 38 1 year ago

garymarcus.substack.com

aliasxneo• 1 year ago

Two years ago, they were on top of the world, having just introduced ChatGPT, and struck a big deal with Microsoft. Nobody else had a model close to GPT-4 level; media coverage of OpenAI was endless; customer adoption was swift. They could charge more or less what they wanted, with the whole world curious and no other provider. Sam Altman was almost universally adored. People imagined nearly infinite revenue and enormous profits.

Part of me likes to think of this as cosmic karma. I know it's been hounded on a lot lately, but the irony is too rich, especially with how hostile they've been to "Open AI."

SecretDreams3• 1 year ago

Sam Altman was almost universally adored.

Ya, I'm going to need some sources for this bad boy lol

returnInfinity• 1 year ago

Before the board drama, I guess?

daveguy3• 1 year ago

Come on now. That's a super clickbait title. The article actually says the race is at a dead heat and LLM training for the same competency is getting cheaper, not that it's "over".

ipsum2• 1 year ago

The author of the article is known to be consistently wrong about AI.

mastazi2• 1 year ago

After reading the article I understand "over" to mean "there is no moat so there is no winner" rather than "it's over because entity X won and its competitors can't catch up"

daveguy3• 1 year ago

I'd consider "over" to be someone won, or it's not relevant anymore. After reading the article, it's neither of those. The race is very much on, but now it's clear that it will be competitive.

nextworddev2• 1 year ago

He's basically a full time OpenAI hater, that's his thing.

Cheer21712• 1 year ago

The article actually says the race is at a dead heat and LLM training for the same competency is getting cheaper, not that it's "over".

Odd, that's exactly what I interpreted the title to mean. "Supremacy" means one side has a total and insurmountable lead and control. That's how lots of China hawks in the West talk about it, arguing that they must do everything to stop China from building their own ChatGPT. The article is a pretty good summary of why no one side will achieve supremacy. And if you have been following Gary Marcus, he's been arguing this for a while now.

SecretDreams3• 1 year ago

The only losers in this AI race are probably humanity and overvalued stock markets. Kind of hard to price in years of future value and then to see some unknown Chinese AIs catch the big boys with their pants down.

I really can't over emphasize the "humanity losing" point, though.

thot_experiment3• 1 year ago

I don't feel like humanity is losing when my 7 year old gpu becomes an effective research assistant, secretary and busywork coding doer for the cost of downloading 10gb of tensors.

SecretDreams3• 1 year ago

The brutal thing about humanity is that most of the time we don't spend much time at all thinking about how things today might impact things tomorrow. We also tend to think more about how things today can be help ourselves, without giving much thought on how it might negatively impact others.

Basically, we're selfish and shortsighted and so it's easy at times to have rose coloured glasses.

nis0s• 1 year ago

Some time ago I interviewed with an AI governance think tank, and one of the topics that came up for ensuring Western-allied AI supremacy was trade security of compute resources. I didn’t want to shoot down their approach on this to their faces because they had a lot of blogs on their site, with OpenAI co-authors, about the efficacy of this approach. But I was never sold on the idea that holding off compute resources would stop someone from researching into better AI, I thought it would just push them to make more efficient AI. Lo and behold.

phantomathkg• 1 year ago

It seems the author forgot the fact that Deepseek can do it because they took 10k, or 50k, Nvidia chips before the export ban happened?

nextworddev2• 1 year ago

almost quarter of Nvidia chips go to Singapore even still, most of which find their way into China..

tokioyoyo• 1 year ago

In grand scheme of things, I still think it doesn’t matter. “American AI labs are years ahead” illusion is being challenged right now. If it’s even 80% as good, and x30 cheaper, you’ll have a very simple choice when you’re making a product. And unlike EVs, this isn’t that easy to just import control.

BoorishBears• 1 year ago

The article starts off on a pretty interesting premise: "I predicted there'd be no major advancements, but also, here are 4 major advancements in the last 4 weeks, many of which are predicated on another massive advancement that came out last year" (a SOTA model leveraging test-time compute)

We're also not seeing any price wars yet: every drop in price has been predicated on models getting faster, and while we don't know the size of models behind the scene, we can pretty reliably infer that they haven't been dropping their margins over time (at least, not at scale)

Anthropic even raised prices on their smallest model vs 2023 right as strong competition emerged.

Also seems strange to make a big deal about there being no moats... then imply that Meta deserves to be called up by Congress for releasing Llama's weights.

Cheer21712• 1 year ago

But isn't deepseek et al able to do its thing because they fired off a ton of queries to the proprietary models and fine-tuned off the outputs?

yawnxyz• 1 year ago

yes, but I don't envy those ML engineers working 24/7 to keep up the rat race

deadbabe• 1 year ago

Although the race isn’t over, Deepseek R1 has made it clear: the huge amounts of money pouring into AI investment is entirely unnecessary.

throwup238• 1 year ago

If it only cost $5.5 million for DeepSeek to create R1, what’s stopping OpenAI from building on their open research to create something even better for $500 million?

Everyone keeps talking about diminishing returns in training and plateaus in reasoning ability but it seems to me that’s exactly what R1 demonstrates with a 100x reduction in training costs: there is a long way to go and loads of low hanging fruit.

elfbargpt• 1 year ago

I think that is indeed the plan as far as I can tell

metadat• 1 year ago

There's still many more ideas to test and fuel to burn before declaring any outcome in the AI wars.

For more substantial content, check out:

The impact of competition and DeepSeek on Nvidia

https://youtubetranscriptoptimizer.com/blog/05_the_short_cas...

https://news.ycombinator.com/item?id=42822162 (70 comments)

treyfitty• 1 year ago

Isn’t Deepseek’s advantage that they didn’t actually start from scratch and that embeddings & training was already supplied? To say that Deepseek is comparable in performance to OpenAI is like saying Kirkland brands is comparable to {insert non white labeled good here}- they’re created off the same inputs. To say that Deepseek is a threat to AI supremacy is hyperbolic. As long as OpenAI innovates at the rate that it’s been innovating, their value is undeniable. Sure, Deepseek may tick after OpenAI’s tock, but the premium is in that tock.

kokanee• 1 year ago

This take discounts the implications of the fact that the efficiency and cost of Deepseek's product are orders of magnitudes better, plus the project is open source. Those facts fundamentally shift the business conditions for LLMs.

tzs• 1 year ago

Question: Suppose you wanted to train an LLM using say 100 GPUs, but for some reason you could not obtain the GPUs. You could however obtain CPUs. Each CPU lacks the massive parallelism of each GPU, so might have a throughout of say 1/100th the TFLOPS of the GPU.

Could you use 10000 CPUs and get a similar level of performance that 100 GPUs would have given?

lostmsu• 1 year ago

Please, refer to the article discussed here: https://news.ycombinator.com/item?id=41808013

TL;DR; 9950x peak fp16 is 10 Tops, 4090 peak fp16 is 650 Tops, and H100 is about 2 Pops.

IAmGraydon• 1 year ago

Since we’re (as in the human race) going to go ahead and push for AGI with reckless abandon, the best thing that can happen is for it to be available to everyone, no moats. I can’t help but feel a bit of schadenfreude given that this was supposed to be the very principle that OpenAI was founded on and that they abandoned at the first whiff of money. Now, it looks like it’s going to happen whether they want it to or not.

cudgy3• 1 year ago

Has anyone validated that deepseek was actually trained using the processors they claim and under the given time constraint?

jerome-jh• 1 year ago

The US may ban export of GPUs to China, but a GPU is not enough to build a training rig. You need memories, and hundred of smaller components. China has the ability to produce every component which goes in a computer. Can the US do the same?

The race for "AI Supremacy" is over – at least for now

Related Stories