Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

https://www.cnbc.com/2024/03/18/nvidia-announces-gb200-blackwell-ai-chip-launching-later-this-year.html

by tiahura • 2 years ago

393 314 2 years ago

paulpauper• 2 years ago

Stock unchanged in afterhours. A lot of people were hoping for a big pop on some big development.

synergy20• 2 years ago

not only that, it lost steam during the day, maybe it was overheated too much and no more news can pump it up any further.

dagmx7• 2 years ago

I imagine it’ll pop in the morning

dxbydt• 2 years ago

guy is messing it up bigtime and in real-time as well. sheesh. none of his jokes are landing. “we had 2 customers. we have more now”. long pause. screen behind him covered with logos of all his customers. pause. pause. finally applause. ok on to the next tidbit.

whole conference has been proceeding like this now. look if you invite cramer and the wall street crowd, you should throw in some dollar figures. like - who is paying for all this. how much. and why. talk is entirely about token generation bandwidth, exaflops and petaflops, data parallel vs tensor parallel vs pipeline parallel - do you honestly think cramer knows the difference between an ml pipeline and an oil pipeline ?

i am watching this conf with my kid - proper GenZ member - who got up after 5 mins and said man who is this comedian, his jokes are so bad, and left :(

gmerc• 2 years ago

Nah, wallstreet doesn’t understand what it’s looking at.

That’s fine, it’s a developer conference for a founder lead company that hasn’t reached the “stock price is the product” state. He’s not trying to optimize the next 5 days of stock.

There’s a full ecosystem grab with Nim there, a new GPU that forces every major datacenter to adopt (or their competitors will massively increase their compute density)

anon2915• 2 years ago

This is a developer's conference, not a financial one.

Takennickname2• 2 years ago

Cramer is an entertainer. Not a developer or an investor.

smallmancontrov3• 2 years ago

You might not like it, but this is what peak performance looks like.

Deasiomlo2• 2 years ago

Why would a Gen z kid care about this conference?

It's not a hipster event.

And the pauses are clearly his style of presenting. Having a artificial pause to tell the audience it would be a good time to react.

You clearly don't like it, I don't mind it.

But just to be clear: we see peak human ingenuity. This right now will be history on how we as humans build AGI, full human robots etc (in case we don't nuke ourselves).

You can read the conference results easily at any it news portal.

There is no requirement from Nvidia entertaining you or your kid.

TheAlchemist6• 2 years ago

Well, stock price is not a good short term indicator about Nvidia developments, nor any company for that matter. Nvidia is doing a very good job.

That being said, their stock is absolutely and hilariously overvalued.

costcofries• 2 years ago

Tell me more about why you believe their stock is hilariously overvalued.

Blammar• 2 years ago

NVDA's forward PE is ~37, about what it has been for the past ~5 years I've been tracking that. So it's not overpriced based on that metric.

If you're convinced the stock is that overvalued, go short some or, if you like to live dangerously, buy some long-term put options (don't be an idiot and buy short-term options.)

I have no idea if NVDA is like Cisco Systems in 2000, or if it's something unique. What I am aware of is that there's around 5-7 trillion that were moved from stocks to t-bills since the Fed raised rates in March 2022. If and when they drop their rates back to the historical ~2.5%, it's reasonable to predict these funds will go back into stocks, which will presumably drive up prices.

rvz• 2 years ago

A lot of people were hoping for a big pop on some big development

They are waiting for earnings projections for such a pop since right now it is extremely overbought and struggling to move past >$1,000 per share.

For now, Microsoft and OpenAI will use these chips, but in the long term they are just looking at this and plotting to build their own chips and reducing their dependence on Nvidia and will be ready to switch once their contracts have run out.

swalsh3• 2 years ago

At 2 trillion, it's all baked in already

_factor3• 2 years ago

Nvidia is no secret. Whatever hidden value is in the stock, is likely already represented.

dagmx7• 2 years ago

FP8 being 2.5x Hopper is kind of disappointing after such a long time. Since its 2 fused chips, that means it’s 25% effective delta.

though it seems most of the progress has been on memory throughput and power use which is still very impressive.

I wonder how this will trickle down to the consumer segment.

azeirah3• 2 years ago

Jensen revealed later that the LLM inference is 30x due to architectural improvements, it's massive. I don't know if it's latency or just 2-3x performance boost with 30x more customers served in the same chip. Either way, 30x is massive.

ephemeral-life• 2 years ago

30x is the type of number that when you see it in a generational improvement, you should ignore it as marketing fluff.

my123• 2 years ago

The 30x number is for a really narrow scenario tbh. Running a GPT 1.8T parameters (w/ MOE) on one GB200

dagmx7• 2 years ago

Yeah and the 30x is largely due to the increase in factors like packaging and throughput. It's not indicative of general purpose performance which is what I was talking about.

Again, I do think the throughput and energy efficiency gains are impressive, but the raw performance gain is lower than I'd have expected for the massive leap in node size etc

modeless3• 2 years ago

He always does that. They stack up a bunch of special case features like sparsity that most people don't use in practice to get these unrealistic numbers. It'll be faster, certainly, but 30x will only be achievable in very special cases I'm sure.

qwertox2• 2 years ago

But Blackwell in the graph is FP4 whereas Hopper is FP8.

kkielhofner• 2 years ago

The other big announcement here is NIM - Nvidia Inference Microservice.

It's basically TensorRT-LLM + Triton Inference Server + pre-build of models to TensorRT-LLM engines + packaging + what appears to be an OpenAI compatible API router in front of all of it + other "enterprise" management and deployment tools.

This software stack is extremely performant and very flexible, I've noted here before it's what many large-scale hosted inference providers are already using (Amazon, Cloudflare, Mistral, etc).

From the article:

'Nvidia will work with AI companies like Microsoft or Hugging Face to ensure their AI models are tuned to run on all compatible Nvidia chips. Then, using a NIM, developers can efficiently run the model on their own servers or cloud-based Nvidia servers without a lengthy configuration process.

“In my code, where I was calling into OpenAI, I will replace one line of code to point it to this NIM that I got from Nvidia instead,” Das said.'

The dead giveaway is "I changed one line of code in my OpenAI code" which means "I pointed the OpenAI API base URL to an OpenAI compatible API proxy that likely interfaces with Triton on the backend via its gRPC protocol".

I have a lot of experience with TensorRT-LLM + Triton and have been working on a highly performant rust-based open source project for the OpenAI compatible API and routing portion[0].

On this hardware (FP4) with this software package 30x compared to other solutions (who knows what - base transformers?) on Hopper seems possible. TensorRT-LLM and Triton can already do FP8 on Hopper and as noted the performance is impressive.

[0] https://github.com/toverainc/ai-router

jimmySixDOF• 2 years ago

This is also the only place Nvidia are getting competitive pressure - from the likes of Groq (and likely but less published from Cerebras) with higher inferance T/s and concurrency utilization/batching [1] so if this proves to be the true then the case for big chip systems (on todays specs) will be harder.

[1]https://twitter.com/swyx/status/1760065636410274162?t=rpbcr8...

YetAnotherNick2• 2 years ago

How is 2.5x disappointing in one generation?

chimney• 2 years ago

Compare to the 10x that was Hopper uplift.

dagmx7• 2 years ago

Did you skip the sentence immediately after that one?

It’s two fused chips. So 1.25x per chip. 25% uplift. Not 2.5x uplift. The 2.5x is for the whole package.

jairuhme• 2 years ago

I haven't listened to Jensen speak before, but am I the only one who thought the presentation wasn't very polished? Not a knock on anything he has accomplished, just an observation that sorta surprised me

acchow4• 2 years ago

He has more important things to do than perfecting a presentation. He likes his employees to message him freely with things they think he can help with.

PartiallyTyped• 2 years ago

I read about this around and tbh I respect it a lot. Somehow Jensen has managed to scale the company and still keep it focused on the product with high eng practices.

jefozabuss• 2 years ago

When he had that slide up with generating everything I was kind of expecting that he'd say this whole keynote is generated including him. That'd have been crazy.

azeirah3• 2 years ago

He said he didn't rehearse well. I think it makes him come across very genuinely, not some dumb hyperpolished corporate blabla

CaptainFever• 2 years ago

In comparison to Apple keynotes, precise, polished, practiced and pre-recorded.

ipsum2• 2 years ago

Yeah he said he didn't rehearse and it really shows.

angm128• 2 years ago

The products, animations and slides are doing some heavy lifting. Most jokes don't land and his presentation is somewhat confusing at times (e.g. star trek intro token count)

swalsh3• 2 years ago

He's selling water in a desert, kind of doesn't matter how polished his presentation is.

cableshaft• 2 years ago

The beginning in particular seemed pretty rough, but he seemed to mostly get into a groove about halfway into it. At least he started talking a lot smoother around then.

sct202• 2 years ago

I think it's a good reminder that objectively great CEO's and leaders can be kind of cringe when presenting. A lot of times people like that get passed up in promotions in favor of smooth talkers.

wmf10• 2 years ago

It's been said that founders are people who can't get hired.

modeless3• 2 years ago

It's typical. He isn't a great public speaker IMO. Not terrible but not great.

erupt7893• 2 years ago

I've been watching his keynotes for as long as I can remember, this is how it's always been

caycep2• 2 years ago

I remember his opening line at NEURIPS 2017, to an audience of grad students and postdocs, "only Nvidia would unveil their most expensive product to an audience who's completely broke"

Then he went into a comedic monologue about GANS. But hey, at least that meant that the CEO was reading the actual conference proceedings...

Havoc2• 2 years ago

Previous ones have been similar. Awkward “you may clap now” pauses etc.

Don’t think anyone cares as long as the company keeps getting the bets right

CoachRufus87• 2 years ago

Its refreshing. Another tech keynote that I regularly watch (Apple) is far too polished these days.

sumedh• 2 years ago

If I am not mistaken he has said in some interviews he is not a big fan of public speaking.

bluedino• 2 years ago

They acquired Bright Cluster Manager a few years ago, who would be next on their list to acquire? It seems like they want to provide customers with the whole stack.

shiftpgdn• 2 years ago

Canonical is a ripe target. Canonical has been trying to grow Ubuntu and other tools in the enterprise world for the last few years without significant success, and much of the Nvidia devkit stuff is built around Ubuntu.

echelon• 2 years ago

Please do not give them this idea.

Ubuntu is actually a pretty great daily driver desktop Linux, and I'd hate for that to lose priority and disappear.

I'm not a fan of what happened to the Red Hat ecosystem for exactly the same reasons.

hipadev23• 2 years ago

Canonical’s culture [1] is the antithesis of what nvidia wants.

[1] https://www.reddit.com/r/recruitinghell/comments/1bec2zk/lit...

pjmlp2• 2 years ago

I would rather bet Microsoft doing that, given their cozy relationship for .NET and main WSL distribution.

At least I would finally get to buy MS Ubuntu PCs at the shopping mall.

kflansburg• 2 years ago

Run:AI https://news.ycombinator.com/item?id=39738342

az226• 2 years ago

Anthropic or Mistral and build AGI/ASI.

stevethomas• 2 years ago

Time to sell. When they start becoming a platform, it means they have nothing more concrete in the near future. Sell now and buy again later once the price corrects.

belter4• 2 years ago

Don't bet against a CEO who knows what is talking about, has 80% market share and an arm tattoo of his own company logo.... :-)

So far the short sellers, learned that bitter lesson.

golergka• 2 years ago

Does a company like Nvidia has to have anything more concrete than newer, bigger and faster chips?

pvg• 2 years ago

If you held and then sold Nvidia stock when they announced CUDA or GeForce Live, you'd now be now a big pile of negative money richer.

Keyframe• 2 years ago

They still have to announce you can send an email through their platform.

mark336• 2 years ago

I wouldn't be 100% against them, but it is disappointing that they don't have any better ideas.

hsuz• 2 years ago

Selling is a bit exaggerated. Scaling up is almost always non-detrimental. But I do feel that NVDA is slowly falling to a stall because there are no exciting new modes of business coming out.

lvl102• 2 years ago

Seems Nvidia is going for maximum margin as they see competition ahead.

theGnuMe• 2 years ago

And they can build a big moat with cuda.

qwertox2• 2 years ago

What is FP4, 4 bit floating point? If so, the comparison graph [0] with 30x above Hopper was a bit misleading.

[0] https://youtu.be/Y2F8yisiS6E?t=4698

sipjca• 2 years ago

yes

fancyfredbot• 2 years ago

That's right.There was mention of a precision aware transformer engine which might make it easier to use fp4, but it's not 30x faster in a like for like way. This shouldn't be surprising since it's more or less two hoppers next to one another on a slightly improved process node. 2.5x seems more likely in cases where you don't exploit a new feature like that or the increased memory.

Havoc2• 2 years ago

bit misleading.

Only partially, because in LLMs FP4 isn't half as useful as FP8. So if you have gear that crushes at FP4 then that's what you use and you benefit from that increased speed (at minimal accuracy loss).

Definitely some marketing creativity in there, but its not entirely wrong as a measure of real world usage

jxy• 2 years ago

curiously, what real world usage actually uses FP4? AFICT, most of the LLMs still use BF16, and even the quantizations down to 4bits and 2bits end up back to 16bit or INT8 for actual computations.

s_m_t3• 2 years ago

How can 4 bits possibly be enough? Are intermediate calculations done at a higher width and then converted down back to FP4?

wongarsu2• 2 years ago

For training FP4 sounds pretty niche, but for inference it might be very useful.

WhitneyLand• 2 years ago

- Training isn’t done at 4-bits, to date this small size has only been for inference.

- Research for a while now has been finding that smaller weights are surprisingly effective. It’s kind of a counterintuitive result, but one way to think about it is there are billions of weights working together. So taken as a whole you still have a large amount of information.

CamperBob22• 2 years ago

The various sigmoid activation functions have the effect of keeping bit growth under control, by virtue of clamping to the +/- 1 range.

yalok• 2 years ago

There are research papers where even 1 bit (not floating point) was enough, with some quality loss.

4 bits is effectively 16 different float point numbers - 8 positive, 8 negative, no zero and no NaN/inf. 1 bit for sign and 3 bits for exponent, 0 bits for mantissa, mantissa is implied to be 4. It’s logarithmic - representing numbers in the range from -4^3 to 4^3, smallest numbers are 4^-3.

anon2915• 2 years ago

The fundamental 'unit' of NN computation is not an individual vector element but rather an entire vector. One of the first results you often learn about in linear algebra is that some axes are more important than others (principal components, singular value decomposition). Thus, it totally stands to reason that the underlying field of the vector is inconsequential but rather the entire vector machinery. All you have to do is make sure that there are enough elements in the vector to get the job done for whatever bit size of element.

wongarsu2• 2 years ago

It's 4 bit floating point, at twice the speed of 8 bit floating point. There's also FP6, doesn't offer faster compute than FP8 but manages to take advantage of the better memory bandwith and cache use of the 6 bit format.

Apparently some people are drawing connections to this paper [1] on 4 bit LLMs, which has one NVIDIA employee among its contributors

1: https://arxiv.org/pdf/2310.16836.pdf

opcode84• 2 years ago

https://arxiv.org/pdf/2310.10537.pdf

Discussed in a previous post https://news.ycombinator.com/item?id=37930663

buildbot2• 2 years ago

Yes this is it!

tamimio• 2 years ago

I think at this point, they should stop making it video “cards” but rather video “stations”, a full tower station with power supply and one giant “card” inside with proper cooling, etc., might also justify the crazy prices anyway.

ufocia• 2 years ago

Probably better to stick to the GPUs. Integration is a low margin game.

caycep2• 2 years ago

granted, at this point we plug the computer into the GPU, so it might not make a difference...

georgyo• 2 years ago

I'd prefer they stick to GPUs, but I think you're over simplifying.

Dell proves that selling complete units is very profitable.

Apple shows that owning the entire stack is immensely profitable.

Nvidia already has significant hardware and software investment. They very well could fully integrate and grab larger slices of the pie.

In fact, Nvidia already has complete appliance like fully integrated machines. But enterprises like to install their own OS and run their own software stack. These appliances have not caught on, at least not yet.

jedberg• 2 years ago

https://lambdalabs.com/gpu-workstations/vector

justinclift• 2 years ago

Heh Heh Heh

"Ultimate AI Workstation". Pricing starts at US$83,549:

https://shop.lambdalabs.com/gpu-workstations/vector/customiz...

Adding every option only adds $2,100 to the price too (totalling $85,649). They should probably just include everything as standard. ;)

jazzyjackson2• 2 years ago

I love that "we installed a working python environment for you" is a front-page value-add

ribosometronome• 2 years ago

Isn't that just a computer? Or an eGPU, if it doesn't contain the rest of the computer?

wmf10• 2 years ago

SXM for desktop would be great but it won't happen. The PC industry can't even adopt things like 12VO.

acchow4• 2 years ago

That’s the DGX GB200 they announced today, with liquid cooling.

herecomethefuzz• 2 years ago

"Platform company" means multi-chip in this case?

Seems logical since it's becoming impractical to cram so many transistors on a single die.

1oooqooq2• 2 years ago

no it means rent seeking.

imagine aws if they also sold all computers in the world, now you can only rent from them

throwaway114602• 2 years ago

So like IBM at the beginning of computers

maximus-decimus• 2 years ago

"For only 100$ a month, you'll be able to turn on the gpu you already paid for"

--Nvidia, pretty soon

0xcde4c3db• 2 years ago

I don't really understand the bird's-eye view of the product line, but judging by some of the raw physical numbers and configurations Jensen was bragging about, it means that they want to basically play the mainframe game of locking high-end applications into proprietary middleware running on proprietary chassis with proprietary cluster interconnect (hello, Mellanox acquisiton).

wtallis• 2 years ago

The lock-in is more of a bonus for them. The underlying problem is that it's impossible to build a chip big enough, or even a collection of chiplets big enough. Training LLMs requires more silicon than can fit on one PCB, so they need an interconnect that is as fast as possible. With interconnect bandwidth as a critical bottleneck, they're not going to wait around for the industry to standardize on a suitable interconnect when they can build what they need to be ready to ship alongside the chips they need to connect.

anon2915• 2 years ago

In this case the interconnects are also doing compute.

dweekly• 2 years ago

It means all the main chips required for a large-scale datacenter. And many of the layers of software on top of it.

Hardware: * The GPU * The GPU-GPU Fabric (NVLINK) * The CPU * The NIC * The Network Fabric (infiniband) * The Switch

And that's not even starting to get into the many layers of the software stack (CUDA, Riva, Megatron, Omniverse) that they're contributing and working to get folks to build on.

amelius• 2 years ago

"platform co."

Platform company, as in they're allowing developers on their AI platform and opening an app store?

fnordpiglet• 2 years ago

The headline is editorialized by the submitter, the actual headline is “Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’” which is arguably worse.

The article doesn’t discuss becoming a platform co but instead discussed ways their existing platform subscription model is evolving to add backwards compatibility testing.

tzm• 2 years ago

Platform co seems fitting, considering Nvidia's data center revenue in the fourth quarter of 2023 was a record $18.4 billion, which is 27% higher than the previous quarter and 409% higher than the previous year.

Seems revenue from inference is growing at a significant clip.

ec109685• 2 years ago

Data center revenue includes sales to companies like Meta that run their chips in their own data centers.

Deasiomlo2• 2 years ago

Double digit peta flop mass produced.

"The computing power needed to replicate the human brain’s relevant activities has been estimated by various authors, with answers ranging from 10^12 to 10^28 FLOPS."

Petaflop is 10^15

Crazy times.

teaearlgraycold• 2 years ago

I’ll be happy with this if we use it to design viable fusion power plants. And I’ll be severely disappointed if it’s mostly used for ad targeting.

steelframe• 2 years ago

I’ll be severely disappointed if it’s mostly used for ad targeting

Obligatory Rick and Morty: https://www.youtube.com/watch?v=xerLPWdyX-M

sbstp• 2 years ago

You are about to be severely disappointed.

belter4• 2 years ago

Fusion plants will be used to power ad targeting Nanoprobes.

geor9e• 2 years ago

Like a lot of the commenters here, I have a problem with this headline. They don't "seek to become a platform company", as in making their own cloud platform where they rent GPU time, meaning they stop selling GPUs to other cloud platforms. That easy misinterpretation makes good clickbait, but no, that's not what the article says - the article has Huang bragging that CUDA already is a parallel computing platform, for a decade or more, and Blackwell Architecture is so integrated and customizable with CUDA (with all its user-extendable kernels and community) that it's thought of as a platform rather than just chip architecture.

baobabKoodaa• 2 years ago

Someone please "chip in" and confirm or deny if this interpretation is correct?

wmf10• 2 years ago

"Platform" can have several different meanings and some people in this thread are picking the most evil meaning as an excuse to shit on Nvidia. It's true that Nvidia is providing the GPU, the CPU, the server, the rack, the network, the drivers, and the orchestration software to run AI training/inference. (If you want that stuff. You could just buy GPUs if you want.) It's fair to call that a platform.

Nvidia is not becoming a cloud provider (beyond a small eval environment perhaps).

basiccalendar74• 2 years ago

they announced a cloud service NIM that hosts LLMs using Nvidia software stack.

https://developer.nvidia.com/blog/nvidia-nim-offers-optimize...

so there is a push towards a platform company, but probably not well explained in this article.

jakobov• 2 years ago

They are claiming a 25x reduction in power consumption. That can't be right. Anyone understand where this number is coming from?

LTL_FTC• 2 years ago

Did you read that in the linked article? I couldn’t find it. But maybe due to the better efficiency with regard to the performance boost (5x) and the ability to now use 27 trillion parameters versus 1.7 Trillion, one can presumably finish the same amount of work in 1/25th of the time and bam, reduction in power consumption. As you say, I’m skeptical the max power draw itself is 25x lower.

wmf10• 2 years ago

I think Jensen said something like needing 25x fewer GPUs (vs. A100) to get the same performance, which amounts to essentially the same thing.

cavisne• 2 years ago

Comes from here [1]. Basically 100 racks of H vs 8 racks of B.

I think there may be a typo though, I assume this also includes liquid-cooled vs air-cooled.

[1] https://nvdam.widen.net/s/xqt56dflgh/nvidia-blackwell-archit...

adamnemecek• 2 years ago

Which Blackwell is it named after?

wmf10• 2 years ago

https://en.wikipedia.org/wiki/David_Blackwell

cebert• 2 years ago

“Nvidia … is becoming less of a mercenary chip provider and more of a platform provider, like Microsoft or Apple, on which other companies can build software.

I can understand from a growth perspective why it’s more profitable for Nvidia if it can become more of a platform service for AI. However, that’s difficult to balance that and partnerships the company already has with AWS and Microsoft. I’d expect to see some acquisitions or competing custom solutions in the future. Fortunately for Nvidia, a lot of AI is still dependent on CUDA. I’m interested to see how this plays out.

dheera3• 2 years ago

My prediction is eventually there will be anti-trust ligitation, they will be required to open the CUDA standard, after which AMD will become a competitor.

NVIDIA could voluntarily open the standard to avoid this ligitation if they wanted to, though, and IMO it would be the smart thing to do, but almost every corporation in history has chosen the ligitation instead.

wmf10• 2 years ago

It would be kind of genius for Nvidia to "open" the CUDA APIs (which have already been unofficially reverse engineered anyway) but not the code. Maybe they'd also officially support HIP and SYCL. Maybe they could open SXM after all competitors have already committed to OAM. They'd create the appearance of opening up while giving up very little.

anon2915• 2 years ago

The cuda api is essentially open... Hip is basically a copy.

CUDA is such a misnomer. Amd doesn't have tensorRT, cuDNN, cutlass, etc. Forcing Nvidia to make these work on AMD is like forcing Microsoft to make windows work on apple hardware... Not going to happen.

nemothekid2• 2 years ago

My prediction is eventually there will be anti-trust ligitation, they will be required to open the CUDA standard, after which AMD will become a competitor.

If AMD isn't a competitor before government intervention, I don't the government forcing nvidia to open up CUDA changes much. CUDA's moat isn't due to some secret sauce - nvidia put in the developer hours; and if AMDs CUDA implementation is still broken, people will continue to buy nvidia.

There has been a lot of trying to get AMD to work - Hotz has been trying for a while now[1] and has been uncovering a ton of bugs in AMD drivers. To AMD's credit, they have been fixed, but it does give you a sense of how far behind they are in regards to their own software. Now imagine them trying to implement a competitor's spec?

[1] https://twitter.com/__tinygrad__/status/1765085827946942923

wmf10• 2 years ago

I think they're planning for a world where half their customers (hyperscalers) just use GPUs and CUDA while the other half (the long tail) use more profitable, higher-level parts of the platform. They don't have the leverage to force customers one way or the other. It would be easier to just sell GPUs, but they know that sophisticated customers can switch to other chips while the platform provides lock-in for smaller customers.

neverokay2• 2 years ago

Doesn’t matter how good your in-house tech team was, your company still outsourced it to cloud infra.

That’s what Nvidia faces. Doesn’t matter how good the current in-house teams are using direct hardware, the trend in corporate is shift to a vendor (Google/AWS).

Nvidia can watch this inevitable shift or get ready to offer itself as a platform too.

bayindirh5• 2 years ago

There are already platforms and projects which provide resources for long tail of science. EU is supporting projects which brings together resource providers with researchers and private companies in some cases.

NVIDIA is not entering an empty marketplace here. Also there's already enough know-how to make things run on cloud, OpenStack, HPC, etc.

Unless they make things very difficult for independent platforms, they can't force their way in that much, from my perspective.

danpalmer• 2 years ago

The hyperscalers aren't going to be on CUDA for long, Nvidia are taking too much of a cut. Google run tensor chips, Amazon and Microsoft both have their own accelerators in the works. They are massively incentivised to do so right now.

Now they do all re-sell Nvidia GPUs in their cloud businesses, but there the cost will be passed very directly on to infrastructure customers, who will see the competing higher level services from those cloud providers (hosted models) likely at lower or competitive prices, and it's going to be harder to justify renting CUDA cores for custom software.

selvan• 2 years ago

Microsoft and AWS would have a partnership with AMD/Intel for their GPUs, if those are capable and widely used as Nvidia's.

Microsoft has partnetship with OpenAI and also with Mistral.

Present convenience may not hold true in future. Nvidia knows that well.

DanielHB• 2 years ago

AWS is pushing ARM hard, yet people still buy x86/x64 compute en mass

Even if AWS has its own hardware+software solution for neural networks it would still take years if not decades to tear off the CUDA platform

wmf10• 2 years ago

More technical details: https://news.ycombinator.com/item?id=39752500

__mharrison__3• 2 years ago

My take from being at the keynote and the content I've seen so far at the conference is that Nvidia's is moving up the stack (like all good hardware vendors are prone to do).

Obviously they are going to keep doing bigger. But the takeaway for me is that they are building "docker for llms" - NIM. They are building a container system where you can download/buy(?) NIMs and easily deploy them on their hardware. Going to be fun to watch what this does to all the AI startups...

djtango• 2 years ago

I'm not that abreast of all the developments in the AI space.

What specific class of AI startups do you have in mind here? AI-aaS startups who provide the "infra"?

fisf• 2 years ago

AI startups who just wrap a (standard) API or model in a thin UI layer. The backend part will be a commodity and the UI layer offers no value proposition.

flessner• 2 years ago

Won't do anything to most consumer facing AI, the UI & convenience is already a major selling point. A bigger threat is that the feature the business is built around makes it into mainline software... there is no demand for (paid) background removal anymore as every iPhone can do it nowadays.

Generally if whatever AI product you have can easily just be a feature in whatever application businesses already use, then you are running a business on borrowed time.

Kinrany• 2 years ago

Oh no, turns out the thing people really want from AIs is generality because everything else can be done in stupid software!

dig1• 2 years ago

here is no demand for (paid) background removal anymore as every iPhone can do it nowadays.

Proper background removal of even remotely complex content is still in demand, especially on a large scale. I doubt you'd use an iPhone to work on >100 of images per second.

jerska• 2 years ago

there is no demand for (paid) background removal anymore as every iPhone can do it nowadays.

PhotoRoom begs to disagree with this statement.

ixaxaar• 2 years ago

There are also open source equivalents btw - https://github.com/geniusrise

gammalost2• 2 years ago

I wonder when we as an industry will start to address the scaling issues in LLMs.It is obviously in Nvidias interest to keep pushing out bigger and better GPUs, but what is the collective interest?

It is already proven that good language models are possible given enough resources. The challenge now is to put these models in a solution which do not require unfathomable amounts of resources for the average use cases.

bayindirh5• 2 years ago

Wasteful software development is easy and keeps momentum for development. As long as growth is king, quick and dirty will always beat well optimized and smaller systems.

This is not a problem with AI only, but with every software we use. Only two groups try to optimize things and try to fit into smaller systems. Passionate programmers and people who is paid to do this (e.g.: phone manufacturers' software teams, etc.).

Difwif2• 2 years ago

Not sure it's fair to characterize modern LLMs as 'wasteful software development" or unoptimized. The implementations do quite an impressive level of optimization with what hardware is available. New theoretical methods w.r.t quantization represent most of our software optimization techniques and we're probably hitting the limit of that shortly with ternary or binary gates.

To your point enthusiasts and developers with strong financial motivation have fairly optimized code. AI definitely falls into the later group. These are not like your typical web app :)

gammalost2• 2 years ago

I don't think it is thanks to wasteful software development. The libraries used for LLMs do a lot to squeeze out the full potential of GPUs.

I think it is more of an information problem. How can we store enough information in weights so that it is possible to train models without a budget similar to OpenAI

bilsbie• 2 years ago

Does this support that 1-2 bit stuff we were hearing about a few weeks ago?

nojvek3• 2 years ago

The issue with bigger LLM or even MMMs is that the bigger they are the more they are cramming and regurgitating the training data, and that opens up to lawsuits.

Making NNs generalize the way humans do it is still a hard problem.

lewhoo2• 2 years ago

Is this indeed established ? Could you provide a link or three ?

nojvek3• 2 years ago

https://avidml.org/blog/llm-privacy-plaigarism/

https://moreisdifferent.blog/p/weird-properties-of-large-lan...

https://www.sarus.tech/post/llms-have-become-excellent-at-me...

nojvek3• 2 years ago

Poor Mayoshi Sun from Softbank. He held a $3.1B position in Nvidia in 2019. That would be 19x at $60B today. He was super bullish on AI and robotics. Just ahead of his time.

flakiness• 2 years ago

Nit: Masayoshi Son

guy-om• 2 years ago

No, we need less AI, less robots, more (alive) humans.

karakot• 2 years ago

we've got quite a lot of humans already.

Nvidia CEO Jensen Huang announces new AI chips: ‘We need bigger GPUs’

Related Stories