Apple collaborates with Nvidia to research faster LLM performance

https://9to5mac.com/2024/12/18/apple-collaborates-with-nvidia-to-research-faster-llm-performance/

by hochmartinez • 2 years ago

56 35 2 years ago

9to5mac.com

gibsonf1• 2 years ago

Hmm, is faster hallucination better than slower hallucination?

henry2023• 2 years ago

Maybe if you hallucinate fast enough no one will notice?

dhosek• 2 years ago

I was initially reading it as LLVM performance and thought, “how cool” and then I realized the V was missing and stopped caring.

ben_w2• 2 years ago

When the answers can be checked by another program and best-of-n results get better with increasing n, yes.

I think TDD code development is this, at least in principle.

weebull2• 2 years ago

If you can write a program to check results you've already solved the issue.

SiddanthEmani• 2 years ago

With competition on the rise, it’ll be a race to provide faster inference speeds for general queries

DannyBee• 2 years ago

Is there somebody not researching faster llm performance?

stefan_• 2 years ago

Does Intel qualify? They seem busy making GPUs for midrange insignificant gaming builds, while all their AI offerings are hopeless.

deergomoo• 2 years ago

As someone who owns an insignificant midrange gaming build and has been irritated at the inflation of GPU prices with crypto and now genAI, I’m glad someone is.

init2null• 2 years ago

Not everyone is willing to pay for 4K, 120 Hz, and quality raytracing. There is a market for less than premium experiences. I wanted some marginal CUDA for Blender, so I got a 3060. This experience is anything but insignificant to me.

rowanG0772• 2 years ago

How is it insignificant. Intel GPUs are the most exciting thing that has happened in GPUs the last 5-10 years imo.

n144q3• 2 years ago

Virtually nobody was buying Intel GPUs:

https://www.tomshardware.com/pc-components/gpus/discrete-gpu...

Looks like their latest GPU is selling out. Hope they actually move the market a little bit and iterate on this.

yjftsjthsd-h• 2 years ago

They're still trying. I mean, not succeeding, but they're making some sort of effort. (On which note... If anyone's figured out how to run ollama/llamafile/... on any of the apparently numerous libraries that supposedly accelerate AI on Intel hardware please share any hints because supposedly I've got the hardware but I can't make sense of any of the docs...)

CharlesW2• 2 years ago

Is there somebody not researching faster llm performance?

The surprising bit is less about what they're working on and more about the collaboration itself, including the mutual and coordinated co-marketing/PR. Apple and Nvidia haven't had a business relationship in over a decade, ever since Apple stopped using Nvidia GPUs in Macs.

This sort of re-engagement and subsequent promotion isn't something that "just happens", in my experience. It's reasonably possible that this could portend additional future outcomes, such as official support for Nvidia-based eGPUs, Apple's licensing of/support for CUDA, Mac Pros with Nvidia GPUs, etc.

ksec• 2 years ago

That is exactly what I was going to say. Nvidia has been a big No No inside Apple after they strained there relationship with MacBook Pro recall with Nvidia GPU causing overheating. Apple's heat dissipation were never good in the first place which makes the matter worst.

So this isn't just about Faster LLM performance. This could possibly opens up a whole new world for Mac ecosystem with CUDA.

SllX• 2 years ago

Probably not, but Apple and NVIDIA collaborating on anything is news. They’re two companies that both want to be the ones wearing the Big Boy pants in any relationship so when they did anything together in the past, it hasn’t worked out for long.

weebull2• 2 years ago

I'm not.

UncleOxidant• 2 years ago

But Apple seems to have gone all in on their own GPU/NPUs, why would they collaborate with Nvidia on this?

comboy• 2 years ago

patents?

CharlesW2• 2 years ago

Personal speculation here: https://news.ycombinator.com/item?id=42456965

threeseed3• 2 years ago

Apple inference is on their own GPU. Apple training is on the cloud with Nvidia.

There are plenty of use cases e.g. Apple Maps, News etc which are public and don’t need their Private Compute Cloud.

username4567• 2 years ago

Complete speculation but could be that Apple supports these operators on their GPUs and they need end-user models optimized to use them. Apple makes their money on consumer devices and need them to be performant for inference on the edge. Getting NVIDIA to integrate the operators may result in more models optimized for operators Apple's devices can leverage to stay ahead of other edge GPUs.

brotchie• 2 years ago

Convinced that Apple has shot themselves in the foot / tied their hands behind their back / insert analogy here, re: privacy <> AI.

Interesting to see how it plays out. Meta and Google have much more permissive privacy policies / stances, which means Meta + Google models are going to get much better faster.

Apple does potentially have an edge with their Mx series of chips re: inference flops. I bet they're hoping that the model quality vs. model size curve continues dropping such that they can run sufficiently powerful LLMs on-device.

We'll see.

cma2• 2 years ago

Apple has a huge legacy burden of defaulting laptops to 2010 levels of memory capacity at 8GB all the way up to mid 2024. All the great AI inference stuff in the M-series is wasted due to that until you get ~80% penetration of the new 16GB min spec.

If they get compelling local inference throughout the OS it may help them drive adoption of new models faster, but App devs won't have incentive to integrate local AI stuff until there is more market penetration of the higher RAM capacities. Apple could perhaps monetarily incentivise them to make up for it. But I think they are just going to heavily have to lean on cloud for everything.

threeseed3• 2 years ago

Apple Intelligence is supported on any M1 Mac: https://www.apple.com/apple-intelligence

And not sure there are many use cases for running mid-sized models locally. Smaller models fit in 8GB and work fine for action handling, tagging, classification etc. And for text/image generation you just get such a better user experience using a cloud hosted model.

threeseed3• 2 years ago

a) Apple already reaches out to ChatGPT (and soon to be others) for the full LLM experience. And there isn’t much benefit to be had from building their own competitor.

b) Privacy is essential if you want to an LLM to interact with you with private information. You can’t just upload unencrypted photos, messages, health data etc to the cloud where it will be accessible by any government with a search warrant. And so it does makes sense to keep it all private and encrypted on device or used in private but restricted compute.

c) Hype is very much coming out of the AI space and so I don’t see the market punishing Apple for not doing more. If anything they should’ve done nothing rather than released the technically impressive but largely useless Apple Intelligence product.

treprinum• 2 years ago

I have an idea - how about Apple letting Mac users use Nvidia cards/chips again?

keyle• 2 years ago

I have an external GPU box that is just there, catching dust... On the plus side I haven't heard a fan for 2 years.

kentiko• 2 years ago

Sell it? If you keep it until it’s worth nothing, it is equivalent to throwing it away.

n144q3• 2 years ago

I always thought they haven't talked to each other for years.

Apple collaborates with Nvidia to research faster LLM performance

Related Stories