Apple's MLX adding CUDA support

https://github.com/ml-explore/mlx/pull/1983

by nsagent • 1 year ago

548 203 1 year ago

gsibble• 1 year ago

Awesome

nxobject• 1 year ago

If you're going "wait, no Apple platform has first-party CUDA support!", note that this set of patches also adds support for "Linux [platforms] with CUDA 12 and SM 7.0 (Volta) and up".

https://ml-explore.github.io/mlx/build/html/install.html

teaearlgraycold2• 1 year ago

I wonder if Jensen is scared. If this opens up the door to other implementations this could be a real threat to Nvidia. CUDA on AMD, CUDA on Intel, etc. Might we see actual competition?

_ea1k• 1 year ago

I think this is the other way around. It won't be cuda on anything except for nvidia.

However, this might make mlx into a much stronger competitor for Pytorch.

teaearlgraycold2• 1 year ago

Oh bummer. Almost got excited.

baby_souffle• 1 year ago

If you implement compatible apis, are you prohibited from calling it cuda?

mayli• 1 year ago

Yeah, nice to have MLX-opencl or MLX-amd-whatever

almostgotcaught2• 1 year ago

CUDA backend

backend

tekacs• 1 year ago

This instance is the other way around, but that's what this is – CUDA on AMD (or other platforms): https://docs.scale-lang.com/stable/

pjmlp10• 1 year ago

Why, everyone keeps trying to copy CUDA while failing to understand why many of us love it.

int_19h4• 1 year ago

Abstraction layers for GPU compute already exist; this is yet another one, so it doesn't change anything substantially. Most of the time code written using such layers ends up running on NVIDIA hardware in prod anyway, so if anything that is a net positive for the company - it means that more people can now develop for its hardware on their devices.

zdw• 1 year ago

How does this work when one of the key features of MLX is using a unified memory architecture? (see bullets on repo readme: https://github.com/ml-explore/mlx )

I would think that bringing that to all UMA APUs (of any vendor) would be interesting, but discreet GPU's definitely would need a different approach?

edit: reading the PR comments, it appears that CUDA supports a UMA API directly, and will transparently copy as needed.

freeone30003• 1 year ago

Eh yes but from my experience its lack of prefetch lends to significant memory stalls waiting for the copy. It might be suitable if your entire dataset fits in VRAM after doing a “manual prefetch” but it killed performance for my application (ML training) so hard that we actually got time to move to streaming loads.

nerdsniper• 1 year ago

Edit: I had the details of the Google v Oracle case wrong. SCOTUS found that re-implementing an API does not infringe copyright. I was remembering the first and second appellate rulings.

Also apparently this is not a re-implementation of CUDA.

skyde• 1 year ago

this is CUDA backend to MLX not MLX backend for CUDA!

liuliu• 1 year ago

You misunderstood and this is not re-implementing CUDA API.

MLX is a PyTorch-like framework.

Uehreka• 1 year ago

This is exactly the kind of thing I wouldn’t opine on until like, an actual lawyer weighs in after thoroughly researching it. There are just too many shades of meaning in this kind of case law for laymen to draw actionable conclusions directly from the opinions.

Though I imagine that if Apple is doing this themselves, they likely know what they’re doing, whatever it is.

MuffinFlavored• 1 year ago

Is this for Mac's with NVIDIA cards in them or Apple Metal/Apple Silicon speaking CUDA?... I can't really tell.

Edit: looks like it's "write once, use everywhere". Write MLX, run it on Linux CUDA, and Apple Silicon/Metal.

cowsandmilk• 1 year ago

Neither, it is for Linux computers with NVIDIA cards

MBCook4• 1 year ago

Seems you already found the answer.

I’ll note Apple hasn’t shipped an Nvidia card in a very very long time. Even on the Mac pros before Apple Silicon they only ever sold AMD cards.

My understanding from rumors is that they had a falling out over the problems with the dual GPU MacBook Pros and the quality of drivers.

I have no idea if sticking one in on the PCI bus let you use it for AI stuff though.

kmeisthax• 1 year ago

On Apple Silicon, writing to memory on a PCIe / Thunderbolt device will generate an exception. ARM spec says you're allowed to write to devices as if they were memory but Apple enforces that all writes to external devices go through a device memory mapping[0]. This makes using an external GPU on Apple Silicon[1] way more of a pain in the ass, if not impossible. AFAIK nobody's managed to write an eGPU driver for Apple Silicon, even with Asahi.

[0] https://developer.arm.com/documentation/102376/0200/Device-m...

[1] Raspberry Pi 4's PCIe has the same problem AFAIK

xuki• 1 year ago

That particular MBP model had a high rate of GPU failure because it ran too hot.

I imagined the convo between Steve Jobs and Jensen Huang went like this:

S: your GPU is shit

J: your thermal design is shit

S: f u

J: f u too

Apple is the kind of company that hold a grudge for a very long time, their relationships with suppliers are very one way, their way or the highway.

VladVladikoff• 1 year ago

Won’t work. No driver support.

dkga• 1 year ago

This is the only strategy humble me can see working for CUDA in MLX

whatever1• 1 year ago

This is the right answer. Local models will be accelerated by Apple private cloud.

hbcondo714• 1 year ago

"write once, use everywhere"

So my MLX workloads can soon be offloaded to the cloud!?

Keyframe• 1 year ago

Now do linux support / drivers for Mac hardware!

lvl1553• 1 year ago

Seriously. Those Apple guys became delusional especially after Jobs passed away. These guys just sat on their successes and did nothing for a decade plus. M1 was nice but that was all Jobs doing and planning. I don’t like this Apple. They forgot how to innovate.

But I guess we have a VR device nobody wants.

jjtheblunt• 1 year ago

It would be funny if you were typing out your response on an iPhone that has been running for 36 hours without recharging.

marcellus232• 1 year ago

M1 was nice but that was all Jobs doing and planning

M1 was launched 9 years after Jobs died. You're saying they had everything ready to go back then and just sat on their asses for a decade?

pxc2• 1 year ago

Seriously. Those Apple guys became delusional especially after Jobs passed away.

Didn't Jobs himself essentially die of delusion?

bigyabai8• 1 year ago

I think we're seeing the twilight of those efforts. Asahi Linux was an absolute powerhouse of reverse-engineering prowess, and it took years to get decent Vulkan coverage and half of the modern lineup's GPUs supported. Meanwhile AMD and even Intel are shipping Vulkan 1.3 drivers day-one on new hardware. It's a cool enthusiast effort to extend the longevity of the hardware, but it bears repeating; nobody is disrupting Nvidia's bottom-line here. Apple doesn't sell hardware competitive with Nvidia's datacenter hardware, and even if they did it's not supported by the community. It's doubtful that Apple would make any attempt to help them.

There seems to a pervading assumption that Apple is still making a VolksComputer in 2025, blithely supporting a freer status-quo for computing. They laid out their priorities completely with Apple Silicon, you're either on Apple's side or falling behind. Just the way they want it.

albertzeyer• 1 year ago

This is exciting. So this is using unified memory of CUDA? I wonder how well that works. Is the behavior of the unified memory in CUDA actually the same as for Apple silicon? For Apple silicon, as I understand, the memory is anyway shared between GPU and CPU. But for CUDA, this is not the case. So when you have some tensor on CPU, how will it end up on GPU then? This needs a copy somehow. Or is this all hidden by CUDA?

MBCook4• 1 year ago

This is my guess, but does higher end hardware they sell, like the server rack stuff for AI, perhaps have the unified memory?

I know standard GPUs don’t.

The patch suggested one of the reasons for it was to make it easy to develop on a Mac and run on a super computer. So the hardware with the unified memory might be in that class.

Y_Y• 1 year ago

The servers don't, but the Jetsons do

ajuhasz• 1 year ago

The Jetsons[1] have unified memory[2].

[1] https://www.nvidia.com/en-us/autonomous-machines/embedded-sy... [2] https://www.nvidia.com/en-us/on-demand/session/gtcspring22-s...

patrickkrusiec• 1 year ago

The physical memory is not be unified, but on modern rack scale Nvidia systems, like Grace Hopper or NVL72, the CPU and the GPU(s) share the same virtual address space and have non-uniform memory access to each other's memory.

freeone30003• 1 year ago

Standard GPUs absolutely do. Since CUDA 11, all CUDA cards expose the same featureset at differing speeds (based on backing capability). You can absolutely (try to) run CUDA UMA on your 2060, and it will complete the computation.

zcbenz• 1 year ago

In the absence of hardware unified memory, CUDA will automatically copy data between CPU/GPU when there are page faults.

fenced_load• 1 year ago

There is also NVLink c2c support between Nvidia's CPUs and GPUs that doesn't require any copy, CPUs and GPUs directly access each other's memory over a coherent bus. IIRC, they have 4 CPU + 4 GPU servers already available.

nickysielicki• 1 year ago

saagarjha7• 1 year ago

This seems like it would be slow…

ethan_smith• 1 year ago

CUDA's Unified Memory uses page migration with on-demand faulting to create the illusion of shared memory, whereas Apple Silicon has true shared physical memory, resulting in different performance characteristics despite the similar programming model.

paulirish• 1 year ago

It's coming from zcbenz who created Electron among others https://zcbenz.com/ Nice.

benreesman3• 1 year ago

I wonder how much this is a result of Strix Halo. I had a fairly standard stipend for a work computer that I didn't end up using for a while so I recently cashed it in on the EVO-X2 and fuck me sideways: that thing is easily competitive with the mid-range znver5 EPYC machines I run substitors on. It mops the floor with any mere-mortal EC2 or GCE instance, like maybe some r1337.xxxxlarge.metal.metal or something has an edge, but the z1d.metal and the c6.2xlarge or whatever type stuff (fast cores, good NIC, table stakes), blows them away. And those things are 3-10K a month with heavy provisioned IOPS. This thing has real NVME and it cost 1800.

I haven't done much local inference on it, but various YouTubers are starting to call the DGX Spark overkill / overpriced next to Strix Halo. The catch of course is ROCm isn't there yet (they're seeming serious now though, matter of time).

Flawless CUDA on Apple gear would make it really tempting in a way that isn't true with Strix so cheap and good.

jitl• 1 year ago

It’s pretty explicitly targeting cloud cluster training in the PR description.

ivape• 1 year ago

If we believe that there’s not enough hardware to meet demand, then one could argue this helps Apple meet demand, even if it’s just by a few percentage points.

nl• 1 year ago

The catch of course is ROCm isn't there yet (they're seeming serious now though, matter of time).

Competitive AMD GPU neural compute has been any day now for at least 10 years.

bigyabai8• 1 year ago

The inference side is fine, nowadays. llama.cpp has had a GPU-agnostic Vulkan backend for a while, it's the training side that tends to be a sticking point for consumer GPUs.

hamandcheese3• 1 year ago

For the uninitiated, Strix Halo is the same as the AMD Ryzen AI Max+ 395 which will be in the Framework Desktop and is starting to show up in some mini PCs as well.

The memory bandwidth on that thing is 200GB/s. That's great compared to most other consumer-level x86 platforms, but quite far off of an Nvidia GPU (a 5090 has 1792GB/s, dunno about the pro level cards) or even Apple's best (M3 Ultra has 800GB/s).

It certainly seems like a great value. But for memory bandwidth intensive applications like LLMs, it is just barely entering the realm of "good enough".

yieldcrv2• 1 year ago

Apple is just being stupid, handicapping their own hardware so they can sell the fixed one next year or the year after

This is time tested Apple strategy is now undermining their AI strategy and potential competitiveness

tl;dr they could have done 1600GB/s

Rohansi2• 1 year ago

You're comparing theoretical maximum memory bandwidth. It's not enough to only look at memory bandwidth because you're a lot more likely to be compute limited when you have a lot of memory bandwidth available. For example, M1 had so much bandwidth available that it couldn't make use of even when fully loaded.

attentive• 1 year ago

how is it vs m4 mac mini?

drcongo• 1 year ago

This was nice to read, I ordered an EVO-X2 a week ago though I'm still waiting for them to actually ship it - I was waiting on a DGX Spark but ended up deciding that was never actually going to ship. Got any good resources for getting the thing up and running with LLMs, diffusion models etc.?

benreesman3• 1 year ago

However excited you are, it's merited. Mine took forever too, and it's just completely worth it. It's like a flagship halo product, they won't make another one like this for a while I don't think. You won't be short on compute relative to a trip to best buy for many years.

adultSwim2• 1 year ago

Do you need to copy to load a model from CPU memory into GPU memory?

orliesaurus• 1 year ago

Why is this a big deal, can anyone explain if they are familiar with the space?

elpakal• 1 year ago

NVIDIA hardware is widely used for academic and massive computations. Being able to write/test code locally on a Mac and then deploy to super computers would make a good developer experience.

That one stands out to me as a mac user.

radicaldreamer• 1 year ago

MacBooks used to use Nvidia GPUs, then Apple had a falling out with Nvidia and the beef stands to this day (Apple didn’t use Nvidia hardware when training it’s own LLMs for Apple Intelligence).

I wouldn’t be surprised if within the next few years we see a return of Nvidia hardware to the Mac, probably starting with low volume products like the MacPro, strictly for professional/high-end use cases.

numpad0• 1 year ago

This PR is an ongoing effort to add a CUDA backend to MLX

looks like it allows MLX code to compile and run on x86 + GeForce hardware, not the other way around.

sciencesama3• 1 year ago

Apple is planing to build data centers with mseries of chips for both app development, testing and to host external services!

mr_toad• 1 year ago

If they were doing that, they wouldn’t need CUDA support. More likely they have internal developers who want to do development on Apple hardware and deploy to Nvidia hardware in production.

lukev3• 1 year ago

So to make sure I understand, this would mean:

1. Programs built against MLX -> Can take advantage of CUDA-enabled chips

but not:

2. CUDA programs -> Can now run on Apple Silicon.

Because the #2 would be a copyright violation (specifically with respect to NVidia's famous moat).

Is this correct?

saagarjha7• 1 year ago

No, it's because doing 2 would be substantially harder.

lukev3• 1 year ago

There's a massive financial incentive (billions) to allow existing CUDA code to run on non-NVidia hardware. Not saying it's easy, but is implementation difficulty really the blocker?

hangonhn• 1 year ago

Is CUDA tied very closely to the Nvidia hardware and architecture so that all the abstraction would not make sense on other platforms? I know very little about hardware and low level software.

Thanks

ls612• 1 year ago

#2 would be Google v. Oracle wouldn’t it?

quitit2• 1 year ago

It's 1.

It means that a developer can use their relatively low-powered Apple device (with UMA) to develop for deployment on nvidia's relatively high-powered systems.

That's nice to have for a range of reasons.

_zoltan_• 1 year ago

"relatively high powered"? there's nothing faster out there.

chvid2• 1 year ago

If Apple cannot do their own implementation of CUDA due to copyright second best is this; getting developers to build for LMX (which is on their laptops) and still get NVIDIA hardware support.

Apple should do a similar thing for AMD.

randomNumber7• 1 year ago

What is the performance penalty compared to a program in native CUDA?

karmakaze• 1 year ago

It would be great for Apple if enough developers took this path and Apple could later release datacenter GPUs that support MLX without CUDA.

dagmx5• 1 year ago

2 also further cements CUDA as the de facto API to target, and nobody would write MLX targeted code instead.

This way, you’re more incentivized to write MLX and have it run everywhere. It’s a situation of everyone wins, especially Apple because they can optimize it further for their platforms.

tekawade2• 1 year ago

I want #3 be able to connect NVIDIA GPU with Apple Silicon and run CUDA. Take advantage of apple silicon + unified memory + GPU + CUDA with PyTorch, JAX or TensorFlow.

Haven’t really explored MLX so can’t speak about it.

musicale• 1 year ago

Nvidia already has unified memory on Grace Blackwell etc.

I guess M5 Blackwell could be better, but there are business and technical barriers to making that happen.

tho234j2344• 1 year ago

I don't think #2 is really true - AMDs HIP is doing this exact thing after giving up on OpenCL way back in ~'17/'18.

NekkoDroid• 1 year ago

I haven't looked into it, but doesn't HIP need everything to be recompiled against it? To my understanding it was mostly a source code translation effectivly.

sitkack• 1 year ago

#2 is not a copyright violation. You can reimplement APIs.

7734128• 1 year ago

The famous Android Java fight is probably the most important case of that discussion.

adastra224• 1 year ago

CUDA is not an API, it is a set of libraries written by NVIDIA. You'd have to reimplement those libraries, and for people to care at all you'd have to reimplement the optimizations in those libraries. That does get into various IP issues.

pxc2• 1 year ago

Copyright can't prohibit compatible implementations that are developed independently through reverse engineering, if the implementers are very careful about the way they work.

(Software patents can, though.)

natas2• 1 year ago

that means the next apple computer is going to use nvidia gpu(s).

meepmorp2• 1 year ago

but it's not an apple-submitted pr

natas2• 1 year ago

they can't make it that obvious

MBCook4• 1 year ago

There’s no evidence of that. The post clearly identifies a far more probable reason in letting things be developed in Mac’s then deployed on Nvidia supercomputers.

dnchdnd• 1 year ago

Random aside: A lot of the people working on MLX don't seem to be officially affiliated with Apple at least in a superficial review. See for example: https://x.com/prince_canuma

Idly wondering, is Apple bankrolling this but wants to keep it in the DL? There were also rumours the team was looking to move at one point ?

jpcompartir• 1 year ago

It seems more like Open Source devs who are looking to build clout/rep with MLX?

Pretty sure Claude Sonnet is actually doing most of the work.

Abishek_Muthian• 1 year ago

I’ve been very impressed with MLX models; I can open up local models to everyone in the house, something I wouldn’t dare with my Nvidia computer for the risk of burning down the house.

I’ve been hoping Apple Silicon becomes a serious contender for Nvidia chips; I wonder if the CUDA support is just Embrace, extend, and extinguish (EEE).

m3kw9• 1 year ago

I thought you either use MLX for apple silicone or you compile it for cudaw

neurostimulant• 1 year ago

Being able to write/test code locally on a Mac and then deploy to super computers would make a good developer experience.

Does this means you can use MLX on linux now?

Edit:

Just tested it and it's working but only python 3.12 version is available on pypi right now: https://pypi.org/project/mlx-cuda/#files

qwertox• 1 year ago

If Apple would support Nvidia cards it would be the #1 solution for developers.

Nevermark2• 1 year ago

If Apple doubled the specs of their Ultra M processor every year, in numbers of cores, RAM cells, internal and external bandwidth, until both the Ultra processor and its RAM plane took up full wafers, .... but still fit in a Mac Studio case, with a new white reverse-power heat extraction USB-C+ cable designed to be terminated at a port on a small wireless heat exchanger dish, which instantly beamed all the waste heat into space, at such high efficiency that the Studio internals could operate at -100 Celsius, and all those cores overclocked, oh they over clocked, ...

Yes we can dream!

It would great if Apple continues pushing M processors to next levels, in part, to go vertical into the cloud.

Or if they start supporting nVidia.

The latter seems less Apple-y. But they must be considering the value of a cloud level Apple-friendly AI computing solution, so something is likely (?) to happen.

bigyabai8• 1 year ago

Nvidia already maintains BSD-native drivers that macOS can use. The only work Apple has to do is implement EGLStream in Quartz, which is probably already done. And even that isn't necessary to get CUDA or video acceleration working. In theory, the only thing stopping Nvidia from putting those features on macOS is signed driver code.

It's not rocket science, nobody has to "dream" for such an ordinary feature. On Linux and Windows it's treated as table-stakes, which is part of why they're desirable cloud OSes and macOS isn't. Apple certainly knows as much, it's not like they have customers begging to bring back Xserve.

adultSwim2• 1 year ago

This is great to see. I had wrongly assumed MLX was Apple-only.

neuroelectron• 1 year ago

Just remember to name for fp8 kernels "cutlass" for +50% performance.

mattfrommars• 1 year ago

It's year 2025 and we have yet to have impact of CUDA like what Java had in the idea, "write once, run it anywhere"

Academia and companies continue to write proprietary code. Its as if we continue to write code for Adobe Flash or Microsoft Silverlight in year 2025.

Honestly, I don't mind as Nvidia shareholder.

raincole• 1 year ago

In the end Java doesn't achieve "write once, run it anywhere" either.

I guess there might be a way to develop apps for iOS or even PlayStation in Java, but my knees hurt just thinking about how many hoops one needs to jump through.

bshacklett• 1 year ago

I’m currently working on migrating Java code from mid-range systems to containers in the cloud. The number of code changes required is near zero. They may not have gotten portability perfectly solved, but it’s pretty darn good compared to many other platforms.

Now, if the industry could just get out of the ridiculous Java 8/11 rut, we’d be in good shape.

bigyabai8• 1 year ago

I'll never get over the way Apple treated OpenCL. They saw the train coming down the tracks, spent so long hedging their bet against CUDA, and threw in the towel the moment actual demand started cropping up. CUDA very nearly had a serious, corporate-funded and write-once-run-anywhere competitor.

Normally I write something snide about not seeing where the puck was headed. But Apple did skate to the puck the puck here, they just did nothing with it.

int_19h4• 1 year ago

Back in the day, the reason why people kept targeting Flash is because all the other alternatives were worse. If you recall, the only thing that made a difference was mobile, where Flash ended up being a liability due to performance and battery lifetime issues. And even then it took a company like Apple, which could rely on its cult status to draw the hard line on mobile Flash, ship iPhone without it (unlike Android which had it, warts and all), and steadfastly refuse to even consider adding it, forcing everybody else to choose between using Flash and supporting the lucrative iPhone ecosystem.

I'm not even sure what the equivalent would be for CUDA tbh.

bigyabai8• 1 year ago

Apple could just, talk to Khronos again. In any protracted discussion of "how can the industry kill Nvidia", we always circle back around to the lack of communication. There was an era where Apple, AMD and even Nvidia all worked on Open Source non-CUDA acceleration primitives. There were working drivers, (a handful of) users, and bindings in multiple native languages. All they needed was industry applications, which would arrive with the crypto mining boom that Nvidia profited off of hand-over-fist. And by then, Apple refused to cooperate with their industry partners, and refused to support OpenCL on iPhone GPUs or Apple Silicon. Metaphorically, this would be like Apple refusing to implement HTML because they wanted to promote their own Flash alternative.

Nvidia won because they don't deal with this level of asinine infighting. If Khronos could bring back a level of mutual respect to their consortium, they could present a serious threat. Apple is the only business still on their high horse; AMD, Intel and Qualcomm all recognize that they need to cooperate.

m463• 1 year ago

"write once, run it anywhere"

I think you mean:

"write once, test everywhere"

Apple's MLX adding CUDA support

Related Stories