Apple Core AI Framework

https://developer.apple.com/documentation/coreai/

by hmokiguess • 1 month ago

367 108 1 month ago

developer.apple.com

Wow, this seems to be a new way to convert PyTorch models to a format that runs across CPU, GPU & Apple's Neural Engine (ANE). [0]

Does this completely replace the previous API, CoreML? [1]

  [0]: https://apple.github.io/coreai-optimization/
  [1]: https://developer.apple.com/documentation/coreml/

earthnail• 1 month ago

Yes. From the CoreAI docs:

"If your app uses model types other than neural networks, such as decision trees or tabular feature engineering, see Core ML."

pzo• 1 month ago

seems they planning to replace it but overall now I'm really confused about this and mlx and coremltools. They should do better work explaining the benefits (and cons) of it and any feature parity between coreai, coreml and mlx.

LoganDark2• 1 month ago

My reading of it is:

- Core ML is for models designed only for Apple platforms

- MLX is for models that don't need to be fast

- Core AI is for models that run everywhere already and also need to be fast

ABS2• 1 month ago

looks to me like the docs don't give a feature-parity table, but they do draw the "role" lines once you read across them:

- Core ML narrows to classic, non-neural ML (its own docs now point you there for "decision trees or tabular feature engineering")

- Core AI takes neural nets and transformers (the new .aimodel format, the new profiler)

- MLX stays the separate bring-your-own-weights track (its WWDC sessions draw no line back to Core AI at all)

coreai-opt is the successor to coremltools on the optimization side.

wahnfrieden5• 1 month ago

Requires OS 27+, so CoreML is still useful for backwards compatibility.

sgt2• 1 month ago

macOS users aren't that good at upgrading regularly, but iOS users are at least obsessive about upgrading to the latest OS. I guess the system almost forces us.

trollbridge2• 1 month ago

This is just a bit exciting, although I wonder how the performance of this will stack up next to the stuff we already do with, e.g., a metal-optimised model which we then load into llama-cpp or whatever. (unsloth is a good example of doing this for you "batteries included").

ElFitz2• 1 month ago

A few months back someone reverse-engineered private ANE APIs and shown some significant performance improvements compared to CoreML and Metal, on both inference and training.

- https://maderix.substack.com/p/inside-the-m4-apple-neural-en...

- https://news.ycombinator.com/item?id=47257931

MysticOracle• 1 month ago

WWDC 2026 Core AI videos

Meet Core AI - https://developer.apple.com/videos/play/wwdc2026/324/

Dive into Core AI model authoring and optimization - https://developer.apple.com/videos/play/wwdc2026/325/

Integrate on-device AI models into your app using Core AI - https://developer.apple.com/videos/play/wwdc2026/326/

franze3• 1 month ago

i am more excited about the ondevice foundation model update that is coming https://developer.apple.com/documentation/updates/foundation... (not much info yet)

but i maintain https://github.com/Arthur-Ficial/apfel so i might be biased

crancher• 1 month ago

Apfel is very useful, thanks for the effort.

cat5e2• 1 month ago

I second this, I’m more excited about dumb local models than something I could never run locally.

trollbridge2• 1 month ago

Thanks for building this! Something I grab on a regular basis, especially for doing simple education of folks about the basics of using LLMs by showing something that's not just a chatbot.

robgough2• 1 month ago

Have you seen that they've added an `fm` tool? It was mentioned in the Platforms State of the Union.

Here's what you get when you run it... https://gist.github.com/robgough/7893602895e7580117475076198...

ElFitz2• 1 month ago

Oh, neat. Totally missed it, thanks!

harrouet• 1 month ago

Wait, is PPC open bar ?

dofm6• 1 month ago

Is ‘fm serve’ OpenAPI compatible?

franze3• 1 month ago

did miss it until now, cool to see it on device and first party. as soon as it lands I will see the impact on apfel.

but i definitely feel flattered, either my little project inspired them or that I reached the same conclusion at a similar time as a team at apple that "hey, this is totally missing"

mips_avatar• 1 month ago

Seems like they still won’t let you run models on GPU while the phone is closed or the user switches apps

tyre3• 1 month ago

This is good. Apps would not be respectful and end up draining users’ batteries to zero in no time.

dofm6• 1 month ago

Are you surprised they apparently didn’t adopt your idea and add an OpenAPI compatible endpoint in Core AI, even if just as a testing tool? I am.

I also really want to hear more about their containerisation/seatbelt strategy now that they are offering MCP support. Not seen any news about Darwin inside their containers system.

(Apfel is a cool project; it’s been the only thing tempting me to upgrade to Tahoe)

mark_l_watson• 1 month ago

Thanks apfel looks useful! I have been experimenting with Apple's foundation models for almost a year and they are useful for embedded applications. I have been taking a deeper dive into local agentic coding tools (starting with 'little-coder --model ollama/gemma4:12b-it-qat') and I put together a tiny free book with some setup advice that might save people a few minutes of setup time: https://leanpub.com/read/local-coding-agents

I have been fairly much pissed off at the "hype in hyperscaler" AI growth (data center environmental and other societal costs) and I support anything we can do to promote local and private AI.

jjice2• 1 month ago

Agreed. The idea of a system wide (and platform wide) on device model being a core part of OS APIs is very appealing. I do like my software more piecemeal, generally, but when it comes to Apple, I really love a lot of the out-of-the-box offerings they have. Just giving software access to something they know exists on these platforms and can use for various small (and likely increasingly large) gen AI tasks is so appealing.

an0malous• 1 month ago

This is why the AI companies are rushing to IPO. By the end of next year you’ll be running most of your AI on device. They have no moat, they’ve reached the limits of scaling, most of the magic can be distilled into smaller models, and they know it

sealeck• 1 month ago

Have we reached the limits of scaling? Sadly it appears that larger model still equals better model

stogot• 1 month ago

It’s still diminishing returns yes? It isn’t Moore’s Law

pixelready• 1 month ago

I think there’s still an open question around are the ultra-large next-gen models worth it? For those of us without early access to Mythos, it’s hard to verify whether it’s been held back from the public due to actually being “too dangerously powerful to release yet” as implied or because the gains aren’t outpacing the costs.

mikestorrent• 1 month ago

Well, let's not forget that text models are not the only models! Video models are much slower and need comparatively more resources, and all they can do even at that size is generate videos a few seconds long. Clearly a ton more work is going to go into those, and demand for them will probably increase as more creative tools get authored using them as a central part of the workflow. Low-res local rendering for preview might be a thing, but the lion's share of the work for high-res, near-realtime rendering is going to be done on huge clusters for a long time yet.

mindwok• 1 month ago

I think GPT 4.5 showed that there is indeed a practical limit we're close too. That was supposedly a high-trillions of parameter model that was deprecated almost immediately because it was slow, insanely expensive, and had questionable benefits over the smaller models. Though apparently the new Mythos and whatever GPT Spud is (if it wasn't 5.5) are back up in the high trillions.

adgjlsfhk1• 1 month ago

yes and no. We've reached the point where larger models are higher quality, but they're also too expensive and slow to be used broadly. The giant models, however are still useful for training smaller models that are actually deployable.

hadlock• 1 month ago

Qwen's ~30B-class models are genuinely good enough for use if you can find a machine with enough memory bandwidth to run them at 30-90 tokens/second. It's been extremely telling that Qwen stopped releasing 120b class models. At some point in the next 10 years (maybe 3?) someone is going to release an Opus 4.5 class 256B model you can run locally. Right now our engineers use about $800/mo worth of opus tokens; at that rate the ROI for local LLM is ~10 months

strictnein2• 1 month ago

Didn't Qwen stop releasing their more powerful models because they're commercializing them?

horsawlarway4• 1 month ago

I want to echo this.

I've been on claude's opus 4.5/6/7 for work for a couple months, and I finally got back to running Qwen A3B 35B... it's incredibly performant and quite capable on semi-reasonable local hardware.

I get ~150 tokens/s on dual nvidia RTX 3090s and can fit the whole 300k context into gpu on a UD-Q4-K-XL quant gguf.

Combined with Pi as a harness, and I'm surprised to find that it feels about as capable as claude did 8 months ago (their 3.x models).

It's not Opus 4.5 levels yet, but it's good enough for a LOT of basic work. I actually downgraded my personal anthropic subscription because Qwen is absolutely fine for implementation work. I still let a better model write a plan, but then I can just switch over to Qwen to implement.

I don't think we're 10 years away from opus 4.5 levels running on cheap consumer hardware. I think we're probably closer to 18 months away, and I suspect it'll be in the 30-60b range, not the 256b range.

PC manufacturers also seem to be betting on local, with a LOT of focus on 64 to 128gb unified RAM machines.

cat5e2• 1 month ago

Huzzah, they’ve lost their stranglehold. Viva la revolution!

ActorNightly4• 1 month ago

Very false.

I use small models exclusively. They aren't a replacement for large models. You need decent hardware to run those models efficiently, as smaller parameter models plain suck and are still slow on macbooks. And affordability of higher end hardware is very limited.

Even at non VC subsidized $/token prices, its still much cheaper to run cloud based models.

davnicwil• 1 month ago

well to be fair that's right now, I think the question is what about in 6 months, 12 months, 2 years?

Where do these improvement curves go? Does the gap close, do they intersect for practical purposes (factoring in cost etc)? Or is the local curve always just a translation of the hosted, lagging behind, or indeed does hosted just pull ahead?

Nobody knows, but it's a very open question I feel, and it certainly appears like the answer might quite reasonably be that yes they intersect on that kind of short-ish term time horizon.

dvt3• 1 month ago

Even at non VC subsidized $/token prices, its still much cheaper to run cloud based models.

On a price-per-wattage level, this is not true, people have done the math on /r/LocalLLaMA many times over[1]. Local models, while not as good as premier models (GPT 5.5, etc.), are like ~80%+ of the way there, and often converge to a similar solution after a few dead ends.

[1] https://www.reddit.com/r/LocalLLM/comments/1kshq4f/electrici...

iwontberude3• 1 month ago

Just need a decent Mac Studio and they are plentiful in used condition and affordable.

viccis• 1 month ago

I just want a tiny tiny model that runs on device that knows for autocomplete that, for example, I want to say "I'll be right back" instead of "I'll be right Brian". That's my #1 AI ask right now. Please, Apple.

cush• 1 month ago

I want Siri to let me “add to my calendar, dinner Peter’s house Sunday at 5pm” and not assume the location is the restaurant called Peter’s House in another state. It’s astounding how poor Siri is at using the data I’ve given it access to

wyager• 1 month ago

By the end of next year you’ll be running most of your AI on device.

I expect I'll probably keep paying for whatever badass high IQ model is running on inference servers at that point

maxdo2• 1 month ago

Why on earth I should switch from a top tier model to much worse local model ? Why do I need to suffer my battery ?

truncate• 1 month ago

You can switch to local models for tasks/use-cases where you don't need top tier models.

romanovcode• 1 month ago

Right now there is no reason since tokens are subsidized heavily. However when OpenAI/Anthropic will drop the $200/month pricing since most likely it eventually will become unsustainable you'd rather get MacBook Pro M6 Ultra with 128GB ram and go local then pay thousands every month for tokens.

hajile• 1 month ago

In the coding realm, I think we'll be seeing 35, 70, and 150B models sold where you pay a few hundred to a few thousand dollars up front and get a year of monthly/bi-monthly updates where they've trained it on new coding documentation and repos.

criddell• 1 month ago

Is there something like this on Linux? For example, if I’m an application developer can I assume GNU Core AI (or whatever it is or would be called) will be there if the kernel is >= some particular version?

wtallis• 1 month ago

On non-Apple platforms, you generally have at least 2+(number of supported silicon vendors) different AI frameworks to worry about. I guess Apple's there now too, between Core ML, MLX, Core AI.

I haven't seen any sign that the framework fragmentation problem is going away anytime soon. NVIDIA wants everyone to do all training and inference with CUDA and to deny that NPUs have any usefulness. Everybody making an NPU has a different framework tailored to their architecture and the limitations they inherited from hardware designed before LLMs existed, and most of them have a another framework for targeting a GPU. And the OS vendor has one or two frameworks they would prefer you use rather than something hardware-specific.

nl• 1 month ago

For practical purposes llama.cpp is this. You can link to it or use the network API.

halJordan• 1 month ago

No there isn't. RedHat and IBM do though, for their distros

teravor• 1 month ago

onnxruntime, llama.cpp (more specifically, ggml), iree.dev is also trying

dvt3• 1 month ago

AI future is clearly local, and my recent pitch has been "infinite tokens." Because that's what my M1 MBP can do; and that's what my RTX3090 can do. I don't need to pay hundreds of dollars a month and no one else does either.

ip26• 1 month ago

Infinite tokens rate-limited to 10 tok/s is 26MTok per month.

doctorpangloss• 1 month ago

10? think closer to 5. 13M is like ~7 codex sessions…

fedeb95• 1 month ago

the real money is in the coding surrounding models to make them efficient at specialized tasks. Casual users want general purpose models, and AI chat apps will stay for them. Most programs can benefit from a specialized AI that can be local, and #programs >> #users.

AdamN• 1 month ago

Also context - there's alot of context out there and it's faster to get it from servers.

It doesn't matter how good the model is if it doesn't have context from data sources.

pmontra• 1 month ago

In the 80s we thought that the future of computing was clearly local, home computers, PCs, Macs, the office server (Novell, then Windows NT with disk shares) etc. Add 40 years and we are back to a centralized infrastructure with the modern equivalent of smart terminals.

The AI future will be clearly... what it will be. Probably bouncing back and forth from local to centralized. However, if there are money to be made by selling things that people run locally, it seems that centralizing creates more power and hence more money.

connectsnk• 1 month ago

Do we know what is the underlying model? Is it a custome model developed by Apple or one of gemma/deepseeks under the hood

jacobr1• 1 month ago

The new siri models will be some variant of the gemini models. This framework seems to be more generalized than that though.

ankit219• 1 month ago

they are also working on activations (w4a8, w4a16 from what i know). if they deliver (and a big if), it means that given their market reach, they can dictate the way sub 100b parameter models are trained and served to a large extent, given their major usecase would be on device (macos and not ios for most of them).

scosman• 1 month ago

Free server-size model access for apps with <2M downloads, getting the same privacy guarantees. Hopefully they scale this up to all apps in time (I assume hardware/cost constrained, but larger devs would pay).

https://developer.apple.com/private-cloud-compute/

wahnfrieden5• 1 month ago

My guess based on the Apple Intelligence Extensions mentions is that they will not scale that up anytime soon, but they will allow developers to integrate with other providers that the user has an account with.

JV00• 1 month ago

Does it mean I can run whatever I want on ANE? Last time I tried it seemed it could only be used by first party features such as Face ID

wahnfrieden5• 1 month ago

Been doing that for years with CoreML

jkman2• 1 month ago

You've already been able to do that once you convert your model to CoreML, it's only MLX that's never been able to use the ANE.

ABS2• 1 month ago

something I haven't seen highlighted anywhere yet, while I find it very interesting, is the distributed inference across Macs (JACCL over Thunderbolt 5), an OpenAI-compatible mlx_lm.server, agentic-on-Mac.

Apple keeps MLX (bring-your-own-weights) separate from Foundation Models / Core AI.

meyusufdemirci• 1 month ago

Waiting for the on device models specially. They will be game changers if they will be clever models enough. BTW, I also have an recap article of Core AI framework, https://appcircle.io/blog/wwdc26-apple-core-ai-framework-exp...

Apple Core AI Framework

Related Stories