What we know about the Apple Neural Engine

https://github.com/hollance/neural-engine

by SerCe • 3 years ago

337 97 3 years ago

sebzim45002• 3 years ago

Can this really be everything publically known about the ANE? Sounds hard to believe, I would have thought someone would have reverse engineered something about it by now.

detrites3• 3 years ago

My question too. This semi-answer on the page seems to contradict itself (source: https://github.com/hollance/neural-engine/blob/master/docs/p... ):

"> Can I program the ANE directly?

Unfortunately not. You can only use the Neural Engine through Core ML at the moment.

There currently is no public framework for programming the ANE. There are several private, undocumented frameworks but obviously we cannot use them as Apple rejects apps that use private frameworks.

(Perhaps in the future Apple will provide a public version of AppleNeuralEngine.framework.)"

The last part links to this bunch of headers:

https://github.com/nst/iOS-Runtime-Headers/tree/master/Priva...

So might it be more accurate to say you can program it directly, but won't end up with something that can be distributed on the app store?

saagarjha4• 3 years ago

Correct. (It is also unlikely that Apple exposes the Neural Engine directly.)

SomeHacker44• 3 years ago

See other commenter above about GeoHot's analysis which is much more in depth.

djoldman• 3 years ago

geohot's findings:

https://github.com/geohot/tinygrad/tree/master/accel/ane

detrites3• 3 years ago

Ok, this is much more like what I expected from the OP.

Anyone disappointed, here be full details on everything.

alexdbird• 3 years ago

This info is a lot less useful to me, a developer for iOS, than the OP. What are you trying to do that makes these details interesting?

ljlolel• 3 years ago

https://news.ycombinator.com/item?id=35302833

gymbeaux• 3 years ago

Is this the same geohot that got persecuted by Sony for jailbreaking the PS3?

ncr100• 3 years ago

Yes and spent years building a self driving kit and AI training background for many regular cars.

ah-• 3 years ago

There's also basic ANE support for Asahi now:

https://github.com/eiln/ane

https://github.com/eiln/anecc

https://github.com/AsahiLinux/m1n1/pull/296/files

rowanG077• 3 years ago

That's misleading. It's much more apt to say it's being worked on. This is not available in any Asahi release at this time.

callamdelaney• 3 years ago

There is an initial commit for ane support from yesterday. That means it's likely a long way off being deployed to asahi-edge.

jeffrallen• 3 years ago

Âne means donkey in French.

Just sayin'.

marginalia_nu2• 3 years ago

Alright, then, Apple Semantic System.

mrweasel• 3 years ago

Which shortened also means donkey, brilliant.

eastbound• 3 years ago

No those initials were already taken by Atlassian Software Systems. They seem to have lodged the paperwork with that name in 2002, and to have dropped it later on (they rather went with TEAM when going IPO in 2015), but back in 2010 when I applied, there was a book (collection of news articles) in the waiting room for candidates titled “Atlassian Software Systems”.

Great guys.

https://youtu.be/VfyUbuFoiBU

bitL• 3 years ago

It's really terrible that Apple markets this as the next big thing but forgets to include detailed documentation so people have to experiment and figure out what works...

lonelyasacloud• 3 years ago

Part Apple's docs haven't been great for a while, part that's just how they roll, and part trying (like most everyone) to figure out what their strategy is going to be in a post GPT4 world [0].

[0] Persist with their own models running locally, how much to integrate with rest of the OS and maintain privacy moral ground, that sort of thing.

barkingcat• 3 years ago

Apple didn’t “forget” they never want to ever release apple proprietary docs. It’s their competitiveness advantage/ moat.

smoldesu• 3 years ago

It's a damn shame, too. If we got the APFS documentation we were promised I feel like we'd be one step closer to world peace.

cynicalsecurity• 3 years ago

Proprietary software, dude. It really sucks.

saagarjha4• 3 years ago

People don't have to do anything. You use CoreML to program it.

grupthink• 3 years ago

It's not that simple. If you have a model that actually does something useful (e.g. not just doing matmul & conv2d) your model will fail to run on ANE and, instead, the device will move it over to CPU/GPU and turn your iPhone into a heater. I literally had to continually wipe down my iPhone with a wet towel to keep it from overheating so I could build, ct.convert, run, and debug a model I was working on. Apple doesn't document how to keep operations on ANE. A model created by coremltools may run on either CPU, GPU, ANE, but you don't get to choose. And, if you don't know what you're doing and naively build a model, you will likely run on CPU/GPU only. If your batch is too large, too small, if you need to transpose tensors, if you need to expand mismatched tensors to matmul them together, if your model has an IF branch, or a loop, if you breathe the wrong way, your model silently falls off ANE. But you don't know what caused it. You have to open Netron and guess. Also, it may run on ANE on one device, but not another. There's no documentation from Apple about any of this. So, no, you do not simply "use CoreML to program it".

nhubbard• 3 years ago

It's probably more nuanced than what everyone else is saying. The Neural Engine compute block can change pretty significantly from one chip generation to the next, and instead of exposing the unstable raw capabilities, they use CoreML as an abstraction to keep the changes out of sight. Is it rather annoying? Yes, but it's a more stable method to keep everyone using the Neural Engine from having their software break regularly.

m3kw9• 3 years ago

Can LLM run in it?

egman_ekki• 3 years ago

maybe https://github.com/apple/ml-ane-transformers

simonw3• 3 years ago

So my phone and my laptop both have the capability to perform 15 trillion operations per second, just in the neural engine?

What kind of things are taking advantage of this right now? It's gotta be more than just Face ID right?

What's my laptop likely to be doing with that?

waboremo• 3 years ago

Scene analysis in photos, image captions, and machine translations are also done using ANE. CoreML also utilizes it when possible.

fauigerzigerk5• 3 years ago

I don't know for sure, but things like text recognition (Live Text) or object recognition in Photos (Visual Look Up) are obvious candidates.

I think neural engine is absolutely key to Apple's strategy. They want people to buy expensive devices and they don't want to process user data on their servers.

Users get privacy. Apple gets money. It's a pretty coherent business model.

flutas• 3 years ago

They want people to buy expensive devices and they don't want to process user data on their servers.
Users get privacy. Apple gets money.

Apple also gets users to subsidize the cost of compute indefinitely (by buying the expensive phone), rather than using their servers.

jjoonathan2• 3 years ago

Privacy isn't the only benefit of local compute, users also get colossal bandwidth, tiny latency, and high reliability.

gcr• 3 years ago

From my understanding (which seems to differ from OP), the benefit of running models on ANE is for much lower power usage, not necessarily speed.

Apple’s always-on Hey Siri wake word detection uses ANE for minimal battery life.

I’m not sure but I think android’s “Now Playing” feature (the always-on Shazam thing that shows the names of songs playing around you on your lock screen ) also uses Edge TPU for similar reasons.

blululu4• 3 years ago

To my knowledge this is mostly used by internal tools, though a number of common 3rd party apis (qr code scan) use hw acceleration under the hood. Internally there is a ton of ML running on the device. The most obvious is touch screen and inputs and the camera. 3rd party developers have acres to this via CoreML, but unless latency is critical it is usually easier to develop and run ml on the cloud. For camera apps using ml, this chip is going to be used either explicitly or implicitly.

simonw3• 3 years ago

Oh the touch screen! That's fascinating, is that definitely running stuff on the neural engine?

iamgopal• 3 years ago

15 trillion operation per second ? Of what kind ? Addition ? Isn’t that mind blowing ?

selectodude• 3 years ago

Matrix multiplication

gedy• 3 years ago

I can imagine some thing like Siri running on device much more effectively against local content. The cynic in me doesn’t want to hope too much for cloudless services like this, but one can hope.

sroussey2• 3 years ago

This is true since iOS 15 moved Siri on device.

conradev2• 3 years ago

It’s used for a variety of things:

- Biometrics (Face ID and Touch ID)

- Image analysis (face matching, aesthetic evaluations, etc)

- Text to speech and speech to text (smaller models on device, used for privacy/latency/reliability)

- Small ad-hoc models like Raise to Speak on Apple Watch, the Hey Siri detector (https://machinelearning.apple.com/research/hey-siri)

These things have been in phones for 5 years now and have been used from day one

simonw3• 3 years ago

Right, but do any of those things really need 15 trillion operations for second? Have they been getting noticeably better with upgraded phone models?

k_bx• 3 years ago

They're putting it everywhere they can. From Notes to pressing pause on Video in QuickTime or Safari and copying text from a frame instantly.

ManuelKiessling• 3 years ago

Running Microsoft Teams. Barely.

IIAOPSW• 3 years ago

Siri wasn't a product. She was an emergent feature they couldn't extinguish.

secretsatan• 3 years ago

Arkit makes use of it on the phone, there’s plane detection and classification, image and object detection, segmentation for people occlusion, probably more behind the scenes.

I find it a little frustrating we aren’t using the built in capabilities of iphones more in our company, i still kinda think apple tech is kind of a pariah in some circles, so we have to run with stuff that runs on cloud that costs us money over, heaven forbid something you could run on an iphone

kmeisthax• 3 years ago

There's an app called Draw Things, for iOS/iPadOS/macOS/etcOS, that uses the ANE to run Stable Diffusion on your phone/tablet/laptop.

nl4• 3 years ago

You can run large language models locally using it: https://machinelearning.apple.com/research/neural-engine-tra...

I know the Swift version of Stable Diffusion from Hugging Face uses this too.

Our_Benefactors• 3 years ago

While it’s true you can run some parts of a machine learning stack, Apple still lags way behind nvidia due to not having CUDA support. It makes a huge difference in both speed and quality of output.

zamnos• 3 years ago

https://www.rewind.ai/ is one that I know of. it only works on apple Silicon but their software records the shit out of everything you do on your computer, so you never have that moment when you half remember something you did the past. just pull up rewind and find it!

anentropic• 3 years ago

Do we think Apple are going to provide more info and maybe a public API over time?

Or they are keeping it obscure for commercial reasons?

Or just not very competent/don't care?

Seems weird having these amazing chips and only blunt tools

my123• 3 years ago

CoreML.

Directly exposing the ANE wouldn't make much sense, as it's an IP block that changes between generations in incompatible ways.

brookst• 3 years ago

This is the answer. CoreML gives you an abstraction over different generations and sizes of underlying NPU.

You might not want the abstraction, but love it or hate it, that’s kind of the Apple way.

It will be very interesting to see what their next chips look like since we’re getting to the point where HW designs will reflect the rise of the, uh, transformers.

enzomanico• 3 years ago

Yes of cour

mmaunder2• 3 years ago

Anyone done any work on using a model for transcription on the local device using the ANE? I've heard it kills the battery. Having to transcribe voice in the cloud is a serious impediment to end to end encryption for certain applications.

intalentive• 3 years ago

This is close: https://github.com/ggerganov/whisper.cpp

mmaunder2• 3 years ago

Thanks!

ls6122• 3 years ago

Does anyone know if the neural engine on the new M1/M2 Max is directly hooked up to the unified memory the way the GPU is?

wmf• 3 years ago

Define directly I guess.

ls6122• 3 years ago

My understanding is that the CPU and GPU both have DMA to the memory at some incredible speed since it’s all on the same chip. Does the ANE have that same DMA speed and latency?

Tkuivamaki• 3 years ago

Siri is still pants tho

cubefox• 3 years ago

Would be interesting to compare its performance to similar AI cores in other SoCs, such as Qualcomm's Snapdragon or Google's Tensor.

What we know about the Apple Neural Engine

Related Stories