Run LLMs on Apple Neural Engine (ANE) 1 year ago
Something to consider for deploying LLMs on the ANE is: https://github.com/pytorch/executorch/tree/main/examples/app...
The model does have some limitations (e.g., need for QAT for 4-bit quantization), lack of a C++ runner to execute the model, but parts of the model are promising.
If interested in further discussion, join the conversation on the ExecuTorch discord channel: https://discord.gg/xHxqsD5b