Running DeepSeek R1 Models Locally on NPU

https://blogs.windows.com/windowsdeveloper/2025/01/29/running-distilled-deepseek-r1-models-locally-on-copilot-pcs-powered-by-windows-copilot-runtime/

by doomroot13 • 1 year ago

37 15 1 year ago

blogs.windows.com

jokowueu• 1 year ago

How much are NPUs more efficient than GPUs ? What are the limitations , it seems it will have support for deepseek R1 soon

tamlin3• 1 year ago

A decent chunk of AI computation is the ability to do matrix multiplication fast. Part of that is reducing the amount of data transferred to and from the matrix multiplication hardware on the NPU and GPU; memory bandwidth is a significant bottleneck. The article is highlighting 4-bit format use.

GPUs are an evolving target. New GPUs have tensor cores and support all kinds of interesting numeric formats, older GPUs don't support any of the formats that AI workloads are using today (e.g. BF16, int4, all the various smaller FP types).

NPU will be more efficient because it is much less general an GPU and doesn't have any gates for graphics. However, it is also fairly restricted. Cloud hardware is orders of magnitude faster (due to much higher compute resources I/O bandwidth), e.g. https://cloud.google.com/tpu/docs/v6e.

justincormack2• 1 year ago

NPU also has no more memory bandwidth than CPU, but then the GPU on these machines doesnt either.

RandomBK3• 1 year ago

Reminder: DeepSeek distilled models are better thought of as fine-tunes of Qwen/Llama using DeepSeek output, and are not the same as actual DeepSeek v3 or R1.

This unfortunate naming has sown plenty of confusion around DeepSeek's quality and resource requirements. Actual DeepSeek v3/R1 continues to require at least ~100GB of VRAM/Mem/SSD, and this does not change that.

darthrupert• 1 year ago

Wait, what am I running on my 32GB Macbook then? I thought it was the 32b version of deepseek-r1.

Plankaluel• 1 year ago

You are running Qwen2.5 32b that has been fine tuned on data that was generated by R1

RandomBK3• 1 year ago

The only 32B distill I'm aware of is `DeepSeek-R1-Distill-Qwen-32B`, which would be a base model of `Qwen-32B` distilled (further trained) on outputs from the full R1 model.

rahimnathwani2• 1 year ago

Deepseek R1 has 671 billion parameters. Even if you could quantize each parameter to just 1 bit (from 8 bits), you'd still need 84GB of RAM just for the weights. There is no 32B parameter version of the V3/R1 model architecture.

bestouff2• 1 year ago

Out of curiosity, would an A100 80GB work for this ?

bestouff2• 1 year ago

Replying to myself: apparently it's not 100GB VRAM but more around 700GB VRAM that's needed to run DeepSeek R1. The gear needed to run that would cost something in the vincinity of 100K€ !

Running DeepSeek R1 Models Locally on NPU

Related Stories