Ask HN: Can On-device AI solve projected energy crisis?

https://news.ycombinator.com/item?id=43965017

by vkkhare • 1 year ago

1 3 1 year ago

news.ycombinator.com

bigyabai2• 1 year ago

No, on-device AI is not a replacement product for the high-consumption AI that is casuing the problem.

vkkhare• 1 year ago

Isn't the inference cost of running these models at scale challenging? Currently it feels like small LLMs (1B-4B) are able to perform well for simpler agentic workfows. There are definitely some constraints but surely much easier than to pay for big clusters on cloud running for these tasks. I believe it distributes the cost more uniformly

bigyabai2• 1 year ago

It is very likely that you consume less power running a 1B LLM on an Nvidia supercluster than you do trying to download and run the same model on a smartphone. I don't think people understand just how fast the server hardware is compared to what is in their pocket.

We'll see companies push for tiny on-device models as a novelty, but even the best of those aren't very good. I firmly believe that GPUs are going to stay relevant even as models scale down, since they're still the fastest and most power-efficient solution.

Ask HN: Can On-device AI solve projected energy crisis?

Related Stories