hagen8

Just switch the model, its not that much effort tbh. And u can also get a cheaper model than 2.5 lite for the same intelligence

This will soon happen with theoretical physics, computer science, and everything which can be verified cheaply. Then, we will have long running projects augmented by agents for 2-4 years while AI companies are collecting data of human workflows. After that we will see AI being able to do those projects by themselves. This will lead to super fast human progress and cheap products. The price of things will be bound by energy and natural resources. Interesting times are ahead of us

Inference costs will go down massively once they use the upcoming GPUs. I estimated that a model like GLM5.2 will be around 0.03USD/M output tokens in 2 years when the Feynman GPUs will be available in 2028. And this did not even consider architectural efficiency improvements. In mid 2027 we will already see a 10x reduction once everyone has switched to the Ruby architecture.

It will be feasible for everyone to have 20 different agents running at all times. A new world is coming

Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.

If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider.

I estimate that the server consumes probably around 500W during inference.

In Germany where 1kwh cost around 0.3USD, 18k tokens inferred locally would therefore cost 0.15USD which is 30x the costs of using an inference provider.

But for ppl who worry about their data, running locally might still be good. However, they should be aware, that it is much less efficient than using an inference provider.

The efficiency gap will also significantly increase as new GPUs will make inference much more efficient.

EDIT: I first thought it'd be 180k token, but thanks to someone mentioning in the comments, it is 18k. I guess with that, it will be tough unless u got electricity almost for free. Also, the inference providers are probably still using H200/H100 for those small models. Once they use GB300 or next year the new Ruby GPUs, inference will be cheaper by a factor of 30. By then, running local models will mostly be about privacy.

In my opinion Opus is waaayy better in agentic orchestration. It feels like it can natively deal with multiple subagents whereas gpt needs to be taught extensively.

This is way to complex... Why don't just use some harness which manages all that and give u a good UI?

Well, the question is what is contributing to the usage. Because as the context grows, the amount of input tokens are increasing. A model call with 800K token as input is 8 times more expensive than a model call with 100K tokens as input. Especially if we resume a conversation and caching does not hit, it would be very expensive with API pricing.

Did u use the API or subscription?

They will sooner or later change that policy or get very slow in keeping up.

But does it use the same agent harness? Because the harness determines the behavior a lot.