The real prices of frontier models

https://playcode.io/blog/real-price-of-frontier-models

by ianberdin • 9 days ago

157 83 9 days ago

Tokenzier aside, a report shared on reddit found that the GPT 5.6 (edit: 5.5) series are incredibly thrifty with CoTs, resulting in cheaper bills than GLM 5.2 (let alone Opus/Fable): https://www.reddit.com/r/ZaiGLM/s/rUoG5adkPh

Chattiness remains an open issue for some of the SoTA open weights & (to a lesser extent) Claude.

semiquaver3• 9 days ago

That link does not mention 5.6.

SwellJoe• 9 days ago

The fact that OpenAI documents theirs is already a big improvement over Anthropic. But, also, the OpenAI tokenizer got more efficient when they last updated it, rather than less. https://mdstudio.app/o200k-base-tokenizer

alansaber• 9 days ago

Interesting. New models are estimated at ~5T params, so 45,000x increase over BERT base (110m). But vocab size of 200k, so only an increase of 7x over BERT base (30k).

kamranjon• 9 days ago

Where do these estimates come from?

satvikpendem• 9 days ago

Interesting that Grok 4.5 nears or exceeds OpenAI and Anthropic models in some benchmarks at only 1.5 trillion parameters per their announcement post.

vlian2088• 9 days ago

BERT is not an LLM, it's an encoder-only model, i.e. it doesn't generate new text. the first somewhat useful publicly accessible LLM was GPT-3 with 175B params, and it was also the last frontier model whose parameter count was disclosed.

lolinder2• 9 days ago

This piece focuses on the cost differences from the tokenizer, which do matter, but I wish they emphasized more that even adding the tokenizer to your calculation doesn't provide you with a good way to calculate cost for agentic coding tasks.

Other traits where models differ that have an even greater impact on your total spend:

* How much context do they load in to solve a given task?

* How long do they spend thinking to get equivalent results?

* How many times do they stop and ask you for input, and are you there to respond to them before the cache runs out?

* Etc.

Incorporating the tokenizer just makes a very imprecise measurement of cost a little bit more precise, but in my own experience I have not found that the token cost is a significant driver of task cost whether or not you incorporate the tokenizer. Everything else about the model's behavior has a much larger impact.

semiquaver3• 9 days ago

Are there any advantages of the new tokenizer? Does it have a larger or smaller vocabulary or just differently weighted?

ianberdin7• 9 days ago

Well, in my view, it's just the most ordinary manipulation to avoid creating unrest. There is most likely no improvement inside.

Of course, these are my guesses, but did anyone feel the difference in the transition from Opus 4.5 to 4.6? In my opinion, no. And it's unlikely to be a matter of the tokenizer.

semiquaver3• 8 days ago

No offense, but I didn’t ask for uninformed speculation. Why even bother chiming in? I can speculate on my own just fine.

sscaryterry3• 9 days ago

My take from this is that Anthropic is screwing us again. I hope AMD shows them up again.

dgellow2• 9 days ago

Could you elaborate?

sscaryterry3• 9 days ago

If I need to put 2 and 2 together for you: https://en.wikipedia.org/wiki/Shrinkflation

Tiberium• 9 days ago

Yeah, Anthropic's current tokenizer in Sonnet 5/Opus 4.8/Fable 5 is much worse than OpenAI's. Also, OpenAI has been using their current o200k_base from the day GPT-4o came out over two years ago. Just a few of my own tests:

- A ~2000-2002 legacy C++ game codebase at about ~90kloc: GPT 1.12M, Claude 2.2M

- A ~30kloc TypeScript codebase: GPT 260K, Claude 437K

In the end, GPT's current tokenizer is ~1.6x-2x better than Claude's current one, depending on your data. And you can check for free for both, for OpenAI just use the open-source libraries, for Anthropic - you have to use their count_tokens endpoint as they don't publish the tokenizer, but the endpoint is free (and allows requests over 1M tokens as well).

onlyrealcuzzo• 9 days ago

Interesting... Naively I'd assume you'd have a pretty unfair advantage on quality if you have materially more information dense tokens.

That doesn't really appear to be the case as GPT and Anthropic models appear evenly matched despite Anthropic encoding the same text into almost ~2x the tokens...

I'd also - naively - assume this would make training their models more expensive. Though inference now dominates, and they'd probably rather have more tokens than less (to charge you for them at future 80% margins).

not-a-llm2• 9 days ago

more dense tokens means more stuff to fit into the embedding space which is per token, so more work to disentangle later

recursivecaveat• 9 days ago

If a given paragraph gets encoded into twice as many tokens, that means the model gets twice as many matmuls to process it. The amount of compute thrown at the problem is increased (everything else constant), which may improve the quality of the result. This is believed to be one of the reasons that 'thinking' tokens improve quality. For long tasks it will lead to more context compactions though which will harm the quality to some degree as well.

not-a-llm2• 9 days ago

you use the wrong word

the Anthropic tokenizer is not worse, its more expensive/verbose

nullsanity• 9 days ago

So, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.

diwank• 9 days ago

this is surprisingly high delta. to make matters worse, reasoning tokens account for the majority of tokens and they are completely opaque so it's hard to tell how much of that is prose or code

f311a• 9 days ago

The real price is how often a model uses subagents that scan your whole repository multiplied by thinking level.

jnwatson• 9 days ago

The real elephant in the room is pricing for KV cache writes and reads. That makes all the difference for tasks with large context.

petercooper• 9 days ago

Cache writes/reads are the majority of the picture for agentic development. When you see these people saying a coding run "used 200 million tokens" or whatever, most of that is cache reads, so it should be the headline price IMO (and is one reason why DeepSeek API is so striking with its minuscule cache pricing).

ricardobeat• 9 days ago

I have reduced usage of Fable and Sonnet 5 to a minimum. Fable in particular is amazing at creative tasks, but not worth the cost for almost everything else. I can have Opus 4.6/4.7 running non-stop without hitting quota, vs maybe 20 minutes of Fable usage.

ianberdin7• 9 days ago

Well, I both agree and disagree with you.

On one hand, the price is just astronomical for Fable, well, not exactly astronomical, but I would say unaffordable. That is to say, so expensive that it is impossible to use.

But on the other hand, Fable is simply incomparable to anything else. I mean, it is just amazing. There is nothing even close to being equal to it.

gervwyk• 9 days ago

I have to agree on fable being in another league. I’ve given it my most challenging problems and almost always comes back with a functional, solution 2-5 prompts away from a finished pr. Literally smashed our backlog - very impressed. What i found most efficient is to add “use sonnet agents for research” gets you really far, and on large not so novel tasks “use opus for tasks” by adding this it spins up many agents, works for 2+ hours in a usage window and completes A LOT of work.

colechristensen• 9 days ago

Fable can solve the problems Opus couldn't. BUT most of the time I'm not having those kinds of problems.

I wouldn't say I'm doing anything groundbreaking but definitely at times obscure and that's when Fable has been able to dig me out of the rut. (the alternative I was actually following was reading textbooks myself to understand the domain better)

lolinder2• 9 days ago

The weird twist here is that I've found that there are times where Fable is actually worse than Opus at a straightforward task, not just more expensive. It'll launch off into its own little world for an extended thinking, then start editing files, and I'll already have spent $5 in tokens before I see enough response to know that it's on the wrong track.

Opus's verbosity is actually a boon sometimes for catching false starts early.

arjie• 9 days ago

After getting used to Opus 4.8, Sonnet 5 is nigh unusable for coding. I much prefer Opus 4.8 + DS V4F for routing. Sonnet 5 is just not useful in the price/performance anywhere. I still get some use of Fable for planning because it can comprehend agent-built codebases (which have repetition of local patterns to a grand degree).

j452• 9 days ago

An individual token, and the level of energy it represents (electricity, or relative effectiveness per model) increasingly seems the space of obsfucation.

This space can be increasingly avoided by becoming, and remaining, efficient and effective with prompts.

dinkleberg• 9 days ago

That is one of the interesting things about Neuralwatt cloud. Their pricing is based on energy rather than tokens (actually they have a token-based alternative, but claim the energy pricing results in 95% cheaper results). I've tried out their subscription offer and it does seem like you get a lot more usage even on the cheap plan. However since the energy metering is pretty much unique to them (at least that I've seen), there really isn't anything to compare it against, so hard to tell how accurate it is.

Regardless, it is cool to be able to contextualize the actual spend in terms of physical energy utilization. It even has a little co2 number (though again, kind of a "trust me bro" metric).

j452• 9 days ago

I think I saw something like that, and it made me realize the watt isn't the only measurement. The watt can be spent effectively, or ineffectively, thus obsfucating mileage per watt.

How we drive AI will cause mileage variances, but over time the improved practices can measurably change.

scottcha• 9 days ago

Hi, I'm co-founder of Neuralwatt. While there aren't other providers selling by energy we also produce the same tokens stats you get at others (input, output, cached) and you can compare using those. The CO2 number is really just multiplying the energy by the CO2-intensity at the time/place we served the request. So everything is actually measured and can be derived externally.

The main observation when you compare to token prices our input/cached energy is much lower than the equivalent token prices while the output energy is generally higher. One of the reasons this is a bit cheaper, especially for agentic systems, is that in an fully built/cached session your input & cached to output ratio is really massive input to very small output (multiple orders of magnitude usually). The other thing we do is we actively work to optimize the system around tokens/joule with the goal of making the most energy-optimal system.

Happy to answer any other questions.

drob518• 9 days ago

Yes, and verbosity (thinking) is a huge factor.

daft_pink• 9 days ago

Honestly, I find performant pricing where they test each model on the same task is much more useful than figuring out the tokens or using the input token tax.

The reason being is that the only tokens I feel I really control are the input tokens, but the whole program seems to just run itself and they just charge you what they want to charge you and it’s more of a black box.

Very interesting article though.

ianberdin7• 9 days ago

We've switched the default model in playcode.io among Opus 4.8, Opus 4.6, Sonnet 4.6, and Sonnet 5. I must admit, Opus 4.8 is quite expensive, and the costs accumulate quickly. Opus 4.6 is about 50% cheaper, while Sonnet 5 is significantly more affordable. According to the data, Sonnet 5 is about 2-3 times cheaper. Fable 5 is unaffordable at all...

Today, I tested Sol 5.6 on various tasks. It performs similarly to Opus 4.8 but is still noticeably more expensive than Sonnet 5. Although Sonnet 5 isn't the top model, it's quite effective for creating typical websites for small and medium businesses. However, they will increase the price starting September 1, as their free offer is ending.

I'm also actively testing Grok 4.5. There's something promising about it. The design is mediocre, in my opinion, but it operates quickly and reliably without any deadloops. Usually, Grok models would fail or loop, but this one is stable.

Overall, I really want a benchmark based on real tasks.

yboris• 9 days ago

Meta: why are the scrollbars hidden / invisible? Awful UX :(

karma_daemon• 9 days ago

The "Honest size of the effect"

Honestly, haha

foota• 9 days ago

It's a bit unrelated, but I've been wondering if LLM providers are using cache read costs to preserve the illusion of constant output token prices. In reality, the longer your context the more expensive output tokens are, but anthropic and openai both have flat output token pricing.

In practice though as a result of cache reads over multiple turns you will end up paying quadratic pricing anyway.

davidw• 9 days ago

Where is it getting more tokens in the same source file? Like if you kept minimizing what you're feeding it, what accounts for the difference?

robbie-c2• 9 days ago

Is it on topic to complain about the various claude-isms in this article? I don't know any actual humans that write titles like "Two floors the rate card hides".

I find my brain disengages once I suspect something of being written by an LLM. If the author didn't put much effort into writing it, should I expect them to have put much effort into fact-checking it?

Edit: this specific title has been deleted from the article. That was not my point! Please put in more effort into writing things that you want others to read! Rather than putting in low effort but being better at hiding it.

add-sub-mul-div• 9 days ago

Yes it's worth commenting about. Not everyone, but many want to know that signal just like you do. It not only provides a useful heuristic about the article, but about whatever product or service they're advertising/selling.

mft_• 9 days ago

"No generation, no estimates - just token counts:"

"Authoritative: it is the same count Anthropic bills against."

"This reframes a headline that looked like good news."

stingraycharles• 9 days ago

Yeah this is an extremely poorly written article. They didn’t even bother to add a “rewrite this article to make it sound less than AI”.

It’s also using a bazillion words to make a point that could be summed up in a single paragraph: there’s a huge variance in the number of tokens required to encode the same content, with code leading the charts.

To be fair, most of this was already known, and Anthropic communicated very clearly about the different tokenizer they started using.

Their compute is also mostly 1:1 correlated to the number of tokens, so I don’t believe in the conspiracy that this is just to inflate prices.

iainmerrick• 9 days ago

I’m normally one to complain about people complaining about these LLM-isms, but yeah, this one really grates on you.

It’s a shame because it’s making an excellent point! It just takes so long to get to the point that the reader loses the will to live.

Yes, I could probably ask an LLM to summarise it for me. No, I’m not going to. I would prefer the author just take care of that for me.

sisve• 9 days ago

Yeah. To often now a days there are articles with really good points, but they are just so verbose and clearly AI generated.

I could live with ai content if it was short and to the point. But it's always so lengthy. Hope that will change.

A tl;dr section at the top and then the long read from ai could also be OK if they marked it.

SecretDreams• 9 days ago

that the reader loses the will to live.

My general vibe hearing about AI.

phist_mcgee• 9 days ago

If you couldn't bother writing it, I can't be bothered reading it.

esafak• 9 days ago

We need to make attribution standard. It's a lie to pretend you wrote something you merely prompted.

jascha_eng2• 9 days ago

Aside of the claudeisms and the obvious AI smell, it overexplains everything and doesn't come to any useful conclusions. It's just not a good post.

The nudge to think about both "tokenization as variable" as well as actual tokens consumed per task is still good.

conception• 9 days ago

A problem is AI by default is not very good at anything. It’s pretty mediocre. With a good harness and a lot of prompting/context - you can get it to spit something out that’s pretty good. Coders have been learning and fighting this fight for a couple of years now.

The issue is that it’s not just code - they suck at writing. Really bad. Unreadable, incoherent, messy.

Humans are also bad at judging the quality of things they themselves aren’t very good at. So a senior swe sees what claude spits out and says “This is trash.” And spends x amount of time getting it to not be trash. And Jr dev thinks “this is magic!” And pushes it to a PR.

So my theory is the people “writing” this AI slop think its great! But actually just aren’t very good at writing copy and don’t have the skill to recognize it and prompt their way out of it.

Or they don’t care. That’s an option as well.

PS for anyone reading, next time AI does something that you aren’t super familiar with that looks pretty good… maybe find an expert to review it.

ianberdin7• 9 days ago

Well, criticizing is, of course, great. But the reality is that English is not my native language and I dictated most of it with my voice, then processed it with the help of AI, translated, added, corrected, and converted.

It is actually a big result of work, a lot of research and attempts. And to just say that "oh, this is AI-slop," I consider unfair, but that is your choice.

There is a difference: - There are people who do, - And there are those who criticize.

smusamashah• 9 days ago

Instead of getting offended by a fair criticism you should learn from it. In your articles consider adding a disclaimer that says exactly what you just said here in your comment here that you post-processed your voice and thoughts through LLM.

LLM speak is like the new corporate speak. Enterprise writing is fulll of fluff and nothings and they all read the same. That sameness is what most readers here are sick of.

(Your comment here that I replied to is also written by AI which is even more sad :| )

dolebirchwood• 9 days ago

There is a difference: - There are people who do, - And there are those who criticize.

Weird shield to hide behind, considering there are also people who can do both.

robbie-c2• 9 days ago

There are people who do, - And there are those who criticize.

There’s no need for that. I’ve done plenty.

d1l• 9 days ago

Bro, this was the only reason I opened the comments, and I'm so happy to see someone else noticed. Even if they removed that, the utter sloppiness of the prose is unbelievable. Offensive, even.

ianberdin7• 9 days ago

Thank you for the criticism. I heard you. I added a TLDR. I cleaned up many AI constructions. By the way, I tweaked it a bit, compressed it.

One way or another, I want to note that yes, this text was made in collaboration with AI. My English is non-native. It helps me translate, helps me structure better. Yes, there is a downside, it can bloat the text with unnecessary words. But that, unfortunately, is the price.

But the key thing is that I tried very hard to share my many years of experience, or rather a part of it, which I acquired, with all of you. And I am very glad that this information turned out to be useful to you.

The key here is: * The information that is written in the article. * Not how it is written, but what I was trying to convey to you.

Thank you very much for reading and responding.

zaptheimpaler• 9 days ago

Wow this is great, it explains what I've seen. My Codex sub seems to last way longer than Claude in real codebases, and Claude eats up a ton of context in the initial read. I thought the harness might be the cause but it seems like tokenization is probably the bulk of it.

dinobones• 9 days ago

Just start pricing in bytes input/output. This whole "token" and "tokenizer" thing is an implementation detail that shouldn't even be leaking out into the API.

Providers change tokenizers all the time with model updates, and it's often not even possible to query/figure out how text is tokenized without actually just sending the LLM a request.

Just switch to charging for bytes of intelligence. Please. Claude Shannon figured this out decades ago.

golem14• 9 days ago

That does not help you compare pricing between models since they can vary by (even internal, thinking) output size.

The best way to measure is really the end-2-end cost, price per task.

iLoveOncall3• 9 days ago

Very unpalatable completely LLM-written article, but on top of that a lot of the fundations and conclusions are completely wrong, the main one being this one:

You will see people claim Claude uses 2x to 4x the tokens of GPT. Our measurements do not support that, and overstating it would undercut the real point.

It's not because a single prompt represents only 1.7x the number of tokens that a model doesn't use 4x as many tokens as another, when running as an agent. This doesn't take at all the number of tokens of the output into account, and the number of tokens of the potential tool calls from this output, which directly feeds back to input tokens.

The article also has a very small test set (16 documents), all of very small length (15K tokens at most, when models go up to 1M in context and agents routinely exceed this and have to summarize).

Complete garbage article.

ac29• 9 days ago

You will see people claim Claude uses 2x to 4x the tokens of GPT.

My guess is something to this effect was in the prompt and the LLM made a point of correcting it

ianberdin7• 9 days ago

Just checked some of your comments. I see you mostly talk about how everything is AI written.

iLoveOncall3• 9 days ago

No I don't. But you seem butthurt. You sure are a great CEO.

socketcluster2• 9 days ago

It's a terrible metric, it's kind of like companies paying employees by the hour for white collar work.

Most people here probably don't know what it was like to work a contract job and being paid based on actual deliverables.

The incentive of AI companies is to create as many tokens as possible to solve any given problem. Just like your incentive as a software engineer is to create as much complexity as possible in order to use up as many hours as possible.

This is why big tech companies have millions of lines of code... They've got thousands of engineers rapidly churning out tokens.

The difference in number of tokens I use in my day job vs side projects is massive. You can see the inefficiency quantified.

Show me the incentive, I show you the result.

NkVczPkybiXICG• 9 days ago

Big tech companies don’t usually pay hourly.

ButlerianJihad• 9 days ago

As a contractor, I was paid hourly all the time. At some points I needed to fill out an actual time sheet each week, that I would submit to my own company, and they used these time sheets to bill the clients.

So yes, "big tech companies" often paid hourly, even if that pay was indirect, to contractors and job shoppers and people who were not direct hires.

socketcluster2• 3 days ago

Yes they do. The contract says $X and Y stock options per year, Z hours per week.

luciana1u• 9 days ago

Anthropic's tokenizer being 2x less efficient means they're essentially charging you for whitespace. premium whitespace, mind you — each space character gets its own attention head.

MPSimmons2• 9 days ago

I don't know if it's fair to say that a tokenizer is being less efficient if it generates more tokens per text. I think it's more fair to say the tokenizer is more nuanced. The question is whether the additional nuance permits better model output, which could justify the additional token cost in inference.

charcircuit• 9 days ago

This article doesn't address the inference side cost. Not all tokens cost the same. If the response is predictable you get a few output tokens for for the price of 1. The further an output token is the cost of generating it grows linearly due to attention.

ianberdin7• 9 days ago

@dang, why is this flagged?

lysium• 9 days ago

I was wondering, too. Maybe this explains it? https://news.ycombinator.com/item?id=48899135

iLoveOncall3• 9 days ago

Because it's low quality, entirely written by an LLM, and also plain wrong?

wgd• 9 days ago

DeepSeek and GLM are left out of the tables entirely: we only have rough characters-divided-by-four estimates for them, not real tokenizer counts, and this post is about measured numbers.

lolwut. The open-weight models are inscrutable black boxes for which we can't possibly get real token counts? Typical lazy clanker, BSing their way out of doing the whole job.

The real prices of frontier models

Related Stories