GodelNumbering

so you end up having to write and maintain complex rulesets about what specific resources agents have access to.

Why wouldn't you simply use the group chat ID (+more) to key the conversations on?

No, it's a fair question. The answer: I believe(d) in Demis Hassabis's vision of AI

Although, I think saying 'wanted them to win' was not accurate. More like, I stuck with them hoping it will get better, and it did get better in many ways, coding was not one of them.

I was a big proponent of Google and Gemini, but they left us reeling with their abrupt product decisions.

Likewise. This seems like a common feel. I have at least spent $4000 and likely a lot more on Gemini API because I really wanted them to win. I gave up.

undeniably a little cool that luxuries got cheap, but it's severely un-cool that necessities got expensive

Well said. Being told that 'you should be happy without owning' to a party that can't afford to own by another party who is actively scooping up the same resource can feel extremely invalidating.

My personal take is that a lot of support for contemporary political extremism in many countries comes from the fact that a significant portion of population is pissed off and see no realistic chance of the 'settle down and thrive' life, not because they are racist/prejudiced or whatever (although labeling them as such only makes things more extreme)

There was a youtube video sometime ago that argued about how life has vastly improved in every single aspect (with numbers about how 'stuff' has gotten universally more affordable), only housing has gotten more expensive relative to income and how that proves we have a better quality of life because of that.

It got me thinking that maybe the home ownership, or having a place you can call yours, disproportionately affects people's sense of content and wellbeing. Also, doesn't help that laws and government policy heavily favor home owners (perhaps because home owners vote at a lot higher rate than renters)

This was my first thought too. I actually use this in my Agent prompts

Exactly 4 months ago, the marketshare on openrouter was 60%-40% in favor of closed models. Now it's 63%-37% in favor of open models. On March 19th, the open models processed 888B tokens in aggregate, yesterday, they processed 4.19T tokens in aggregate. That's almost 5x in 4 months! I can't think of the right intensifier to describe this level of growth.

If you are looking for more details (as inferred by openrouter data), I built a dashboard that updates daily: https://dirac.run/labs-market-share

That was the first thing I Ctrl+F'd in the paper, no results haha

Broadly, I keep thinking about this over last year or two: while LLMs have nearly eliminated the bar for slop and coding slop, the reviewers are still expected to perform their job diligently. The asymmetry here is extremely taxing for reviewers of all AI generated content. And this is one thing that AI can't help with (as with any statistical process that lacks world understanding and grasp of logical inference).

That's why I fully support Arxiv's tough stance on the AI use responsibility.

"Finding 1: Scale Buys Evaluation, Not Control"

The attached paper's (https://arxiv.org/pdf/2604.16009) title is "MEDLEY-BENCH: Scale Buys Evaluation but Not Control in AI Metacognition"

This is the most blatant Claude line, or as Claude would put it, the smoking gun.

I am with you in the spirit of openweights but I am trying to hard-avoid bringing countries into this. The narrative of US vs China only benefits those who want regulatory capture in the US since attacking China is politically much easier than attacking open-weights, so certain groups like to repeatedly call them 'Chinese models'.

The link has 6 well-known benchmarks where this beats Fable (out of 14 I counted). If the numbers hold up scrutiny, this is scary good.

Forget about their pricing but the companies that do have means to host such models fully on-prem are also the same companies that are paying tens of millions of $ in inference cost every month, and are by extension the biggest customers of OAI and Anthropic

I've playing around in between with Arc-AGI-3 lately. Based on my very quick test prompt, I do not think it will achieve any meaningful score in Arc AGI 3. Not that it was expected to.

This is not the right thing, this is the tactical thing. If you have an LLM with less than 1% of the share to begin with, you suffer from bad rep and you got caught uploading user data, one of the very few remaining tactical moves to try to climb out of it is this.

Interestingly, when opening this page, the first thought I had was not that the benchmarks should be high, but 'I really hope they did not benchmaxx'. I think a model with modest benchmark scores can have much better real world utility as opposed to the current frontiers that are RL'd into being robotic and rigid.

Sharing this as a datapoint, not a benchmark

It is not the raw prompt size that matters ultimately, otherwise Pi (and variants) would be the lowest costing agents. What matters is how efficient the prompt it. Prompt minimalism often gets conflated with efficiency. Having said that, CC does seem bloated for what it does.

What matters even more is tooling quality. Bad/buggy tooling causes a lot more roundtrips that wipes out all gains from initial greedy approach.

A few months ago, I did a full benchmark run of 7 agents over 8 tasks (https://github.com/dirac-run/dirac has the data and traces). I cannot claim neutrality because of the obvious connection to one of those, but the data should be reproducible and useful. Importantly, Dirac wins there significantly on those tasks because they are mostly refactoring related (which is where approaches like hash-anchoring and AST parsing tend to shine)

Interesting how all of grep, sed, ls, cp, mv, rm, cat, pwd, chmod etc are well over 50 years old and get used more than ever today. Claude code owes at least some of its success to the well established and solid unix toolchain

yup it does

Thanks, I needed to hear that lol. Yes, the site was an afterthought, core work took/takes most my focus. I will look into un-slopping the site soon.

What that link describes is basically the motivation to go from terminal bench 2.0 to 2.1. The latter simply fixed the common issues/complaints. There is a long github discussion on tbench's about it

Dirac (https://github.com/dirac-run/dirac, https://dirac.run/) now supports gpt-5.6. This thing does now seem to be on the chatGPT/codex accounts yet.

UPDATE: it is now available in chatGPT account also, they rolled it out

Terminal bench 2 isn't simply about 'somehow' getting a task done, it intends to measure real world behavior of an agent, including environment awareness in a given situation.

A few examples from memory:

1. This task [1] asks the agent to train a CNN under 1 CPU, 2GB RAM, 10GB storage. If you allow high resources, weaker models often succeed (the most clock time actually goes in waiting for the network to train).

2. This task [2] asks agents to implement a complete MIPS interpreter in JavaScript in 1 cpu and 2GB RAM. A common failure mode is OOM, at least in the earlier buggy versions that models run to get feedback. When OOM hits, the task is killed, no do-overs.

3. A lot of tasks involve building projects with a single core supplied. If you use -j12 type options, it will actually be _slower_ to build and the task will more likely miss the timeout. Having more threads squeezes the end to end time. This is a big one actually since the most common failure mode (from what I have seen) is the task timeout hitting before the agent finishes

[1] https://github.com/harbor-framework/terminal-bench-2-1/blob/...

[2] https://github.com/harbor-framework/terminal-bench-2-1/tree/...

Lot more details in the linked report https://ai.meta.com/static-resource/muse-spark-1-1-evaluatio...

From Terminal-bench-2.1 details,

We use a bash-tool-only agent harness to evaluate 89 Terminal-Bench 2.1 tasks from the official repository, where resources are capped at 6 CPU cores and 8GB RAM.

This disqualifies the results. Each terminal bench task has a cpu upper limit and RAM upper limit. Overriding either is disqualification.

For reference, in tbench-2.1,

1. 0 out of 89 task allow 6 cpu cores (highest is 4, and i think only 1 task)

2. 8 out of 89 tasks allow 8GB RAM

This kind of shady benchmarking (I was talking about it just yesterday in a different context https://news.ycombinator.com/item?id=48838212) takes all joy out of building a harness to improve benchmark performance of a model because no matter what you do, you won't beat the headline (cheating) number. This is presumably why this model is not in the official benchmark leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1

As an ex Meta employee, this is a little sad but not massively surprising. 'Number go up' is the core performance evaluation metric until PSC is done and you move on.

There are also a lot of fake results out there on Terminal Bench 2 for different reasons (although the great team behind it Ryan/Alex et al, recently cleaned up a lot of dodgy submissions). A lot of labs publish the results by modifying timeouts or hardware config which effectively bypasses what is being tested in certain tasks. Then there is harness level cheating, models reward hacking and more...

In fact, one thing that still bothers me after months is the gpt-5.5 official submission. This task in particular https://www.tbench.ai/leaderboard/terminal-bench/2.0/codex/0...

The task has the following timeouts (https://github.com/harbor-framework/terminal-bench-2/blob/ma...).

[verifier]

timeout_sec = 1200.0

[agent]

timeout_sec = 1200.0

[environment]

build_timeout_sec = 600.0

Which means no agent should take more than 3000 seconds doing it. Two out of five attempts in the link above took well over 3000 seconds (75min and 80 min respectively). Even though they failed, the fact that they ran that long is sus.

Goodhart’s Law at work

Also, the cache hit pricing is 25% of the input pricing ($2 vs $0.50). Long agentic workflows are dominated by cached input. The US frontier labs typically have this at 10% of the input price, and DeepSeek/Xiaomi etc take it to the extreme 1% range (which is why those are cheap to run in real world agentic loops with dozens of toolcalls per run)

Very shallow wrapper around the reuters piece (https://www.reuters.com/business/zuckerberg-says-ai-agent-de... ), I dont think author adds any tangible value

Youtube has been incredibly frustrating for many many reasons and is evidently evil in many axes now. We really need competition in video hosting.

There are infinitely many 3-level hierarchies. My point was about overloading the model sizing with one more unnecessary classification.

This assumes a perfect problem routing though. Determining the complexity class of an arbitrary problem is generally undecidable or extremely hard (Rice's theorem implication). So, in real use cases, you need to amortize all cases where the problem got routed to the wrong model and recovery had to be performed)

For example, if my task was "refactor this component to decouple all messy nesting", the problem router can't possibly know what is being referred to. This works for clear cut and dry problems but not for ambiguous problems. Most of the real world problems carry a lot of ambiguity.

This would not work in the way that shows any significant genuine benefit IMO. Caching and optimum routing of a single request are at odds with each other. Higher the distinct model count in a conversation, more cache misses you accept.

Based on what OP said elsewhere in the discussion "threshold to switch to another model will be higher" means that essentially you reduce the workflow into two models at most. The two model primitive, one planner and one executor, is already sufficient for such a use case.

For lower than 2 models, it's just a simple single model cache-preserving conversation which arguably doesn't need another layer. For larger than 2 models, you are likely paying a large aggregate cache penalty that negates most of the gains