cpard

This was mostly because Sonnet 5 worked longer and read more to get there, consuming 1.9x more tokens.

I have experienced similar behavior between opus and haiku when benchmarking Dara engineering tasks. The “cheaper” model takes many more turns to figure out the task and this is without taking into account other important factors.

Another interesting behavior that I observed is that Haiku tended to cheat more maybe because it was having a harder time to find the root cause of the problem.

Benchmarking and evaluation of agentic systems is very interesting and if there’s one thing that someone should keep from the Databricks post is how important is for everyone to build and run their own.

There’s an emerging pattern in agentic systems and this project is a great example.

A deterministic layer like a compiler or generator of code with some kind of IR that the LLM generates and feeds it with.

I feel we will be seeing this more and more in the near future.

Human mathematicians could become “priests to oracles.”

Priests were interpreting the oracles (at least at a place like Delphi) according to the context of the people asking the questions aka participating in politics of that ancient times.

Subjectivity was a feature and I’m not sure that fits to mathematics though.

I wonder if mathematics as a science field moves more into engineering or if a different branch will emerge that is closer to that because to the point of the article, science is about understanding not just results.

Most framework vendors don’t have an incentive to make things less obscure. The agent framework is free/open source and they make money primarily from selling observability products for agents. Even if they don’t intentionally obscure things, they just don’t have the motivation to optimize that part.

Do you enumerate the options of the algorithms to the models? I've tried to do "algorithmic discovery" with these systems, e.g. openevolve, and to be honest the models didn't really focus on that part.

Instead they were focusing more on optimizations of the existing algorithm that has been implemented. Maybe it's an artifact of the problem I was throwing to them (I was asking to optimize the implementation of select_k in Arrow, which is currently using a max-heap streaming algorithm).

I've started documenting my journey with this here: https://www.kostasp.net/posts/16-ai-experiments-apache-arrow in case you want to take a look. Any advice would be highly appreciated, I'm looking for more inspiration on how to torture myself with that stuff.

I'm personally interested in this problem and it's a quite active research area right now.

My feeling is that the research is converging to what the paper claims, that the combination of two is the right way to do it and it's a matter of how you combine the two as part of the harness you built that makes the difference.

At the AID-Wild / ACM CAIS 2026 workshop that happened recently, there are plenty of examples in the accepted papers on that.

A great example is AI-PROPELLER: Warehouse-Scale Interprocedural Code Layout Optimization with AlphaEvolve. It uses AlphaEvolve and Vizier to evolve compiler code-layout heuristics. (https://arxiv.org/abs/2606.00131)

but is it worth the effort from a PR perspective for them? I guess we will have to wait and see.

Row level and summary stats are both diffs over values that can tell you that something changed but not whether the * meaning * has changed. What I'm working on is providing more information on how the meaning changes.

What questions I'd like to answer with the diffing is more like: will the grain go from one-row-per-user to one-row-per-user-per-day, will a key stop being unique, will a join start fanning out and quietly double a measure, will something additive become non-additive.

This diff is over structure but this structure is latent in the transformation that produces it and to make things harder, if we are talking about some declarative language being used (e.g. SQL) the code doesn't even describe how things are getting done, but what the output would be.

What I've ended up doing is recovering the structure from the code by analyzing it and then using * cheap * profiling than a full row compare.

As an example, my equivalent impact sub-command output would be something like this: "this change makes account_id non-unique three models downstream"

This is really neat. I’m working on something similar but for data artifacts not just code. It’s very encouraging to see that this kind of tooling helps both humans and models, that was what made me starting to work on that.

Curious to see when a post from OpenAI will appear with the corrected theory or something. This seems to be an ideal scenario for them to go after another scientific case. They have the theory, they have the experimental proof that it’s wrong, exactly what you need for an agentic loop to do its work.

Or maybe what works in math doesn’t work with chemistry?

I don’t think the flex here is the amount of code alone. Their goal is to show that AI can improve productivity, the number of lines is just the proxy to that. This article is a marketing piece after all.

Now someone can argue that lines of code are not a good proxy of engineering productivity, but I wouldn’t be surprised if the audience they target with this content is not the HN commenters of this thread.

Thank you!

* …tools, UMP does for memory - negotiated operations over a portable, signed, bi-temporal record … *

What is a bi-temporal record? I don’t think I’ve heard the term before and I’d love to learn more.

It’s clear that Anthropic is building harnesses for specific use cases now and turns them into products.

This is the equivalent of Claude Design but for security.

Different harness, different packaging and obviously different distribution because the persona is different.

It’s funny because from all the posts I’ve read from companies reporting on Mythos, everyone is building their own harness for it.

Cisco even published a specification for one.

But Anthropic is the one who has figured out how to package and distribute this. Great GTM!

I'm more curious about going from text to Prela instead of going from text to SQL and measuring any difference in the performance there. On one hand models have been trained on a lot of SQL on the other hand they are really good in mathematical reasoning too so thinking in Perla might be a natural fit for them.

SQL, JS, Excel are really hard to substitute because of how widely used they are by people. Even if something new comes up that it's objectively better, so far has always failed gaining traction because of this reality.

I wonder though, is such a dialect better for agents? Have you tried to measure if an agent performs better expressing queries in such a language instead of SQL?

I think I didn’t articulate myself very well on my reply. I actually wanted to say that I agree with you and emphasise again the need for educating users for the complexity of these projects.

What you describe has been pitched by many different products for different parts of the data platform. Fivetran for example claims to do that for the extraction and loading part, good old Informatica was offering the ETL in a graphical interface etc.

The problem that many teams ended up having is the explosion of the tooling needed by data teams.

Of course it is. What you describe is one of the reasons that ELT became popular, if you couple it with a variant type and schema on read, you have a very powerful and flexible architecture.

But there’s no free lunch, building and maintains data infrastructure that is reliable requires work. Many companies don’t realise that when they start their analytical journey and aggressive marketing doesn’t help. That’s the point I was trying to make.

Replicating the Postgres WAL to S3 and Iceberg reliably is a hard problem but it’s not accurate to say that no ETL is needed here.

maybe you can say it’s more of an ELT pattern but anyone who’s interested into using this for realistic analytics they will have to transform the data at some point.

If an org is early enough to think that they can use a solution like this and just get in duckdb and start spitting out reports, they will be up for a really bad experience.

Please educate people to do the right thing and realize the scope of the work they are facing, it might feel that it hurts your growth in the short term but it will benefit you greatly in the mid-long term as a vendor.

The comments are definitely not worth reading. It’s a very sad thread, you literally had to go through all of them to find one that wasn’t about hate and stating some facts about the issues of the code.

This is a good article and hopefully it served its purpose of generating awareness for the platform but I have to disagree with the core claim of the article.

Distribution was always the hardest part in building a business, sure today it’s easier to vibe code something in a day or two but there is a reason that for a long time now the standard advice to technical founders has been to not build before you have conviction on the product market fit.

And product market fit means that you have also figured out how to get to the market, aka distribution.

This article is very well written and you can enjoy it regardless you agree or not.

It made me think, considering how much content is being written with AI, would an LLM ever write something like this?

For some reason they I can’t explain, the article smells human and I don’t use the word smell as a bad thing.

I guess, in your case the needle was there so it moved.

Quake Arena and LAN parties during college created some of the best memories I have related to computer games.

Hopefully I didn’t sound too critical of the post because this wasn’t my intention. The post delivered what was needed and thank you for this!

The reason I asked the question is because in the case we don’t need the rest, it would be better to not use this terminology for these systems. We already anthropomorphize LLMs too much and although I get the marketing value of that, it’s not always to the benefit of the people who interact with them.

Please do write the rest of the posts!

This is a great post and I really appreciate making the cognitive science terminology clear.

the author is doing a great job telling what is missing from the current memory frameworks for agents but what is missing in my opinion is also an argument about the necessity or not of these missing components.

* Large language models amplify inconsistent technology and quietly reinforce consistent ones. *

This is another way of saying that the tools you equip the LLM affect their effectiveness, in other words, the harness you build around them matters and matters a lot.

At the end of the day, the language you pick, enriches the harness with the toolchain, libraries etc. it offers. This is most evident with the toolchain as the author mentioned but if you think about it, picking a specific framework that constraints the choices the model can make (e.g. the Ruby on Rails example) is also affecting the behavior it has.

My understanding so far is that that Mythos (and any model in general) can produce candidate reasoning but you really need a system around that reasoning that is capable of producing auditable security findings.

So, success is coming not just from the model but also from the harnesses they built around it. The Cloudflare post was more detailed on that front and I wish the rest would share more about it.

The Cisco spec is interesting too, it pretty much describes an architecture of a harness: https://github.com/CiscoDevNet/foundry-security-spec

UPDATE: Launch was a success! 400K+ views, and multiple companies reached to use my IP. Read more here

It seems that he managed to get what he wanted from the hardware and I'm happy for them.

He said something interesting at the beginning of his post, he compared the cost of the hardware to the cost of his time based on his FAANG salary. Which is an interesting way to think of this, but the rest of the article didn't make me understand if at the end he did save money/time based compared to just rend on the cloud.

Also, outside of the power cost, hardware has other costs too, you need to operate it, maintain it, set it up, etc. all that require time. I mean, even the process of figuring out if it had a good enough ROI compared to cloud, takes from your time (collecting data, analyzing data, etc etc).

You are making a great point here. I think it’s not just the amount of information and complexity of the domains today, it’s also human nature and emerging politics too.