gronky_

Mobile rendering of the post has some issues. Tables are overflowing and not responsive for example

Don’t know about other claws, with NanoClaw the agent can only rewrite code that runs inside the container.

You can see here that it’s only given write access to specific directories: https://github.com/qwibitai/nanoclaw/blob/8f91d3be576b830081...

True. There’s a setting for Claude code though where you can add apiKeyHelper which is a script you add that gets the token for Claude Code. I imagine you can use that but haven’t quite figured out how to wire it up

Keep in mind that this isn’t about users - the top agents on the leaderboard aren’t running an actual product on the benchmark.

If they are running their production product as is, then of course whatever is built into the product is fine.

This ok from your perspective then?

def make_pass@1_agent(agent, n):

    def retry_agent(problem):

        for attempt in range(n):

            result = agent(problem)

            if result.success:

                return result

        return result

    return retry_agent

It’s a pass@1 benchmark. When submitting you need to check a box that there was only 1 attempt per problem. See here for example: https://github.com/SWE-bench/experiments/pull/219

Building multiple attempts into your agent is stretching the rules, even if technically it’s acceptable

I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance.

71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running.

But the top rated submissions aren’t running production products. They generally have extensive scaffolding or harnesses that were built *specifically for SWE bench*, which kind of defeats the whole purpose of the benchmark.

Take for example Refact which is at #2 with 74.4%, they built a 2k lines of code framework around their agent specifically for SWE bench (https://github.com/smallcloudai/refact-bench/). It’s pretty elaborate, orchestrating multiple agents, with a debug agent that kicks in if the main agent fails. The debug agent analyzes the failure and gives insights to the main agent which tries again, so it’s effectively multiple attempts per problem.

If the results can be reproduced “out-of-the-box” with their coding agent like they claim, it puts it up there as one of the top 2-3 CLI agents available right now.

I see it a bit differently - LLMs are an incredible innovation but it’s hard to do anything useful with them without the right wrapper.

A good wrapper has deep domain knowledge baked into it, combined with automation and expert use of the LLM.

It maybe isn’t super innovative but it’s a bit of an art form and unlocks the utility of the underlying LLM

It will catch those sneaky bugs

Stating that Israel doesn’t have a right to exist has been recognized to be an antisemitic statement by many prominent institutions.

It’s a radical statement that effectively denies the rights of millions of people to exist and is especially problematic given the historical context of the establishment of Israel.

The statement gets thrown around so much in certain circles that it’s gotten normalized. You’ve apparently lost sight of or never stopped to think what actually means, to the point where you’re providing it as an example of an innocent statement that got you banned for no reason. Taking this statement out of radical activist circles and into the real world won’t go well.

Take some time to educate yourself and reflect on what it actually means.

Yes, it uses Supabase, doesn’t roll its own

I just tried the demo on the homepage and I don’t know what kind of sorcery this is but it’s blowing my mind.

I input a bunch of completely made up words (Quastral Syncing, Zarnix Meshing, HIBAX, Bilxer) and used them in a sentence and the model zero-shotted perfect speech recognition!

It’s so counterintuitive for me that this would work. I would have bet that you have to provide at least one audio sample in order for the model to recognize a word it was never trained on.

Providing it to the model in text modality and it being able to recognize it in the audio modality must be an emergent property.

They’re all in. They announced they’ll add support for it in the desktop app and the API in the coming months: https://x.com/OpenAIDevs/status/1904957755829481737

Currently supported in the Agents SDK https://openai.github.io/openai-agents-python/mcp/

we started using it recently at my work. the code changes walkthrough is nice

I think the same can be said about AI-assisted writing…

I like the ideas presented in the post but it’s too long and highly repetitive.

AI will happily expand a few information dense bullet points into a lengthy essay. But the real work of a strong writer is distilling complex ideas into few words.

I think these tests are useful as regression tests - unit tests can be really helpful when making changes down the line, tipping you off that you missed something. Also much easier to refactor when there’s good test coverage.

From the PR: unit tests: what are they good for?

Answer: Personal opinion - writing unit testing is not fun. It becomes even less appealing as your codebase grows and maintaining tests becomes a time-consuming chore.

However, the benefits of comprehensive unit tests are real:

Reliability: They create a more reliable codebase where developers can make changes confidently

Speed: Teams can move quickly without fear of breaking existing functionality

Safe Refactoring: Code improvements and restructuring become significantly safer when backed by thorough tests

Living Documentation: Tests serve as clear documentation of your code's behavior:

They show exactly what happens for each input They present changes in a human-readable format: "for this input → expect this output" They run quickly and are easy to execute This immediate feedback loop is beneficial during development

I tried generating the same test with all 5 models in Qodo Gen.

o1 is very slow - like, you can go get a coffee while it generates a single test (if it doesn’t time out in middle).

o1-mini thought worked really well. It generated a good test and wasn’t noticeably slower than the other models.

My feeling is that o1-mini will end up being more useful for coding than o1, except for maybe some specific instances where you need very deep analysis

From my understanding faster-whisper optimizes the inference without changing the model itself. Here they seem to be changing the model architecture but not applying other optimizations.

50% on its own doesn’t make this the current best choice for production. But I imagine this could become the new base model that all of the inference optimizations are applied to.

Wonder if it’s plug and play or if faster-whisper and others would need to reimplement from scratch?

GH repo: https://github.com/aiola-lab/whisper-medusa

I think you may have misunderstood the figures.

Based on my understanding, only 1:20 passed the automated acceptance criteria (build, run, pass, increase coverage). Of those that made it through to the human review, “over 50% of the diffs submitted were accepted by developers” according to the paper

Yes, that’s definitely the main reason. It’s called “burying the lede”.

Saving $6M is key information that makes this story interesting. It’s buried all the way at the bottom of the first blog and is completely missing from the second blog which focuses specifically on the migration

HN with tags and filters would be great

Who are a few of your best twitter follows for AI?

Can you share a few of your best twitter follows for AI?

https://archive.is/nsECJ

We made a lot of investments in hardware…

Sounds bizarre until you remember that over 80% of servers are still on prem

I want to see a serious statistician look at the Hans Niemann chess cheating case.

There could very well be similar bias and shoddy statistics at play there

The long term trend isn’t reversing anytime soon

The more senior the role within a company, the more people can thrive on BS alone.

If an entry level employee is terrible, it will be obvious in a matter of days. It takes years for bad hires at the VP level and above to be recognized.