HN user
This is super interesting, and I like the idea of verifiable artifacts that an agent can produce, i.e. notebooks for analysis, links to the source for some claims. Building for scale, it would be interesting to know how the author thinks about automating that and building benchmarks to automate testing the quality
I mean, they basically did that to make sure that Composer 2.5 sits on the right side, and to a quick reader looks good
This is an interesting finding, but very specialised. It would also be great to get some more information about the benchmark. Is it just a collection of files with vulnerabilities, or are they hidden in a real codebase, where LLM based approaches will not be able to scan every file like a static code scanner is able todo.
It's an interesting post, but i'm a bit skeptical on their decision to report the best run for each agent, and not just the mean over the 5 runs. We have seen this as well in running benchmarks, that variance within one setup can be pretty big.
Just a question on the benchmark. It states that it is on real world code, but all the repos in the dataset are intentionally vulnerable repos right, not real world codebases that have reported vulnerabilities?
It goes a bit up and down, compared to two years ago, in my feeling, both anthropic and open ai coding models have made massive jumps. In between big releases I do feel the quality of the models varies over time. 2 years ago I got annoyed with the models being crappy autocompletes, right now I'm just managing them more rather than coding myself
There definitely is the danger of a lot of garbage being shipped, but with both the models getting better and better, and more tools and ways of working being discovered, I believe the quality of what is being outputted is going up as well. Right now, a tool like claude code is more like a junior dev. you really need to steer it well, and know what you are doing, to make sure it doesn't output slop. but with the ability to create agents that work together, implementing design agents, coding agents, review agents, even a non-technical person will be able to design a workflow that spits out quality code, and I think that time isn't far away.