HN user
I’d suspect the harness to massively affect token use and optimisation
Yes they do according to databricks -> https://www.databricks.com/blog/benchmarking-coding-agents-d...
That's why I find comparing models on benchmarks only gives the tendency. We should be comparing model x harness to have accurate metrics.
What is your harness with every model ?
my use cases stop aligning to swebench pro around 50% accuracy, and more closely align with DeepSWE.
What do you mean by that ? If the model is higher than 50% on swebench pro then it tends to drift from what you like it to do, like DeepSWE benchmarks ?
Shouldn’t we fear they start doing only close source like most us labs once they catch up in market shares ?
Well, I had not heard of RLM before, just read the paper, thank you for introducing me to your lazy version !
I’ve never seen the Id approach before, that’s a good idea ! Though I was wondering how do you manage to keep costs low within the 7 agents ?
Giving the accuracy-token graph and not the accuracy-cost graph, thus we cannot easily compare costs with other models, is not a way to gain my trust
I’m truly impressed by your work ! I don’t know if this is planned for near future, but how about adding energy efficiency benchmarks ? Because running locally is a great feeling, but the electricity bill should not be forgotten
Well, had a meeting with a VC that literally brought his note taking 3rd person to the meeting, did not even say hello. Tbh, I still find it creepier than an AI transcriber.
This seems really bad…
I guess the technology used here must be ground-breaking lol
Did not seem to find how much tokens per second he achieved with this setup ?
Thus the prices in switzerland are higher than anywhere else. I don’t know about you, but I’d have no use of 25gbits/s
Lol next time I’ll just apply with 4 accounts and maybe get in once.
Are many people using HackerRank ATS ?
Have you tried opencode go ?
Would you recommand some ressources about how multiple neural engines are used in data centers ?
I guess it was inevitable. Is it only RAM related ?
Funny one x) Though I ain’t sure if even more data is useful on hackernews
Great way to honor Tony and his work
In the end, sorting prs and vulnerabilities has been the same for open source maintainers. How about adding a credibility score to every github account ? Couldn’t that cut sorting times ?
3B param on par with opus 4.5 sounds interesting. Will read the full article before making my mind
Lucky me, I never go out without my 256gb unified ram mac x)
Funny but my phone is overheating
I know you don’t judge a book by its cover, but a vercel extension does not look professional nah ?
This might actually be clever since Microsoft dev will be longing claude code features and might result in copilot getting way better
« LLMs can successfully author CODA kernels » That might speed up progress in this area then
Is there some kind of memory enabling, for instance, an agent to remember guidelines on a repo without having to feed at the beginning of each session 4 markdown files and spending the corresponding tokens each time ?
I can’t seem to grasp why everyone says only slop gets produced by AI models (and particularly those world models). Imo it’s shit in -> shit out. Great work can be achieved using those. Slop gets produced by careless users.