HN user
I guess it all comes down to what a meaningful gain is. I agree that 10-30% is meaningful and if “software is a gas” this will lead to more software. But my expectations had become anchored to the frontier labs marketing (10x), and in that context the data was telling me that LLMs are a good productivity tool rather than a disruptor of human labor.
BTW thanks for the links to the studies
I don’t work in SWE so I am just reacting to the claims that LLMs 10x productivity and are leading to mass layoff in the industry. In that context the 6-12% productivity gain at a company “all in” on AI didn’t seem impressive. LMMs can be amazing tools, but I still don’t think these studies back up the claims being made by frontier labs.
And I think the 6-12% measure reports is from a 2025 not 2024 study?
A few studies over different time frames:
[1] https://www.youtube.com/watch?v=1OzxYK2-qsI [2] https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-gen... [3] https://www.youtube.com/watch?v=JvosMkuNxF8 [4] https://www.faros.ai/blog/ai-software-engineering
"The technology has improved so rapidly that this study is now close-to-meaningless."
You could have said that anytime in the last 3 years, but the data has never shown it to be true. Is there data to show that the current gen models are so much better than the last gen models that the existing productivity data should be ignored? I don't think the coding benchmarks show a step change in capabilities, its generally dev vibes rather than a large change to measurements.
Sorry for the late response!
My guess is they didn't publish it because they only measured it at one company, if they had the data across the cohort they would have published.
The general result that review/re-wrok can cancel out the productivity gains is supported by other studies
AI generated code is 1.7x more buggy vs human generated code: https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-gen...
Individual dev productivity gains are offset by peers having to review the verbose (and buggy) AI code: https://www.faros.ai/blog/ai-software-engineering
On agentic being the saviour for productivity, Meta measured a 6-12% productivity boost from agents programming: https://www.youtube.com/watch?v=1OzxYK2-qsI&si=ABTk-2RZM-leT...
"But it's different now" :)
Studies have shown that software engineers are very bad at judging their own productivity. When a software engineer feels more productive the inverse is just as likely to be true. Thats why anecdotal data can't be trusted.
Please provide links to the studies, I am genuinely curious. I have been looking for data but most studies I find showing an uplift are just looking at LOC or PRs, which of course is nonsense.
Meta measured a 6-12% uplift in productivity from adopting agentic coding. Thats paltry. A Stanford case study found that after accounting for buggy code that needed to be re-worked there may be no productivity uplift.
I haven't seen any study showing a genuine uplift after accounting for properly reviewing and fixing the AI generated code.
I think an OpenAI paper showed 25% of GPT usage is “seeking information”. In that case Google also has a an advantage from being the default search provider on iOS and Android. I do find myself using the address bar in a browser like a chat box.
https://cdn.openai.com/pdf/a253471f-8260-40c6-a2cc-aa93fe9f1...
The productivity studies on software engineers directly don't show much of a productivity gain certainly nowhere near the 10x the frontier labs would like to claim.
When including re-work of bugs in the AI generated code some studies find that AI has no positive impact on software developer productivity, and can even have a negative impact.
The main problem with these studies are they are backward looking, so frontier labs can always claim the next model will be the one that delivers the promised productivity gains and displace human workers.
"History doesn't repeat itself, but it often rhymes", except in the world of computer science where history does repeat.
Radiology has proven to be one of the most defensive jobs in medicine, radiologists beat AI once already!
https://www.worksinprogress.news/p/why-ai-isnt-replacing-rad...
My bad. What was the result when they measured productivity after rework across the entire co hort?
I think you are quoting productivity measured before checking the code actually works and correcting it. After re-work productivity drops to 1%. Tinestamp 14:04.
This study does a good job of measuring the productivity impact. It found 1% uplift in dev productivity from using AI.
This Stanford study on developer productivity found 0 correlation between developers assessment of their own productivity and independent measures of their productivity. Any anecdotal evidence from developers on how AI has made them more or less productive is worthless.
Isn’t this what Tao is addressing in the link, that LLMs haven’t encoded reasoning? Success in IMO is misleading because they are synthetic problems with known solutions that are subject to contamination (answers to similar questions are available in the textbooks and online).
He also discusses his view on the similarity and differences between mathematics and natural language.Tao says mathematics is driven entirely by efficiency, so presumably using natural language to do mathematics is a step backwards.
I think people make comments on LLMs not being smart in reaction to the comments from the leaders of AI labs that LLMs are so smart they could/will lead to mass unemployment.
Are SWE’s really experiencing a productivity uplift? When studies attempt to measure the productivity impact of AI in software the results I have seen are underwhelming compared to the frontier labs marketing.
My impression from Tao’s answers is that there is a gulf between the reality of what mathematicians actually do and the benchmarks that the frontier labs tout. It doesn’t seem trivial to close that gap and create an LLM that can replace mathematicians.
Meta internal study showed a 6-12% productivity uplift.
The productivity uplift is massive, Meta got a 6-12% productivity uplift from AI coding!
The studies I have seen show that AI written software is 70% more buggy than human written code [1]. I am curious where you get your data on AI code having “a similar reliability to accepted software”?
[1] https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-gen...
Maybe you are right, but maybe it’s radiology all over again.
Why assume the breakdown between benchmarks and RoI is due to humans? The map is not the territory, the benchmark is not reality, there world is more complex than computer scientists understand.
Good deal for Donald Trump Jr.
Superficially it looks like a lot of these AI features are available elsewhere, what differentiates Aluo for e-commerce?
Meta engineers who were above median users of agentic coding saw a 6-12% increase in throughput.
Is AI really topping the charts in country music?
Forever, tech has become a never ending sequence of bubbles and hype cycles.
Matt Levine never worked in crypto, but he knew SBF/FTX was a scam. Humans are humans.
METR's study uses a 50% sucess rate. For most enterpise applications a 50% pass rate is unacceptable in automation, more like >99.9%.