Code Reviews Don't Scale

https://medium.com/@ben.g.northmore/the-death-of-code-reviews-as-we-know-them-d9a3e8d6795e

by surprisefox • 21 days ago

4 13 21 days ago

g-b-r3• 21 days ago

Read the summary first. Challenge the claims. Decide where the risk is likely to be. Then use the code to verify those claims instead of using the code to discover them in the first place.

You get that with tiny commits, that make confirming the commits' text as easy as possible; if they're hierarchical it's even better, but that requires using a decent commits viewer (absolutely not github and the likes).

surprisefox6• 21 days ago

I think it's a fair point, now code gen is much quicker, keeping commits slim is a challenge in itself. Agree the big repo platforms don't really do any favours in that department

g-b-r3• 21 days ago

but I guess this is written by an AI

surprisefox6• 21 days ago

Not AI. Just a bored developer with an evening to spare.

It's interesting though people seem to assume everything is ai-generated now

dieselgate2• 21 days ago

Right on, I've been quite bored from 8-4 hours lately and have been commenting a lot around forums etc and can relate. Writing articles like this is more productive. I agree about the AI comments, it's basically written in the HN Guidelines "Please don't complain about tangential annoyances" or multiple others like "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

dieselgate2• 21 days ago

The diff is still treated as the source of truth. I'd almost reverse that.

I think this is interesting but it seems brittle. It makes me think of the Agile Manifesto "Working software over comprehensive documentation" - not that I evangelize for Agile.

surprisefox6• 21 days ago

Totally with you, it does feel fragile. Something hard to come to terms with for me is how much we actually care about the way something has been done itself

if there are three equally good ways of solving a problem, it seems like reading every line stops feeling like a good use of time (and maybe even kills the speed gains of ai assistance)

surprisefox6• 21 days ago

Thanks for the discussion by the way :) It's refreshing to chat outside of my own mind and get a fresh take

othmanosx• 21 days ago

I think AI should complement the human review, not replace it, and I think everyone already agrees with me on this.

The problem with AI reviews, is that they add a lot more text to read, the PR description is another wall of text, test steps are very verbose, most comments are just noise.

I believe AI should instead help pointing the dev to the important stuff and help identify noise, this would actually make the review go faster.

since the introduction of AI, the bottleneck for shipping code shifted from producing code to reviewing it, and the PR reviewing experience on Github is lackluster, leaving a lot to be desired.

my team now ships too much code and it sits in review for too long, reviewing AI code is not easy and we need a better tool to review code, especially on large PRs, Github no longer meets my needs or use cases so I had to build something myself.

I got frustrated enough that I built my own reviewer around this idea: instead of adding commentary, it groups files into folders by complexity, surfaces the handful of changes that actually need deep review, and marks the mechanical noise so you can blow through it. On migration PRs it cut my review time down a lot.

As a web dev, I know what works and what doesn't, I hated how AI bloats the PR with more text, so I changed that to make the AI help me categorize the files into folders, splitting them by complexity, and pointing me to the important stuff I need to review indepth, and flag the noise so I can skim it faster, that sped up reviewing time immensely, especially on migration PRs where most is mechanical name changes.

I can share more if people are curious, I'd rather not turn this into an ad.

TomOwens• 21 days ago

Having a bottleneck with AI-generated code is probably a good thing.

Consider before AI. In cases where code was written by a trusted human, you had confidence that they were doing the right stuff - the people understood the context and requirements, the changes were based on those requirements, and tests were updated. You could rely on some basic automated checks to run a linter and static analyzer, execute the tests, and check test coverage. You can defer longer-running tests to a later stage, but people can merge their work in and collaborate. You can have blocking reviews for untrusted outside contributors or new employees and review the rest later.

After AI, the input is a mess. I haven't personally seen the worst of it, but I've read about people generating code before any requirements development or analysis. Code is generated without understanding what it is supposed to do, which also prevents good testing. Even with good requirements, people generate solutions that aren't right, and the generation step may remove failing tests or make unintended changes that aren't covered by tests. So much code is generated from untrusted sources and rubber-stamped. Introducing a bottleneck where a human can slow the process down is a good thing here.

Maybe if we can get people to understand how to use AI tools responsibly and have humans take full accountability for the output and changes, we can get back to the old way. Trusted humans and an AI agent assistant can make changes that are solid enough to merge. Independent control over the pipeline and basic automated checks would give just enough confidence in the change, and delayed review before deploying or enabling the feature would work. But not until trusted humans actually maintain oversight over their tools.

Code Reviews Don't Scale

Related Stories