We 3.5x'd Our Pull Requests with AI: Now We Catch Fewer Bugs

https://brodzinski.com/2026/07/3x-pull-request-size-ai.html

by flail • 8 days ago

7 5 8 days ago

rurban2• 8 days ago

Exact the same bullpoints and trade offs I am facing. More code, faster, more bugs, esp. more security bugs.

Part of the more bugs is that the tools find much more bugs and write much better tests than before with only fuzzing disturbing our peace for a while.

flail2• 8 days ago

We also find that the agents tend to find false positives. Especially when it comes to security. In theory, you can get an automated pen test every week. In practice, it drowns you in triaging reported vulnerabilities, as you find a significant part of them aren't something that you'd like to address, like ever.

Synthetic7346• 8 days ago

What models do you use for automated pen tests? Don't the Frontier labs block security stuff?

rurban2• 8 days ago

Yes, I have the same. Then I have to clear memory and adjust AGENTS.md (symlinked to CLAUDE.md). Or delete the work. Happens rarely though. Like 5%.

In very rare cases it insists to being wrong. Then I have to switch models.

We 3.5x'd Our Pull Requests with AI: Now We Catch Fewer Bugs

Related Stories