Agentic test processes, LLM benchmarks, and other notes on agentic coding fr

https://danluu.com/ai-coding/#llm-variance

20 2 15 days ago

simonw• 15 days ago

This piece really needs a date on it, otherwise sentences like "I've been using AI fairly heavily since last November" aren't going to age well.

I found a date in https://danluu.com/sitemap.xml (which I read while wondering why archive.org listed that site as "excluded from the Wayback Machine") - the article is listed there as "2026-07-03" and the other dates all look credible.

Herring• 15 days ago

If you're using randomized testing as "extra credit", to catch a few more bugs, or to replace traditional software testing processes, you can just tell an LLM to look for risky areas of the code and find invariants that might be violated and fuzz them. This works ok. When I've convinced people to try some randomized testing, they usually start here and find quite a few bugs they're happy to have found. Due to the nature of who's interested in trying out novel-to-them test techniques, this is often from people who've worked on some of the most well-tested and reliable code at the company and they can find bugs in their own relatively well-tested code.

Counterpoint: I just tried a few rounds of LLM-generated fuzzing on my current side project and couldn't find a single bug. Maybe try coding more defensively?

Agentic test processes, LLM benchmarks, and other notes on agentic coding fr

Related Stories