lukasco

Shipping a lot faster definitely puts a big strain on the QA side of the house. But I'm not at all convinced that things have gotten worse even with "AI slop". CI/CD and releasing features more quickly has long been proven to make software more reliable rather than less. But to work, it requires much much better test coverage. CI/CD and AI code go hand in hand. It's all about the test coverage. Not enough tests, then exactly as you are saying: ship 10x the bugs (which actually means your software will fail catastrophically). And you mention leadership (and others did too). Here's my view: if your company leadership doesn't care about software quality, then it doesn't matter if you used Fable, or agile, or 10 year old agile, the software won't be great. You'll never have the time to do things well.

I still find it hard to develop harnesses because you can't really test many turns with an llm in the middle. I suppose doing LLM as judge is one way to start to tackle this kind of thing.

I feel like in some ways this problem is starting to self-correct, sadly by creating the dead internet. If there's no business model to creating content since it will get scraped, then no content will get created.

So many comments here talking about the downsides. The only reason to do a rewrite is because there are massive upsides. Maybe the implicit point is that the upside (memory safety must be the biggest), isn't worth the downside (lots of bugs to be figured out before you trust it).

Quite a lot of projects are trying this "rewrite to a new language using LLM", both internally, or externally (like is here). For me, they confirm some (slightly controversial) takes.

1. human code reviews are dead. We don't yet know what's next. Two reasons they are dead: too much code to review, and code reviewing sucks (who wants to spend their days reviewing code?) 2. Not knowing how to review LLM code is a big barrier to adoption, but bigger regression test suites (testability/evals) is almost certainly the direction. 3. There are a lot of projects that haven't moved to more modern infra because it was too hard. Now it's much easier. Sure stuff will go wrong. Sure it all has to be tested. What's new here? 4. Programming languages for LLMs are coming. 5. Projects that don't allow AI coding will be forced to come around or fade.

Separately, bit off topic:

New projects will often have LLMs built in, so non-determinism will be inherent in the project. No amount of code review will be able to eliminate that.

This is the real problem with LLMs. There is no way to separate code from data. At best, models could be trained on tokens that indicate untrusted data coming in. But then the untrusted tokens could also be messed with.

I've wondered if it would be possible for there to be two input streams: 1, for prompt, 2 for untrusted data. But I suspect that transformers would still only optionally decide what each one was for. So it would still be a prompt level suggestion, rather than a hard and fast rule.

I mean, I'm guilty too. Claude code scrapes everything in sight whenever I search for something.

I've written a small tool to query Github to see if a particular feature has been released for an open source project. I had to use an API token to not get immediately blocked. But I don't think rate limits are new for Github.

Cross platform building and packaging in C/C++ is such a hot mess. There's so many dimensions to unpack I don't even really know where to start. (I say this as the person who has been packaging GIMP for Mac for the last good number of years.)

OK, here's a few (with a MacOS slant): - compilers (gcc, clang, and their many versions) - libc (and friends) compatibility (I can't say I even ever delved into this one, but it's bit me) - package manager (macports, homebrew) - building for backward compatibility; what's the earliest MacOS version to support (the package managers either like to build for the OS they are running on, or force you -- yes I'm looking at you homebrew) - dependency and dependency version management (love you pkgconfig) - build system for each package (cmake, autotools, meson, ...) - bundling everything into an application - turning that application into a Mac application - code signing and notarization - creating the DMG - debug symbols - crash detection and notification Like right now, libheif on 26 can't be built for 11 and it's not clear why (or maybe they just fixed it...but it's been weeks)

Same here. Before auto, couldn’t handle the constant stopping because of backslashes no matter how many things I permitted.

Yeah, that looked pretty cool. But I’m always sceptical of these announcements until I try them.

Totally agree. And really, we use the llms to be the universal layer, not the harness. It’s already nigh on impossible to eval a harness with multiple turns (at least as far as I’ve seen), so multiply by specific llm prompt…)

Fun, seems to be working. Do you have levels? All the games I played seemed to have a static level.

This is the big question! Is it really sensible to distribute locally running software at all? Maybe this leads to the locked down Mac App Store (and Windows I'm sure)? Apple's been wanting to lock down the Mac for ages, and this might be what makes people accept it.

It sounds like harnesses might have to start to have model by model system prompts, though retrying works, I guess. It reminds me of the ancient times when browsers all read HTML and CSS differently, and differently on different devices. In that sense, this is nothing new. I was going to say, at least we don't have different device types, but then, the model still has to output the right variant of `grep` as well.

It's not YOLO, but auto mode in Claude Code does reduce the amount you have to approve significantly. And frankly, without it, progress is constantly interrupted by permission requests. It's all I use. Don't even really switch into Plan mode manually anymore.

This is the big thing I'm struggling with:

Code reviews are still a critical step in most workflows. Though seems like everyone uses them for a different purpose: extra pair of eyes to meet a regulation/security, style police, and what this one says: maintainability.

And at the same time, code reviews are now a massive bottleneck in the development pipeline. Frankly, in a lot of teams they have been that for a long time. Though many would argue it was the only thing stopping absolute crap going into production.

But in a world where none of us writes code anymore, and I think we're there, even if the future is "just not evenly distributed" (Gibson), why should we have to do code reviews, the worst job of all?

Leaves me thinking that code reviews will land in the dustbin of history.

But for now I don't have a better solution.