HN user
ROCm has improved but the reality is you're still fighting the driver stack more than the models. If you're actually doing local inference on AMD you're spending your time on CUDA compatibility layers, not the AI part. Two lines of python is marketing, the gap between demo and working AMD setup is still real.
Regex and NER both have fun edge cases for redaction. Medical record numbers that look like dates, addresses embedded in prose, account numbers with varying formats. Detection method matters more than people think.
laziness makes you understand the problem before writing anything. an LLM will happily generate 500 lines for something that needed 20 because it never has to maintain any of it.
The sanitised optimism problem mentioned upthread is the real gap here. Event stream logging tells you what tools were called and in what order, but it doesn't tell you whether the agent's self-reported outcome matches reality.
LLM as judge drifts in weird ways if you don't have ground truth to calibrate against. Good that you've got that built in. Would love to see eval stability tracking over time though, same prompt different day sometimes gives different scores.
Biggest change for us has been alert volume. AI-generated code ships faster, which means more deploys per day, which means more things to monitor...
The "every new model is THE one" cycle is getting a bit old but the Capybara tier thing is actually worth paying attention to.
The approval layer problem is interesting because the naive approach (approve every action) doesn't scale, but the fully autonomous approach terrifies anyone running these in production...
I've noticed the time-of-day variance too. My working theory is it's related to load, not model changes. Same prompt at 6am Sydney time (when US is asleep) consistently gets better results than the same prompt at noon. The "ignoring instructions" behavior usually means it's working from a compressed context where earlier instructions got summarized away.
Retry logic modeling is a great inclusion most cost estimates miss this completely. In my experience retries account for 15-25% of total cost in production agent systems, especially with tool calling where validation failures trigger re-prompts. Would be useful to see a "worst case" scenario that models cascading retries (retry triggers another tool call that also retries).
The file interaction heatmap is a smart addition. When debugging long agent runs, knowing which files got touched 30+ times vs once tells you immediately where the agent got stuck in a retry loop. One thing I'd love to see: a timeline view that highlights when the agent re-reads a file it already read that's usually the clearest signal of a stuck loop.
The 71% reduction is interesting but I'd want to see where those tokens are actually going in a typical agent session. In my experience running multi-step coding agents, the git output itself is rarely the bottleneck...