efromvt

I’ve noticed it having weird message dropping and replay in general, but the compaction boundary has been pretty solid.

This was more interesting/creative than I expected on both sides (the prompt and the existing safeguards). I love that obscure Cloudflare validation turnstiles seem unsuspicious based on training data.

DSLs are a great middle ground for 'use LLM to turn ambiguous spec into something well defined', with the caveat that without discipline they'll inevitably expand until you should just have the agent write whatever the final language is.

There's a context tax up front (which will hopefully be less relevant over time) and then you really need a compiler/linter with helpful errors to keep it on the rails, because there is no corrective context in pretraining for something novel.

A purely descriptive DSL is just a convention, which is useful, but doesn't inject reliability the same way an enforced syntactic contract does.

I think like social engineering, it will always be an issue to some degree, and we'll build safeguards until it's at a 'societally comfortable' baseline level. Which is maybe not particularly comforting, but I don't see us closing Pandora's Box here.

Same as last month for once - optimizing how well agents can work with a new language [1]. I've been able to 2-3x success rate and drop total tokens for complex tasks significantly (though the initial syntax dump is rough - need to do some ablation there to get it down).

The best part has been that I think it's significantly improved things for humans too; it's weirdly satisfying to be able to measure improved ergonomics. Also, since a big pitch/theory was that the language should be ideal for agents as a result of the original nice things for humans it was designed for, it's a relief to be able actually measure a concrete lift.

[1] https://trilogydata.dev/ - SQL with types, composable functions of arbitrary complexity, and a native semantic layer.

I guess technically it is in the SQL standard, but optional, as S098? I agree that SQL is sorely lacking here and I'm hoping that the OLAP side innovation (presto, bigquery, snowflake, duckdb all seem to do better) help push it forward.

never use system python, always use virtual envs. (a bad answer, but agents do remove the setup boilerplate). UV does relatively completely solve this but it's a big dependency to take so understand why people don't rely on it

Out of curiosity, how often are the resource limits the bottlenecks? What do harnesses do to help here - limit parallelism? More efficient tools?

Second comment, having read in more depth (really love the auto-layout detail!) - the spec doesn't seem to naturally support layering (which is useful in some multi-axis automatic cases) - any plans for composability?

Op1M5 was scarring the first (dozen) time around! Thank you minelayers

I wonder if all those different heavy shenanigans are just to get the hollander's gauss/hunchback autoconnon in the right torso by default.

(original Mechcommander is still the best of the series, even if mission 5 broke my soul for awhile).

Semantic types as the extra formatting factor is super useful because they are concise encodings of a lot of formatting boilerplate.[1] Do you envision the flint type registry being shared/extensible? Why not have that be a data property itself?

[1] landed on pretty much the same spec for higher level tether charting on vegalite

Very much in agreement with this but especially on the UX side I find that 'what is intuitive to me' is not universal because you come with a bunch of contextual priors you can't clear. I wish there was more tooling focused on how to use tools to expand customer empathy past what you can directly experience; that's an inherently audience-limiting approach.

(I admire PMs that can cross that chasm; it's also why they're often bad at infra tooling, because you can just short-circuit the empathy).

Exactly, so it has a success rate of 0 and infinite cost/completion on your relevant benchmark. If the benchmark doesn't map to what you need it to, then yeah, it's not a useful input.

If you don't have a backend, then it's all telemetry, right? And backend logs don't capture a lot of the UX side of things - how a call got triggered, from where, etc (which yeah you can start to instrument, but then that's telemetry).

Isn't the benchmark working exactly how it should in that case?

It feels like you could argue that since you control nature/nurture it's very possible to create a model aligned to an arbitrary spec - there is no theoretical reason it's not possible given N runs, and you only need to take the successful one. (ethically very.. questionable in humans) I think it's much trickier to define that spec, much less measure it and validate that a model is aligned to it.

I'm not sure the security/safety stuff is entirely in their control at this point, though you can argue that they are indirectly responsible through encouraging regulation via their positions on safety/risk.

I find it interesting that when drawing this parallel you mention that some devs 'get it' and 'build a great culture'; I think this is exactly where the analogy breaks down. Good managers get great results from people (and for people! they are linked).

Good AI managers are just running optimization loops at more declarative levels. Yeah, you need to get comfortable with less personal review of code for both, but I think the differences outweigh the commonalities - it's much easier for someone with a more 'traditional' IC model to be successful with agents then they would be with management, and I think most (good) management training would be entirely irrelevant. Parallels are maybe tighter to higher IC progressions.

I think it often useful to push the conversation down "we built a system for humans that dealt with this, what from that is or is not applicable for agents in the same context"? Humans randomizing resume review for screening is pretty known; I've seen companies try to fight it with things like hiding information, panel reviews, etc - it's unclear to me how effective those would be for agents (honestly, it was unclear how effective those were for humans). I was depressed about the hiring process before we had AI screening and I remain depressed about it.

This starts to sound more like ‘social engineering a human assistant’, so there’s a degree of required specialization that does meaningfully increase costs.

I think the theory is that it’s not purely static - you need to keep training and tuning the model (even just for general knowledge upkeep with current architecture) and so the infra/data is a contributory moat. Exfiltrating weights would get you a depreciating asset (plus we have all the lovely legal and regulatory frameworks to further protect them, which IS more like publishing)

million times this - getattr on every dataclass is a wild choice

The SELECT machinery is the product with databases! SQL often the shortest description of the processing logic, and the database has an efficient local execution engine that can prune/reduce data read based on the plan. Very hard to match in app, especially when joins get involved.

Why would an agent supersede a service for a well-defined workflow contract that does not require an agentic loop? I assume both will need to exist.

I’d be interested in the benchmarking if you ever write it up! People do seem to assume LLM as a judge/panel improves outcomes (and arguably it does in cases like code review?) but I suspect it is very situational and the priors from human panel of experts don’t always translate cleanly.

Been working on optimizing CLIs for cheap agent use and figuring out how to build integrated agentic features that aren’t a full chat interface. Agent UX optimization is kind of fun! Much more testable than human UX, though it’ll be interesting to see how much generalizes across model families.

Been doing this to improve/simplify the grammar for Trilogy[1], a streamlined SQL language - I’ve been planning a redo of one feature and it’s nice to be able to rapidly get feedback on various syntax success rates. Also been particularly useful to optimize error messages, which should help people too.

[1] https://trilogydata.dev/

Fantastic to have PyO3/Maturin guides too - the rust/python/typescript turducken I’ve always wanted.

repetition of "belt-and-suspenders" kills me with opus, especially because it always means the model is suppressing something I would want to be an actual failure

I think the perception is that it is not 'only marginally better'; whether or not you specifically agree that perceived quality gap lets them differentiate on price.

I'd further say that there are probably enough rational actors running evals out there that the marginally better is not pure vibes for the cases where people are spending lots of money, but I only have direct line of sight to some of those eval suites. Maybe everyone is irrational and anthropic is exploiting that!