jarodrh

True, but human review isn't visible in the logs, hmm, however I suppose retry patterns can, but frugon doesn't currently track this, so we can't quantify this yet.

Frugon analyzes your existing logs offline on cost and quality tier. The "looks fine per request" is exactly the reason why --judge exists.

Earlier, cyanydeez inspired the consideration of a metric "effective cost per judged success", which could also answer this point.

Try it on your logs, and tell me if anything falls through the cracks.

As well. Someday costed opcodes and sharded redundancy for all of this;

Hmm, noted.

If an algorithm selects according to rank, there's a cost/score/fitness/survival/error function.

- Yes, what I'm saying is that frugon has a small enough candidate pool that it can all be evaluated at the same time. It doesn't need to search through it.

Awhile back I had to match exterior paint to printed paint swatches and I'm stereotypically not good at colors, so finally I arrived at just A/B; this one or this one and discard.

- This is exactly the shape the judge prompt takes in frugon.

See: src/frugon/measure.py

Then search: JUDGE_PROMPT_TEMPLATE

Down the rabbit hole we go.

Is this a convex optimization or non-convex optimization problem?

- Neither, frugon sidesteps that issue by having a small finite candidate selection that it evaluates completely. No search needed because the problem is discrete and small.

OpenInference and OpenTelemetry have a schema for agent sessions.

- Good to know for when I look into it.

Accuracy is sensitive to nonce and temperature without variance in the model hyperparameters

- Yes, this is a current weakness in frugon...Hmm, I see, so I'd essentially need to establish a baseline (with whatever the default temp is) + nonce, then compare both spreads. Noted.

Good thread. Give frugon a try on your own logs, and tell me what you think.

Very interesting thoughts. This is a whole discussion on its own.

The ones that I mulled over that I feel have legs in my line of thinking are:

minimize cost to converge upon sufficiently low error problem

- This is the one. Frugon does an easy/hard split to keep quality within a certain tolerance, and find the lowest costed model within that constraint. The judge handles the "sufficiently low error" side of that via sampled prompt logs.

Evals and OpenInference (OpenTelemetry) might be useful.

- Evals already a given. This is frugon's wheelhouse, however it currently only supports OpenAI-style jsonl. I'll definitely consider adding support for OpenInference.

A simple experiment: vary only a nonce in the prompt and compare output value. The nonce is a parameter. The model is a hyperparameter.

- This is an intriguing idea. This would actually enhance/strengthen frugon's stance on the "minimize cost to converge upon sufficiently low error problem". Noted.

Yes they do, it's quite the popular topic atm. RouteLLM (OSS - frugon's savings bands are based on their research), OpenRouter's auto mode; I actually commented on a /show post not long ago: Wayfinder Router (neat project) - https://news.ycombinator.com/item?id=48704373 and more.

My stance on the router point is a config-led (deterministic) one, even if the eventual industry consensus is a hybrid of dynamic (AI-led) + deterministic. Both sides still require evals/evidence for policy creation, which is what frugon provides.

Thanks. Yes, local models are gaining a lot of traction.

The measure/judge step uses LiteLLM, so it does sample local models. I just tested "--candidates ollama/llama3.2:1b", and that works - ignoring the lack of rich UX for local/unpriced models, as I was focussing on cloud cost, but you've inspired me to give this area some polish.

Noted: "response time as a new dimension of judging" - Added to the roadmap.

Try it and let me know if you hit a wall.

Interesting project. Most of what I read appears to deal well with a linear agent session.

How does it handle concurrent agent/subagent spawning?

Do they fall under the same sessionId (where they're potentially fighting over the warm route breaking it with every task-type change) or get their own individual sessionIds (where you must rewrite the shared prefix for each agent)?

Thanks. I'm not sure if you're asking "how does frugon compare provider models" or "can it compare across provider models" or even "how does frugon compare to other tools"...

On the first: It does this utilizing pricing from LiteLLM registry, and quality tiers from LMArena.

On the second: A log of calls for OpenAI can get recommendations for Gemini or Anthropic or DeepSeek etc. and vice versa.

On the third: It's free, local and offline. Hosted tools normally require you to plug them into your live traffic, whereas frugon reads logs you already have. And you have the benefit of exploring the source code.

You got cut off at the end, but I hear your point.

I completely agree that cost-per-successful-outcome is the goal, and "cheapest" is never the entire solution, but only routing to the cheapest isn't what frugon does. If it did that, then it would just pick something like GPT-4.1-Nano (one of the current cheapest models regardless of quality) for everything, and it would be useless.

The whole pipeline in frugon is predicated on quality tier gating. Frugon doesn't strip out quality. This is based on quality tiers from LMArena leaderboard data (CC-BY licensed). The easy/hard split only recommends routing calls predicted to survive the downgrade.

$ is operationally useless since $ is model depenedent

model+harness are symbiotic or antagonistic

To these points, frugon gives you --measure (you can compare the outcomes yourself via prompt sampling from both models - current and candidate/recommended) and --judge (championing a model to determine whether the recommended/candidate model actually successfully holds the quality of the current model from the prompts sampled). The win/tie rate summary outputted by the judge tells you whether the candidates you're comparing preserve the outcome you're currently getting from the current model measured on your own prompts. So, if your model succeeds at the prompt and the candidate ties this (judge's verdict), then you've successfully evaluated that you can swap to a different model at a lower cost. It's measured on your task distribution as a quality confirmation of frugon's initial offline quality tier gating routing recommendation.

Success rates are task-dependent, as every user prompts a model differently which could either get a successful outcome or a failure-doomed-to-retry outcome. No benchmark/table can tell you what your agent+subagent's odds of success are, unless all you do is pass an unaltered prompt that has been externally benchmarked. Hence why --measure and --judge exist. You must confirm the initial routing recommendation with your own evaluation (human/ai/evals)

However, you have made me think of combining the initial dollar cost analysis with judge outcome as an "effective cost per judged success" metric...which is a great idea, as they are currently separated.

Thanks. Yes, that's exactly right, and aptly named so in the code.

A few things to note: the prompt is actually deliberately tie-biased; the tie only then breaks on clear material differences: factual error, missing information the prompt requested, or just a complete failure to follow the instruction. The judge is explicitly told that response/output length, wording, style and formatting are not quality differences.

To ensure there's no favouritism, the judge only sees the outputs (A and B), it never sees whether the output is from the current model or from the candidate; and to ensure no funny business occurs, the outputs for A and B are randomized.

The default model is chosen automatically from the highest-tier model in the log, but this could invite self-bias (this is cautioned in the report output), so you can override this with --judge-model

You can check out the prompt for yourself in src/frugon/measure.py - search: JUDGE_PROMPT_TEMPLATE

That's an interesting thought, and one I'll take note of, but that would be a different tool. However, if you look hard enough I'd say we're tackling the same issue. I'm just choosing to look at the problem from a cost perspective as opposed to a raw token generation/prefill perspective. The mindset can be applied to both sides, but Frugon prices the cloud side.

The end goal is essentially the same and your mind went to that point, "primary-subagent model architecture". This is what the tool helps you figure out. It's not there to hold your hand and explain what your architecture should be, as I wanted it to be a small simple tool that would give the user insight into triggering your exact thought process. The thought process of breaking down their tasks by type, mapping that to individual models regardless of app or dev tool, regardless of cloud or local (the thought process transfers). It shows the user that they could route a portion of their calls to a cheaper model. It's then up to the user to understand the task type and point those calls to different models.

To directly answer the operationalization statement, it depends on what the call log is from (app/harness).

If harness, then the direction would be to pin models per role per task type; the routing recommendation maps directly to that, and measure/judge allows you to verify this before switching.

If app, then this is a similar shape as above, where you would then categorise and pin those calls identified in the recommendation to the model recommended, and as above, measure/judge allows you to verify before switching.

You're proposing an optimisation for throughput whereas Frugon is for spend. Same issue, just different lens.

Thanks! I love a good meaningful name

Frugal + on

:P

We've all been there where we choose the easy least efficient route just cuz (laziness...can afford it), but we've mostly done this consciously. The example in the article however does take this to another level and understandably so, as it's so easy now to take the easier route when you have no idea what you've requested is actually doing under the hood. I'm sure if you explained it to them, they'd immediately go...oooh I'm building my car every time I need to drive to work. As many comments have already stated, it's programming 101.

It's almost no different to using an expensive frontier model to do a general scouting task. Use the right tool for the job. A hammer can do most things, but should it be used for most things...

I think one of the most important non-technical things everyone should strive for is a mental awareness for figuring out what the right question is.

"Exactly!" said Deep Thought. "So once you do know what the question actually is, you'll know what the answer means."

Hmm, I honestly think a harness should be more about workflow, control, ease of use, memory optimization...other things I can't think of right now, the model/subscription being the least of them. Most of my experience has come from using Claude Code so I can only speak from that, but I will at some point pick up 'Pi' when I'm feeling adventurous.

I feel completely free to use any number of subscriptions alongside Claude. Nothing restricts me from utilizing LLM API keys or CLIs pinned to roles in the harness via config. However, that being said, I use this harness confident in using Anthropic models as my main model provider alongside other models (GPT*, GEMINI*, Cursor - Composer 2.5 etc) as more like workhorse models to spread/manage cost on large projects.

If cost is the underlying concern, then you might want to consider your actual cost associated with each task you run on your top frontier model. It might just be that not all of your tasks need to be sent to the most expensive model.

The article's own example makes the point I'm about to make:

To give an example, just doing Typescript type fixes with this model across 50 files cost me $54 this afternoon.

That's all because it ran through the most expensive frontier model for a mechanical task that a cheaper model could easily handle.

What hardly gets mentioned is that most people don't actually measure what each task costs them on a granular level. A lot of the waste comes from running everything through one expensive model without considering breaking the big task into smaller tasks farmed out to cheaper models.

Whether the labs' economics hold is above my paygrade. It costs what it costs. What I can control is my own usage.

Like everyone else leaning in heavily on AI usage, my tokens started running out mid week...sometimes within a couple of days. I had to do something about it or double my spend. So I started tracking my cost per task type a few months ago and it completely changed my workflow. The lowest hanging fruit was the mechanical stuff. Moving that to the cheapest models was a game changer, and much faster to boot. Mid tier models take the workhorse tasks. The frontier heavy hitters are now only used for judgment calls like reviewing and planning. Spend dropped dramatically.

Freeing up all those tokens made me even more ambitious to explore parallel ways of working, to get even more out of what I was already paying for.

reactordev pretty much got it right. In the config, you say "this role uses this model". You decide up front which model handles which kind of task, instead of one model doing everything. All config driven.

You're not choosing a model every time you prompt. It's already set in your config. The main shift in the mind is writing prompts to a role rather than to a specific model.

Orchestration works very well for me, but not in the way most people seem to be pushing for, with middlemen scoring and routing every request. For coding, the routing is mostly solved at the config level. The harness lets you pin models per role, and that covers most of what a per request router promises.

On your actual question though, I think the loop you're describing does break the flow and gets very frustrating, but it's been a long time since I've experienced this.

Three things happened to me in the past few months: I've become cost conscious, I wanted to get more done faster, and I wanted to be able to do a lot more at the same time (in parallel). With that I developed my own workflow that works well for me. It's a config-led setup routed by tiers: cheap fast models for mechanical work (lookups, log reads), mid models for implementing against written specs, strong models for judgement and review. It's config on top of a standard harness, nothing exotic.

For me...my flow state has moved from tackling code line by line to traversing the layers of the entire system design in my mind, and being able to clearly articulate this to a strong model.

Interesting. Read through the comments and the cache issues and context loss issues were already mentioned and they are all good valid points. I'll go from the positive side.

I think the part that resonates most with me is the deterministic rule based routing. It can even double as a policy that you can audit. Configuring which calls would go external and internal.....this could work well for companies who want to enforce certain privacy policies on what data goes to external frontier models vs internal local models. I think the privacy angle is stronger here over the cost angle.

I believe a config-led setup is the way over a model trained on routing. Opus (I'm most familiar with) and I'm sure other models already have the subagent spawning skills down-pat. I basically do a similar routing by config setup but using Claude with some rules and a framework.

So if any routing is needed it might be at the point you've identified....locally on device config-led. Neat. If I get the chance I will check it out and give it a whirl.