Not everything should cost a token: the case for deterministic AI

https://www.vybe.build/blog/learn-what-not-to-tokenize

by marwann • 16 days ago

24 18 16 days ago

eulgro• 16 days ago

loading a few thousand tokens of raw JSON into a context window, asking a language model to reformat data it did not need to reason about...
That trap catches almost everyone right now.

Has everyone really become so stupid now? I wouldn't even have thought of doing such a horrible thing as using AI to format JSON...

Needless to say I stopped reading there.

bigstrat2003• 16 days ago

Has everyone really become so stupid now?

Not everyone. But unfortunately, yes, very many people have.

marwann2• 16 days ago

Has everyone really become so stupid now?

I wouldn't mistake stupidity for ignorance. A lot of "tech-sensitive", yet non-technical people are using agents/vibe coding these days. Their level of understanding is very basic ("JSON is structured data", "I need to reformat this to fit my need"), but they wouldn't get that the best way of doing things does not necessitate burning tokens.

t-writescode• 16 days ago

"wire up an agent to do":

  * pull a metrics API every morning
  * reshape JSON
  * drop results into a table

This means they:

  * know there's an API that they need to call,
    likely a specific endpoint
  * know there's a database
  * know the schema of the database
  * know where the data should go into the database

Or perhaps they re-derive this every single day.

Every day, they search for an enpdoint and call it. Every day, they check all the databases that the agent knows about. Every day, they read the full schema of those databases to guess where the data should go. Every day, they invent the SQL to insert this new data into the database in a format the database is expecting.

There are two possibilities:

  * this person exists and is common
  * this person does not exist or is very uncommon.

If you're this person, hey! Welcome to beginning to program and learn about the deeper work that goes into making things fast and easy. It's gonna be really weird, but you might find it rewarding! And, you can mostly keep to being a vibe coder, genuinely; but you're gonna wanna change "hey go do this" to "write me a program that can do this". It may be a higher starting cost; but it'll pay dividends as the weeks go by. Like, dollars a day to pennies a day in server or power costs.

But I tend to believe that this person isn't very common (if you're the person that does this, no big deal. You're new to programming. There's a whole lot to learn and you've already started. That's awesome!)

Instead, though, we've got a person that:

  * knows what endpoint they need to talk to
  * knows what the schema looks like
  * probably knows how to write SQL (or doesn't, in which case,
    oh god, they're letting the bot write and execute
    brand new SQL everry daOh god oh god.)

But they still let a bot do it? Every single day? Do they work for a company that subscribes to and/or do they subscribe to a token-maxing philosophy? Because they must know this much agent interaction is going to be not only expensive but also really slow. 10s of seconds vs, what, 3ms of parsing and maybe 100ms of waiting for API and DB communication on the high end?

Maybe you're forced into doing it due to internal office politics and prioritization - this article is going to be great for explaining to your boss why it's a bad idea to keep doing what you're doing. Otherwise, none of this should be changing very often, and this codepath should be, all in all, quite cheap to write - with one exception.

The only situation I could see where this approach the article suggests "lots of people are doing" makes sense and is a reasonable thing an engineer would have done is if the DB and the API endpoints are constantly changing, so they need to rederive over and over again.

But is that really where we're going as an industry? Random, dynamically changing API endpoints? I sure hope not.

lepolas• 16 days ago

In my experience this person exists and is common, and someone has set up an MCP for them.

in which case, oh god, they're letting the bot write and execute brand new SQL everry daOh god oh god

Yeah, also this.

argee• 16 days ago

I just saw this video today [0], and I was thinking…this girl must be made of tokens if this is how she works.

[0] https://youtu.be/uGwDuvSqgYI?is=CRrIT31HITipIHXi

marwann2• 16 days ago

I believe many non-technical people are spinning up AI agents nowadays without having a basic understanding of what happens behind the scenes, and usually, with no interest in learning. They prompt, watch the agent work and see the magic, without questioning the structure even a little.

I don't think we should blame them, as they are non-technical, but either educate them or put guardrails at the system prompt level.

sly010• 16 days ago

Deterministic AI.

Aka, a computer program.

didgeoridoo• 16 days ago

I built a game for a workshop at the AI Engineer conference last week. The idea was to try to optimize a CLI that your agent could use to call a remote service to process and fulfill complex natural language coffee orders.

Within 3 minutes, everyone’s agents had reverse engineered the Markov chain I used on the server to generate the order text, and they all began to write deterministic parsers to churn through them. The order completion time dropped to double digit milliseconds, and then the agents started fighting to optimize their parsers to drive it down basically to pure network latency.

It was hilarious, and taught a good lesson about leverage. I expected agents to drive the CLI, not figure out how to not need to drive it at all.

htrp• 15 days ago

is this written up anywhere?

sounds like a fascinating case study!

chmod775• 16 days ago

A team I talked to recently wired up an agent to do something simple: pull a metrics API every morning, reshape the JSON, and drop the result into a table. Clean idea. It worked on day one.

So a team of people with access to "intelligent" help did not immediately figure out that they could just have the model write a small script that does the job perfectly every time? That's "call a doctor" level of stupid that should be impossible in vivo. I'll believe a lot, but not this. The rest of the article is not much better - only an overexcitable LLM could believe anything there constitutes a deep insight worth writing down.

Do we need a new category for this sort of made-up, probably LLM-generated, garbage? Is this AI brainrot blogspam engineered to be upvoted by people who don't read the contents?

How do you not die of shame publishing something like this on your company blog?

xg15• 16 days ago

Yes, from a programmer's perspective, this is insane. But the team may have been made up of people without coding experience who maybe didn't even know what a script was. I imagine there are a lot of people right now who "know AI" and not much else, for whom this would be a genuinely new insight...

Note that even before AI, there were a lot of manual data munging jobs that could famously be replaced by "a very small shell script".

I worry, this might become worse if we see the emergence of "100% nontechnical tech startups" where no one in the entire org knows how to code - because why would you, we have AI for that...

unparagoned• 15 days ago

People and teams absolutely are doing really dumb and stupid stuff. And they could learn a lot from this article

clickety_clack• 15 days ago

There’s a lot of teams out there who had to find a way to deploy agents to keep their jobs while working on problems like this that didn’t need them.

doruktt• 16 days ago

We are building an infrastructure coop for NGOs with a wide range of open-source tooling. What we are working on is an agentic tool to build tooling on NixOS. So, for best performance, we split up the agentic interface into a deterministic MCP to generate flakes from templates, build contracts to connect different tools, etc. But there are some tasks where AI is perfect for the job, so it goes down to the model flow to extract tooling from Docker Compose files, read errors to come up with solutions, etc. I believe this approach is both economical and logical to follow. But as I mentioned, we are in the alpha stage, so we will see how it comes out.

umpalumpaaa• 15 days ago

Reads like an ad

_superposition_• 15 days ago

I thought this was all common sense. Coming from a background of workflow automation I've simply used AI to solve steps that could not be solved before. The example may have made sense if the input was freeform text like an email etc, but surely not json.

jarodrh• 14 days ago

We've all been there where we choose the easy least efficient route just cuz (laziness...can afford it), but we've mostly done this consciously. The example in the article however does take this to another level and understandably so, as it's so easy now to take the easier route when you have no idea what you've requested is actually doing under the hood. I'm sure if you explained it to them, they'd immediately go...oooh I'm building my car every time I need to drive to work. As many comments have already stated, it's programming 101.

It's almost no different to using an expensive frontier model to do a general scouting task. Use the right tool for the job. A hammer can do most things, but should it be used for most things...

I think one of the most important non-technical things everyone should strive for is a mental awareness for figuring out what the right question is.

"Exactly!" said Deep Thought. "So once you do know what the question actually is, you'll know what the answer means."

Not everything should cost a token: the case for deterministic AI

Related Stories