LLMs aren't remotely like compilers or power tools

https://blainehansen.me/post/llms-not-power-tools/

by blainehansen • 3 days ago

38 50 3 days ago

This sounds like someone who hasn't used compilers for long. I remember plenty of fun compiler undefined behavior that had essentially non-deterministic outcomes.

And actually using power tools in the real world is pretty uncertain, try running a straight starting cut with a multitool against wood grain and then come back and talk to me.

lou1306• 3 days ago

Sure, but those were likely bugs in the compiler's codebase that in principle could be isolated and fixed. With an LLM there is no bug and there is no fixing procedure: all outcomes are context-driven and (more or less) nondeterministic by design. I don't mean this in a dismissive way, but there is clearly a difference there.

kilpikaarna• 3 days ago

And actually using power tools in the real world is pretty uncertain, try running a straight starting cut with a multitool against wood grain and then come back and talk to me.

I especially liked table saws and backhoes as examples of _automation_.

Krssst• 3 days ago

Another way to think about this: if the compiler has a miscompilation, the user is usually not blamed for the mistake (unless gross lack of QA). If the LLM makes a mistake, the user gets full blame. Thus, slowly combing over the output is required.

juxtapose• 3 days ago

That's not a good counterexample.

Formal languages are called 'formal' because they have rigorously defined semantics. The 'undefined' parts, if anything, reveal that most of the expressible sentences have clearly defined semantics, rather than lack of them.

In addition, the semantics of 'undefined' behaviors is not 'non-deterministic' in the literal sense, since it is, well, undefined.

blainehansen5• 1 day ago

Original author here. I think words like "intentionally designed to be completely predictable" signal my understanding that this is only an intent rather than a perfect reality. I was trusting my audience to understand that obvious nuance.

camkego• 3 days ago

I've got to give the author credit, it seems like this is written without an LLM. But, there seems to be a lot that is arguable.

For example, the article says "Natural language is the interface of last resort."

But, is that actually true? Isn't the AI boom happening because N/L is truly available as an interface?

Anyway, please keep writing and asking tough questions.

trescenzi• 2 days ago

Natural language is an imprecise interface. It really ideally is the interface of last resort. If you have a straight line from problem to solution you do not need natural language. It’s only if there isn’t one that you do. And in that case the same sentence can mean different things to different people even those who are native speakers. The most obvious example to me is something like “next weekend” which is a term that can mean different things to different people at different times. But it’s a deeper problem than that.

The AI boom is happening because people what to believe that natural language is precise. Not because it’s actually an effective way to interface with everything.

throw3108225• 2 days ago

the article says "Natural language is the interface of last resort."

I wonder why he decided to express this thought in natural language though. Should have tried formalising it.

blainehansen5• 1 day ago

Original author here. Yes, if I had the capability of writing a computer program or a formal proof that could stand in for this essay full of opinions and metaphors and heuristics, I would have.

It's fully general, but it's also furiously ambiguous. Humans fall back to natural language when no other more precise way to describe what we want is available to us, either because it doesn't exist or we don't know it comfortably. We draw diagrams and blueprints and write math equations for a reason!

smcg2• 2 days ago

I think it is true, but I think the author underestimates how difficult it is for us to reason in ways other than natural language. We have to be taught it and we have to practice it.

vivzkestrel• 2 days ago

natural language will never be 100% deterministic

blainehansen5• 1 day ago

Hey original author here. Natural language is always easy and available as an interface, but it's extremely low fidelity. It's not easy to come up with better interfaces for tasks, but when we do they tend to be much better (as in more clear, faster, higher quality, etc).

pjmlp• 3 days ago

The author misses on machine learning compiler optimisation passes, dynamic compilers (aka JIT), profile guided data optimization passes, and that LLM driven compilers is becoming a hot topic in compiler research, see recent SIGPLA or PLDI papers.

Additionally they also miss on low-code/no-code orchestation tools like Boomi, Workato, Opal and co, where programming is moving from drawing boxes, into agent prompting.

Those are also programming from all practical business purposes.

mycall• 3 days ago

Is there not a way to use a transformer that is deterministic in rounds as it is just maths? I thought it was temperature which randomized results but perhaps it is also something else.

Of course, the content returned is non-deterministic.

suprjami2• 3 days ago

If you specify the random seed and set temperature to zero, you'll always get the same response. As you said, it's just math.

That doesn't mean the response is correct though. You've just hard-locked the model to one possible response selection.

If you and I do this but we use different seeds, we still have no way to tell whose response will be better for every question or even a class of questions.

There will be responses where the model is outright wrong and we want to change seed or introduce some randomness by raising temperature, in the hope the model answers correctly next time.

Now we're back to running with a random seed and high temperature.

inigyou7• 2 days ago

Actually you don't get the same response. Batching and different evaluation strategies can influence, say, the order that numbers are added together, which changes the output.

suprjami2• 3 days ago

You had me up until you equated LLM usage to unreliable human delegation.

No human being is going to genuinely suggest to glue cheese onto a pizza to stop it sliding off.

These things aren't people. Stop anthropomorphizing a computer program. They'll make all the mistakes humans can make, as well as a new class of mistakes because they're just fancy next word prediction with no lived experience.

rjrje• 3 days ago

Indeed.

Just to add - I’m working on something very novel and LLM’s are absolutely useless at them. It’s actually comical to see the outputs - when you work on novel stuff you quickly see what these models actually are.

inigyou7• 2 days ago

Apparently if you show one the recently found counterexample to the Jacobian conjecture, it spins in circles double-checking it because it knows the conjecture is true.

throw3108225• 2 days ago

No human being is going to genuinely suggest to glue cheese onto a pizza to stop it sliding off.

That mistake was made more than two years ago by whatever experimental version of Google's AI overview (which has strict performance requirement- i.e. needs to answer within a split-second) was up at the time. In other words, it was a very small and primitive system specialised in spitting answers as quickly as possible without a second thought. I hope you realise that basing your assessment of what LLMs can or can't do on that example is not much better than suggesting to put glue on pizza. A mistake that, if we were to adopt your reasoning, would in turn set a hard limit to the analytic skills of all humanity.

bluefirebrand2• 2 days ago

It's a clear illustration that these systems are not oracles. No matter how much they've improved since then, we should assume they are still capable of making hilariously big mistakes. Only now the obvious mistakes are fixed and the big mistakes will be a lot more subtle and a lot harder to catch.

jaredcwhite• 2 days ago

Nice try with the gaslighting. Google's AI "summaries" get things wrong all the time even now.

blainehansen5• 1 day ago

Hey, original author here.

But fundamentally any metaphor will be inadequate, LLMs are too weird. It doesn't make sense to compare a transformer-based deep neural network that autoregressively generates language tokens based on their embedding in a high-dimensional semantic space to really anything that's ever existed.

You might not have read the full post ha

vrighter• 3 days ago

i recently heard a colleague telling another "vscode can't really help you when writing yaml files". I said that the yaml lsp supports loading a json schema, all you need is just one comment at the top to point the lsp to it, then you get reliable error reporting with red squigglies on every single keystroke.

He just shrugged and said "oh i just use ai".

What takes me one keystroke takes him a full minute. And he didn't even consider the use of actual tools

pennomi• 2 days ago

The very existence of a “do anything” tool causes humans to forgo learning the multitude of tools we already have. I suspect there will be no next generation of software engineers.

inigyou7• 2 days ago

How many buttons in your IDE's toolbar do you use?

Oh wait - that's an anachronism. The older generation of IDEs, like VS and Eclipse, had toolbars. VSCode doesn't have one.

david-gpu3• 2 days ago

The existence of a "do anything" IDE causes humans to forgo learning the multitude of tools we already have. I suspect there will be no next generation of software engineers.

People said stuff like that in the 90s. I was there.

thewhitetulip• 2 days ago

I've seen people comitting to git using AI lol

the_real_cher• 2 days ago

This person is taking things too literally.

By power tool we simply mean speeds up work.

And by that definition LLMs are absolutely a power tool of epic proprtions!

If you ever coded before LLMs existed its like night and day.

inigyou7• 2 days ago

Evidence from listening to slopcoders speak says they feel it's night and day. Evidence from observing reality shows a 20% speed increase at best and 50% speed decrease at worst and almost always a significant reduction in quality.

smcg2• 2 days ago

Twitter is a rather crude place, but it can be succinct. An exchange I saw recently:

A: "I spent more tokens than the entire OpenCode team combined."

B: "Why aren't you more successful?"

Jemaclus• 2 days ago

I was also skeptical, but I started measuring quality (I'm using rework as a rough metric) and quantity (PR throughput), and I've found that, on average, LLMs are 60% worse at writing code (meaning they rework at a rate 60% higher than humans) but my team produce PRs about 2.4x faster. So objectively my team's numbers don't match up with yours: our speed is about 2.5x increase _on average_ and a 60% reduction _on average_ in quality. Some basic math suggests that our ROI is something around 50-60% improvement over just humans.

I'm not convinced that throwing more money (i.e. tokens) at the problem would make us any more efficient, but 1.5x improvement ain't nothing.

And just to address the "night and day" bit.. if you gave a construction foreman a machine that made his team 20% more productive, he'd take that all day long and tell everyone it's "night and day." I'm not sure what your threshold is, but +20% isn't nothing either. That foreman would hit more deadlines than not, and get more bonuses than not.

blainehansen5• 1 day ago

Original author here. Metaphors become real, and they shape the way people think. I think it's dangerous to compare LLMs to deterministic systems because it misleads people into treating this situation we're in like other kinds of automation, which it absolutely isn't.

Also, if you want me to believe that LLMs have made you that much more productive, please share some links for useful things you've built recently using them. I'm thrilled if actual things can be made, but I haven't seen it yet.

jdmdndndn• 2 days ago

People here take things too literally....

Yes an llm is not a compiler. But both speed things up... That is the point...

An llm is not a power tool but they both speed things up!

I don't care about determinism for something that is a simulacrum of though ... That is a feature not a flaw...

Finally we are getting better at this ... I get that we are going to lose our jobs a devs and I get that it is scary.. But honestly good riddance.. There is much more to do in this world that we can do after we just let go and accept that what we knew is gone ...

Just to be clear .. A correction Will likely come for the AI industry ... But a correction happened 25 years ago and the internet is still here :)

Just remember to not take life too seriously... You'll never get out alive :P

LLMs aren't remotely like compilers or power tools

Related Stories