The Bitter Lesson of LLM Extensions

https://www.sawyerhood.com/blog/llm-extension

by sawyerjhood • 8 months ago

144 79 8 months ago

dsign• 8 months ago

I don't know, even ChatGPT 5.1 hallucinates API's that don't exist, though it's a step forward in that it also hallucinates the non existence of APIs that exist.

But I reckon that every time that humans have been able to improve their information processing in any way, the world has changed. Even if all we get is to have an LLM be right more times than it is wrong, the world will change again.

vessenes3• 8 months ago

  > "If I could short MCP, I would"

I mean, MCP is hard to work with. But there's a very large set of things that we want a hardened interface to out there - if not MCP, it will be something very like it. In particular, MCP was probably overly complicated at the design phase to deal with the realities of streaming text / tokens back and forth live. That is, it chose not to abstract these realities in exchange for some nice features, and we got a lot of implementation complexity early.

To quote the Systems Bible, any working complex system is only the result of the growth of a working simple system -- MCP seems to me to be right on the edge of what you'd define as a "working simple system" -- but to the extent it's all torn down for something simpler, that thing will inevitably evolve to allow API specifications, API calls, and streaming interaction modes.

Anyway, I'm "neutral" on MCP, which is to say I don't love it. But I don't have a better system in mind, and crucially, because these models still need fine-tuning to deal properly with agent setups, I think it's likely here to stay.

zby3• 8 months ago

MCP is another middleware story - this always fails (hat tip Benetict Evans).

robot-wrangler4• 8 months ago

I always see the hard/complex criticism but find it confusing.. what is the perceived difficulty with MCP at the implementation level? (I do understand the criticism about exhausting tokens with tool-descriptions and stuff, but that's a different challenge)

Doesn't seem like implementation could be more simple. Just JSON-RPC and API stuff. For example the MCP hello-world with python and FastMCP is practically 1-to-1 with a http/web flavored hello-world in flask

vessenes3• 8 months ago

There is a LOT under the surface. custom routes, bidirectional streaming choices (it started as a "local first" protocol). Implementing an endpoint from scratch is not easy, and the spec documentation moves very quickly, and generally doesn't have simple-to-digest updates for implementation.

I haven't looked in a few months, so my information might be a bit out of date, but at the time - if you wanted to use a python server from the modelcontextprotocol GitHub, fine. If you wanted to, say, build a proxy server in rust or golang, you were looking at a set of half-implemented server implementations targeting two-versions-old MCP specs while clients like claude obscure even which endpoints they use for discovery.

It's an immature spec, moderately complicated, and moving really quickly with only a few major 'subscribers' to the server side; I found it challenging to work with.

vidarh8• 8 months ago

The thing is, MCP is little more than another self-descriping API format, and current models can handle most semi-regular API's with just a description and basic tooling. I had Claude interact with my app server via Curl before I decided to just tell it to write an API client instead. I could have told it to implement MCP instead, but now I have a CLI client that I can use as well, and Claude happily uses it with just the --help options.

If you don't already have an API, sure, MCP is a possible choice for that API. But if you have an API, there is decreasing reasons to bother implementing an MPC server the smarter the models are getting vs. just giving it access to your API docs.

btbuildem2• 8 months ago

MCP came in a bit too early, when the conceptual shift of hadn't fully kicked in yet. I see it as a bit of a Horseless Carriage, and I think Skills came in to counter that. My sense is that this will settle into a sort of self-assembling code golem, where ambiguous parts are handled in LLM-space, and clear, well-defined things are handled in code-space.

mkagenius5• 8 months ago

Skills are the actualization of the dream that was set out by ChatGPT Plugins .. But I have a hypothesis that it might actually work now because the models are actually smart enough for it to work.

and earlier Simon Willison argued[1] that Skills are even bigger deal than MCP.

But I do not see as much hype for Skills as it was for MCP - it seems people are in the MCP "inertia" and having no time to shift to Skills.

1. https://simonwillison.net/2025/Oct/16/claude-skills/

zby3• 8 months ago

I still don't get what is special about the skills directory - since like forever I instructed Claud Code - "please read X and do Y" - how skills are different from that?

mkagenius5• 8 months ago

The difference is that the code in the directory (and the markdown) are hardcoded and known to work beforehand.

bavell2• 8 months ago

Not really special, just officially supported and I'm guessing how best to use it baked in via RL. Claude already knows how skills work vs learning your own home-rolled solution.

simonw2• 8 months ago

They're not. They are just a formalization of that pattern, with a very tiny extra feature where the model harness scans that folder on startup and loads some YAML metadata into the system prompt so it knows which ones to read later on.

CuriouslyC• 8 months ago

Skills do something you could already do with folder level readme files and hyperlinks inside source, but in a vendor-locked-in way. Not a fan.

mkagenius5• 8 months ago

It's definitely not vendor locked. For instance, I have made it work with Gemini with Open-Skills[1].

It is after all a collection of instructions and code that any other llm can read and understand and then do a code execution (via tool call / mcp call)

1. Open-Skills: https://github.com/BandarLabs/open-skills

pluralmonad• 8 months ago

They are just text files though. I'm sensitive to vendor lock-in and do not perceive a standard folder structure and bare text files to be that.

vidarh8• 8 months ago

Skills are just markdown files in a folder that any agent that can read files can figure out.

Just tell your non-Claude agent to read your skills directory, and extract the preambles.

robot-wrangler4• 8 months ago

Skills are like the "end-user" version of MCP at best, where MCP is for people building systems. Any other point of view raises a lot of questions.

Aren't skills really just a collection of tagged MCP prompts, config resources, and tools, except with more lock-in since only Claude can use it? About that "agent virtual environment" that runs the scripts.. how is it customized, and.. can it just be a container? Aren't you going to need to ship/bundle dependencies for the tools/libraries those skills require/reference, and at that point why are we avoiding MCP-style docker/npx/uvx again?

Other things that jump out are that skills are supposed to be "composable", yet afaik it's still the case that skills may not explicitly reference other skills. Huge limiting factors IMHO compared to MCP servers that can just use boring inheritance and composition with, you know, programming languages, or composition/grouping with namespacing and such at the server layer. It's unclear how we're going to extend skills, require skills, use remote skills, "deploy" reusable skills etc etc, and answering all these questions gets us most of the way back to MCP!

That said, skills do seem like a potentially useful alternate "view" on the same data/code that MCP is covering. If it really catches on, maybe we'll see skill-to-MCP converters for serious users that want to be able do the normal stuff (like scaling out, testing in isolation, doing stuff without being completely attached to the claude engine forever). Until there's interoperability I personally can't see getting interested though

vidarh8• 8 months ago

There's no lock-in there.

Tell your agent of choice to read the preamble of all the documents in the skills directory, and tell it that when it has a task that matches one of the preambles, it should read the rest of the relevant file for full instructions.

There are far fewer dependencies for skills than for MCP. Even a model that knows nothing about tool use beyond how to run a shell command, and has no support for anything else can figure out skills.

I don't know what you mean regarding explicitly referencing other skills - Claude at least is smart enough that if you reference a skill that isn't even properly registered, it will often start using grep and find to hunt for it to figure out what you meant. I've seen this happen regularly while developing a plugin and having errors in my setup.

fzysingularity2• 8 months ago

I definitely see the value and versatility of Claude Skills (over what MCP is today), but I find the sandboxed execution to be painfully inefficient.

Even if we expect the LLMs to fully resolve the task, it'll heavily rely on I/O and print statements sprinkled across the execution trace to get the job done.

mkagenius5• 8 months ago

but I find the sandboxed execution to be painfully inefficient

sandbox is not mandatory here. You can execute the skills on your host machine too (with some fidgeting) but it's a good practice and probably for the better to get in to the habit of executing code in an isolated environment for security purposes.

sawyerjhood• 8 months ago

I agree with you. I don't see people hyping them and I think a big part of this is that we have sort of hit an LLM fatigue point right now. Also Skills require that your agent can execute arbitrary code which is a bigger buy-in cost if your app doesn't have this already.

vidarh8• 8 months ago

Skills are less exciting because they're effectively documentation that's selectively loaded.

They are a bigger deal in a sense because they remove the need for all the scaffolding MCPs require.

E.g. I needed Claude to work on transcripts from my Fathom account, so I just had it write a CLI script to download them, and then I had it write a SKILL.md, and didn't have to care about wrapping it up into an MCP.

At a client, I needed a way to test their APIs, so I just told Claude Code to pull out the client code from one of their projects and turn it into a CLI, and then write a SKILL.md. And again, no need to care about wrapping it up into an MCP.

But this seems a lot less remarkable, and there's a lot less room to build big complicated projects and tooling around it, and so, sure, people will talk about it less.

stingraycharles2• 8 months ago

Skills are good for context management as everything that happens while executing the skill remains “invisible” to the parent context, but they do inherit the parent context. So it’s pretty effective for a certain set of problems.

MCP is completely different, I don’t understand why people keep comparing the two. A skill cannot connect to your Slack server.

Skills are more similar to sub-agents, the main difference being context inheritance. Sub-agents enable you to set a different system prompt for those which is super useful.

j2kun2• 8 months ago

I don't see how "they improved the models" is related to the bitter lesson. You are still injecting human-level expertise (whether it is by prompts or a structured API) to compensate for the model's failures. A "bitter lesson" would be that the model can do better without any injection, but more compute power, than it could with human interference.

idle_zealot• 8 months ago

A "bitter lesson" would be that the model can do better without any injection, but more compute power, than it could with human interference.

This is what I expected the post to be about before clicking.

marshall300791• 8 months ago

The bitter lesson here is that it all goes back to natural language rather than lower level of abstractions

zby3• 8 months ago

I believe that what we need is treating prompts as stochastic programs and using a special shell for calling them. Claude Code and Codex and other coding agents are like that - now everybody understands that they are not just coding assistants they are a general shell that can use LLM for executing specs. I would like to have this extracted from IDE tools - this is what I am working on in llm-do.

Der_Einzige• 8 months ago

Zero discussion around LLM sampling. How do you leave such a gaping hole in such a written piece? I know it's not AI cus AI wouldn't be that sloppy.

ttkciar4• 8 months ago

Local inference users are all about sampling, but users addicted to commercial inference services are wary of sampling, because they have to pay by the token.

jdblair• 8 months ago

a funny thing happened why i added and emacs eval mcp tool to claude code

https://hachyderm.io/@jdblair/115605988820465712

lerp-io2• 8 months ago

can someone explain to me the difference between MCP and calling a cli tool eg curl or whatever i still don’t understand i’ve been using ai for years now.

virajk_31• 8 months ago

MCP is tool calling with continued context/rich context, tool calling alone will PROBABLY die after single call whereas MCP keeps continuity by design (You can use MCP for tool calling but not vice versa). Hope this help you understand.

lerp-io2• 8 months ago

no still doesn't make sense lmao. you call api and get output, no?

bloppe• 8 months ago

How is this related to the bitter lesson?

ttkciar4• 8 months ago

The author speculates that bigger/smarter models interpreting vague directives to utilize general-function tools will outperform more precise and detailed directives to utilize narrow-function tools:

Granted to use a skill the agent needs to have general purpose access to a computer, but this is the bitter lesson in action. Giving an agent general purpose tools and trusting it to have the ability to use them to accomplish a task might very well be the winning strategy over making specialized tools for every task.

uriegas2• 8 months ago

I was thinking the same thing. Maybe is that at the end the author seems to imply that agentic AI will work simply because models have become better regardless of the way we make them agentic (i.e. MCPs, skills, etc).

ttkciar4• 8 months ago

Well, that's just great.

The academic community has been using the term "skill" for years, to refer to classes of tasks at which LLMs exhibit competence.

Now OpenAI has usurped the term to refer to these inference-guiding .md files.

I'm not looking forward to having to pick through a Google hit list for "LLM skills", figuring out which publications are about skills in the traditional sense and which are about the OpenAI feature. Semantic overload sucks.

How do we deal with this? Start using "competencies" (or similar) in academic papers? Or just resign ourselves to suffering the ambiguity?

Or maybe the OpenAI feature will fall flat and nobody will talk about it at all. That would frankly be the best outcome.

lupire2• 8 months ago

The way NNs and LLMs solve this problem is by processing context and activating middle layer nodes to disambiguate local ambiguities. Have you tried increasing your context window?

jswny• 8 months ago

Skills are an Anthropic feature

ttkciar4• 8 months ago

Whoops, you are right. I misread the article.

touristtam2• 8 months ago

What about open? Or ai? Neither is really what they are offering. Open they are not (weight doesn't count) and don't get me started on that statistical machine they call artificial intelligence. Misleading through and through.

WatchDog• 8 months ago

The most useful LLM "extension" isn't even mentioned in this article, and that is shell use.

An LLM with a shell integration can do anything you need it to.

AIorNot• 8 months ago

A man with a spoon can dig a swimming pool but Id prefer a backhoe

btbuildem2• 8 months ago

sudo apt-get install backhoe

uriegas2• 8 months ago

"I expect us to go back to extending our agents with the most accessible programming language: natural language."

I don't agree with this. Natural language is so ambiguous. At least for software development the hard work is still coming up with clearly defined solutions. There is a reason for why math has its own domain specific language.

euroderf2• 8 months ago

Natural language is so ambiguous.

As a former tech comms guy I will say:

Natural language can be bent into arbitrary precision. Write something, then enter a read-rewrite-reread loop as the devil's advocate (this is key) until it stops being ambiguous or having multiple conceivable interpretations.

Yes with English this process can be a pain in the butt, until you get the hang of it.

stavros• 8 months ago

The problem is that it's very hard to anticipate all possible edge cases. Programming languages force you to do a lot of that work up front, English doesn't. It's the difference between writing Javascript and writing Typescript, except orders of magnitude worse.

thfuran• 8 months ago

You’re never going to make a nontrivial statement in English that you couldn’t find two people who wouldn’t perfectly agree on its meaning. Or probably even a trivial one. Sure, at some point you can say “no, you’re clearly misinterpreting what I’ve said” or “you’re inferring something that wasn’t implied”, but English doesn’t have a formal spec or a reference implementation, so that’s kind of meaningless.

didroe• 8 months ago

The problem is, what's ambiguous or precise is subjective. Your devil's advocate needs to reflect all of the possible readers, and that isn't possible.

There's a good reason we use jargon in professions, or more constrained and less ambiguous languages for maths/coding

jgalt212• 8 months ago

Legalese? It can be both precise and ambiguous, depending on both its construction and the reader's aptitude and comprehension.

n4pw01f2• 8 months ago

This process can be handled by a “turn server”

Was a pain to set up, but you can score the context completion and then if the score is under 98% or something, “ask” clarifying questions of the requesting agent or person or system

zbyforgotp• 8 months ago

That’s why we need progressive hardening: https://github.com/zby/llm-do/blob/main/docs/concept_spec.md

arjie• 8 months ago

Custom GPTs are pretty old, but I recently found a use for them. My wife wanted some meeting note-taking and task recording assistance and I found that making a Custom GPT with a trivial Notion API that was scoped to one page[0] with structure that was encoded in the API was a quick couple-hour thing that unlocked a lot of utility for her (the default Notion MCP is "too broad"). It helped that this Custom GPT sits in her ChatGPT UI and she doesn't have to have another app or whatever to make it work.

We liked it quite a bit, but it led to some funny things. We use Reminders to keep our home to-do lists, hers and mine in one list with two sections. I wanted to take this existing flow we had and make it work with a Custom GPT. It's practically impossible because Reminders:

* doesn't have a good API through EventKit

* requires a pop-up permission grant in the UI

So in the end, I did end up making somewhat of an MCP server for it, running it on an old Macbook Pro I had and then sticking Amphetamine on in closed-lid display-sleep mode hooked up to my Tailnet and exposed via a Cloudflare tunnel so that we could use ChatGPT to interact with the thing. Yes, you can see how insane that whole thing is. But there's quite a lot of value to have your AI agent just be the one thing.

0: https://wiki.roshangeorge.dev/w/Blog/2025-10-17/Custom_GPTs

metalrain• 8 months ago

Skills.md will in time have same problem as MCP, they will bloat the context. I wonder if we could just have the scripts without the descriptions and LLM would have been trained to search the most useful things in specific folder.

crackalamoo• 8 months ago

This seems like a solvable engineering problem. For example, you could have a lightweight subagent with its own context for reading the skills and determining which to use

iamcreasy• 8 months ago

ChatGPT apps, announced this month, feels a lot like original ChatGPT Plugin announced 3 years back. The only difference is how plugin are invoked. For ChatGPT plugin, we have to choose one from a drop down, and for apps - we could just include a plugin name in prompt.

Is there any other difference in the end-user side?

The Bitter Lesson of LLM Extensions

Related Stories