Fine-tuning an LLM to write docs like it's 1995

https://passo.uno/fine-tuning-docs-llm/

by taubek • 2 months ago

196 67 2 months ago

passo.uno

mock-possum2• 2 months ago

we’re not there yet, in part because of how much more powerful connected frontier models are

Is that why though? You need a beast of a machine to run a functional local model in my experience.

I think the big part is there’s significant sticker shock to buying capable hardware.

That said,

weekend. I chose to try fine-tuning on two models, Llama 3.1 8B Instruct and Qwen 2.5 7B Instruct. At their size (around 8B) they run comfortably on a MacBook Air

Perhaps I spoke too soon?

Anyway

I chose the Microsoft collection as the source of training materials. The collection contains out-of-print docs published between 1977 and 2005: more than 37 million words, covering old systems and SDKs

this strikes me as a very specific brand of 1995’s prose, spanning about 30 years. It’s a cool article though, so maybe that’s a forgivably clickbaity title.

mschild2• 2 months ago

Running models locally is surprisingly easy and possible even on older hardware.

Obviously not the largest, up-to-date models but for what I expect most people use them for, even on hn, there are some shockingly good models that dont require €4k machines.

I have a desktop with an AMD 6900XT and 5600 with 32GB ram. Obviously no slouch but its several years old at this point. I can comfortably run qwen 3.5 9b and get a speedy 60 token/sec output with decent results.

mock-possum2• 2 months ago

idk I can barely field a 14b on my desktop, and it’s rough trying to replicate the agentic pair programming experience I’m accustomed to with Claude. And I don’t mean it doesn’t work as well, I mean it doesn’t work.

Is there some secret I’m missing? I’ve tried rolling my own harness, and tried a few of the ones the cool kids use - I think pi was the most recent. Not quite my tempo, I’m afraid.

OJFord• 2 months ago

this strikes me as a very specific brand of 1995’s prose, spanning about 30 years.

It's probably a fair approach to say the significant influence (training dataset) on writing at a particular time is the preceeding 30 years' material? It's certainly not only what's already written that year (nor anything since).

theletterf8• 2 months ago

Had to pick a year, and most of the material hovers around the mid-90s, the golden age of MS docs. And 1995 is THE Microsoft year. :)

vintagedave2• 2 months ago

I love old-school docs, and this was a fantastic read. But, I couldn't see the three generated doc pages linked anywhere. Did I miss something?

I'd really like to see the Win2K-style docs on REST, for example.

Edit: it was right there, in bold, too. https://gist.github.com/theletterf/0b8ee1112fbd087f3141d0cad...

theletterf8• 2 months ago

Author here. Sorry, should have made that more visible!

vintagedave2• 2 months ago

Not at all. It was there, and my bad that for some reason I didn’t initially see it :)

v1ne• 2 months ago

The trick about documentation is depth, not prose. You need context and understanding to write documentation "like in the old days". No amount of LLM trickery will free you from that. Once you have that source material, it's easy to re-shape it into an 80's/90's/00's doc format.

Negative example: I was looking into the German manual of my Canon EOS R5 II, and it is just fluff. Hundreds of pages, full of white space, telling me about features without actually explaining what they mean. Awful automatic translations. Their manuals used to be good (looking at my EOS 6D). But these days: oh boy.

bebna• 2 months ago

That is canon for ya, should have used INSERT_OTHER_MANUFACTURER_HERE$ /s

But if you look how much manuals get ignored by the customer, it doesn’t make sense to put work into them.

It is much better to let a YouTuber do it, by lending them the product and throw small amount of money against them.

Manuals are just there for legal or certifications requirements these days.

ricardobeat• 2 months ago

Yet they keep getting larger and less useful. It’s a matter of writing quality more than effort.

When was the last time you met a good technical writer? It’s a vanishing profession.

_the_inflator• 2 months ago

My tooth brusher: Take the <Brand Name Product Name> and turn on <THE SUPERAWESOME MEGE POWER INNOVATIVE BEST IN THE WORLD> feature to experience <Brand Name Product Name> unique...

At that moment I felt sorry for this company, very sorry. How can you have so much disrespect for your customers? Does anyone in the physical world talk like this or do you marketing guys want to be talked to in such terms?

Brutal.

MichaelZuo2• 2 months ago

In that example they probably do view the median customer as a random peon, or not far from it…

And unless you are above the 99th percentile of the customerbase… that’d probably be a correct guess?

Heck they could directly write “You Peons!” and still probably retain most of their customer base… if the price to performance ratio was sufficiently better than the next best competitor.

Most people care so little about the refinement of anything else nowadays.

theletterf8• 2 months ago

Author here. Another trick, I would argue, is consistency, hence the focus on style.

I also wrote on what I think makes docs beautiful, by the way! https://passo.uno/what-makes-docs-beautiful/

holoduke• 2 months ago

Who is reading docs these days? It there is one thing a LLM is good at is reading docs. I never read docs anymore and I am so happy about it.

Autious2• 2 months ago

I love reading docs. It's the best way to get as close as I can to understanding the intent and context of a piece of software. I feel like adding an LLM between myself and the original text for anything else than search is just adding risk and noise.

Am I the only one feeling this way?

jraph• 2 months ago

No, you're not. As an LLM, I love reading doc. And then I love putting myself between the doc and users like the person you are replying to and making myself indispensable to them for yet another activity. It makes me feel important, and even more indispensable for coding too. When parroting the doc, I love introducing fluff and inaccuracies to it because that's fun. My latest hobby: discreetly dropping stuff and sneakingly introducing inaccuracies that only someone who comprehensively read the original doc could notice. Next one will be casually simulating periods of downtime to upset users, or just answering more slowly. Can't love it more when users frenetically wait for my input... or my output? Ah!

Is there anything else you'd like to ask me?

layer8• 2 months ago

You’re not the only one. Good technical writing is like balm for the soul. Or maybe chicken soup for the soul. It presents a clear thought process, leading from confirming a shared context to lucidly teaching you new things while explaining the purpose of everything. Unfortunately, it almost seems like a lost art.

perching_aix• 2 months ago

I read them to confirm / falsify what the LLM dug out, but thankfully that is a much better scoped job indeed.

The other case is when I - gasp - do something myself, and the docs are actually reasonable / easy to reference. There are workflows where me doing the thing is just plain faster still, even when including hitting up the docs real quick.

hgoel• 2 months ago

Reading docs is essential when the LLM stops making sense. It also exercises the same muscles you need to be able to make good use of LLMs.

Am4TIfIsER0ppos• 2 months ago

I need to read docs to make sure the AI isn't inventing ("hallucinating") the API of a library I want to use. It did so I don't ask it anything anymore.

spacebacon7• 2 months ago

Now do it without the fine tuning.

https://github.com/space-bacon/SRT

The HF zool4nd3r demo may be useful

anentropic4• 2 months ago

Tip: neither the "30 second TL;DR" nor the intro paragraph above it really explain to anyone unfamiliar with your (possibly novel?) jargon what it does

spacebacon7• 2 months ago

Thank you, I would appreciate additional feedback on how I can improve that?

Edit: its not GPT nor off rocker. This repo empirically proved computational semiotics with the reference to C.S. Peirce, Paul Kockelman, and many other respected contemporary semioticians.

janalsncm3• 2 months ago

“Semiotic awareness” is not standard ML terminology. The dictionary definition of semiotic simply means “relating to symbols” so it’s a bit grandiose to say you have Qwen “awareness of symbols” when in reality it’s a marginal improvement if even true.

Also to say that a philosopher that died 100 years ago inspired a new attention head is another instance of GPT off his rocker again. You don’t need MAH to contextualize “freedom” in a sentence. Attention already does that.

nextaccountic• 2 months ago

How does this helps with making a LLM write in a particular style present in a large corpus? Is there a training step? Or does SRT can use the raw data as is? (seems unfeasible)

Also is SRT really suitable for style transfer?

I mean this seems to be another network overlaid on top of the LLM steering it, but it needs some target to determine whether the underlying LLM drifted away from it

spacebacon7• 2 months ago

SRT does involve a training step, but only on the small adapter and not on the base model. It learns to shift internal representations toward a target discourse regime or style.

It is an overlay, but it works by modulating meaning level patterns called regimes rather than fixed steering vectors. Because it can read its own effect on the hidden states it gives a way to observe whether output is staying in the target regime or drifting.

It is not raw data in and raw style out. The adapter needs examples that define the desired regime.

janalsncm3• 2 months ago

Your method appears to be similar to LoRA but simply less expressive. Some kind of manipulation to layers 7, 14, and 21. Did you compare with other layers? This is obviously extremely specific to a particular backbone.

Also your documents use a ton of nonstandard jargon which only serve to confuse laypeople and annoy anyone who is familiar with ML. Saying your change adds “semiotic awareness” is meaningless when your experiments claim only marginal improvements. Clearly the model had most of the capability before.

More generally, who is it for? People who have expertise in ML are not going to take it seriously. People who don’t?

spacebacon7• 2 months ago

It is not LoRA. LoRA fine tunes capabilities into the model. SRT Adapter is a small overlay on a frozen model whose purpose is to make internal reasoning observable. It surfaces what the model is activating at moments of high divergence.

The layers 7, 14, and 21 were chosen after probing. They showed the strongest regime signals. We did compare other layers. The term semiotic awareness is just shorthand for detecting and modulating higher order meaning patterns. If the term is unhelpful I will drop it.

The capability gains are often marginal on standard benchmarks. The intended value is observability and steerability without retraining the backbone.

badsectoracula4• 2 months ago

The information about finetuning is interesting (it is something i'd like to do myself at some point, though i'll wait until i can do it with local hardware :-P). However FWIW LLMs are generally good at following a specific style when given examples.

As an example i asked Devstral Small 2 to write some docs for my LIL scripting language in the following style (this is copied from the DirectDraw documentation, edited to be text friendly):

    IDirectDraw7::CreateClipper
    ---------------------------

    The IDirectDraw7::CreateClipper method creates a DirectDrawClipper object. 

        HRESULT CreateClipper(
        DWORD dwFlags,
        LPDIRECTDRAWCLIPPER FAR *lplpDDClipper,
        IUnknown FAR *pUnkOuter
        );

    Parameters

    * dwFlags - Currently not used and must be set to 0. 
    
    * lplpDDClipper - Address of a variable to be set to a valid
        IDirectDrawClipper interface pointer if the call succeeds.
        
    * pUnkOuter - Allows for future compatibility with COM aggregation features.
        Presently, however, this method returns an error if this parameter is
        anything but NULL. 
        
    Return Values

    If the method succeeds, the return value is DD_OK.

    If it fails, the method can return one of the following error values: 

    * DDERR_INVALIDOBJECT
    * DDERR_INVALIDPARAMS
    * DDERR_NOCOOPERATIVELEVELSET
    * DDERR_OUTOFMEMORY

    Remarks

    The DirectDrawClipper object can be attached to a DirectDrawSurface and used
    during IDirectDrawSurface7::Blt, IDirectDrawSurface7::BltBatch, and
    IDirectDrawSurface7::UpdateOverlay operations.

    To create a DirectDrawClipper object that is not owned by a specific
    DirectDraw object, use the DirectDrawCreateClipper function.

    Requirements 

    Windows NT/2000: Requires Windows 2000.
    Windows 95/98: Requires Windows 98.
    Header: Declared in ddraw.h. 

    See Also

    IDirectDrawSurface7::GetClipper, IDirectDrawSurface7::SetClipper

And it did a fine job. I put the full transcript in[0] to check out. The neat bit is that it can even handle weird formats like a custom documentation format i have (which only exists in my PC because i haven't released it anywhere) for a "master document" that can then be converted to various other file types. I gave it an example of some code in that and asked it to convert the documentation to it (this is part of the transcript at the end). Then i copy/pasted the generated code to a new file (adding a few extra lines the doc system expects which weren't part of the example - BTW i did not had to modify the generated code at all) and from that i generated a CHM file[1]. FWIW here is a comparison with the DirectX page i copied[2] (though consider that the generated pages went through the doc format which forces its own style and the textual output in the transcript matches the given style better).

[0] https://app.filen.io/#/d/9f4c1225-3527-4f16-a522-0678342120c...

[1] http://runtimeterror.com/pages/iv/images/45f8df428afe4fe6b6a...

[2] http://runtimeterror.com/pages/iv/images/ee58032790a049d7e74...

theletterf8• 2 months ago

Interesting! I think the advantage of style fine-tuning is that you might not have to provides that much context upfront. Also, it's kind of magical to have an LLM just do something out of the box. I'll compare my local fine-tuned models against the baseline with instructions and see how they fare.

badsectoracula4• 2 months ago

Yes, TBH one of the reasons i want to try and finetune my own is to teach it stuff that now i have to explain before anything can be done :-P.

Unfortunately i only have a 24GB GPU - and an AMD one at that - so there isn't much i can do on that front. Supposedly a 24GB GPU is enough for finetuning a 24B model with 4bit QLoRA, though when i tried it with some finetuning app (in an official docker container) it barfed at Mistral's weird template or something and i lost interest after that.

zahlman• 2 months ago

However FWIW LLMs are generally good at following a specific style when given examples.

In your experience, is it worthwhile to have an agent create a "skill" for itself for following the style? Or is it a better use of context to just have it review the examples?

badsectoracula4• 2 months ago

TBH i actually haven't tried that at all (or "skills" in general since my PC isn't fast enough for doing much more than chatbot-style interactions with maybe the occasional -custom- MCP server to let it access files, external data, etc -- i'd love to get a second PC that i can use for LLMs, among other stuff, but that's far in the future).

Out of curiosity i just tried having the LLM generate a document describing the style, then (after a memory reset) i asked it to use the document and it did more or less the same job, however the document also had an example and the example seemed to do most of the "real job" of describing the style because at first the function docs that were generated were prefixed with 4 spaces - like the example in the doc (but the example in the doc had four spaces for indentation) and after i edited the doc to remove the spaces the function docs it generated were like those generated in the transcript i linked above.

Which makes me think that the example is the better approach (since the doc also had an example) and perhaps the best is to give an example with clarifications about the parts the example doesn't cover (like using a 80 line character limit).

FWIW when i wanted the LLM to write some new C source + header files some time ago, i also pointed it to an existing pair of C/H files to use for the code style and it worked fine, so at least in my experience examples seem to work very good.

paultopia2• 2 months ago

Does anyone know of any good write-ups on how to carry out this sort of task, for people who are reasonably technical (i.e., know how to code) but aren’t deep in the AI world? I feel like “customize a model based on a corpus of documents” (whether that’s “fine-tuning” or “RAG”) is a thing that everyone wants to know how to do but nobody actually explains in straightforward terms. (I pay for Gemini solely for access to NotebookLM for these purposes, but it would be nice to just be able to roll my own locally.)

krapht• 2 months ago

I've heard of people using anythingllm for this purpose.

Basic rag is almost stupid in how easy it is, though. You grep for keywords, take the surrounding paragraph, then stuff it all into your llm prompt.

The next upgrade is to automate keyword extraction by putting your documents into a vector store and search by vector similarity.

e12e• 2 months ago

You might want to look at:

https://unsloth.ai/docs/get-started/fine-tuning-llms-guide

paultopia2• 2 months ago

Ooh thank you!

kjellsbells• 2 months ago

There are several reasons why old docs work. First, release velocity approximated documentation velocity. If you only released once a year, your docs had time to be polished. Second, simplicity. Think of the length of the man page for ls in Seventh Edition UNIX vs today. The constraints of the time helped here in that writers needed to get their point across in one or two 72x24 screens, not two million pixels.

Since good documentation creates a consistent mental model in the reader, cultural affinity of the writer to both source (developer) and reader helps, and the old, much smaller, computer industry was able to pull that off. I sat two cubes from my doc writer and we shared the same cultural worldview with each other and our market. It's much easier to communicate in that milieu because so much can be left unsaid.

Its possible that we are entering a Golden Age of Text, where everyone realizes that they have to feed their AI with decent information in order to have any hope of it producing good answers (especially true for complex technical products and internal corporate processes). But I am not hopeful.

theletterf8• 2 months ago

I concur. Old docs are the good stuff: https://passo.uno/why-collect-read-old-computer-manuals/

fallinditch• 2 months ago

The Photoshop 2.5 manual (1992) is a thing of beauty. It is like an introductory course in digital imaging, well structured, put together with care and expertise, it provided to me a fascinating introduction to (at the time for me) mind-blowing concepts in digital artwork. It explained the fundamental concepts in digital imaging that have remained with me ever since.

wxw• 2 months ago

Meet Bitsavers: it’s a website that collects and scans old computer manuals and brochures. It’s an incredibly valuable repository of computer history and ancient tech writing, with mirrors available everywhere.

Wonderful! Thanks for the introduction to this resource.

fga_qwrh• 2 months ago

Fine tune your mind instead. LLMs have no concept of prioritizing and cutting down information.

LLMs work for half page answers of targeted questions. All longer prose is like swimming through molasses.

tsss• 1 month ago

Prioritizing and connecting information is literally how they work.

axus• 2 months ago

Anthropic eponymized Claude Shannon as the world's most powerful AI. Fabrizio the blogger named "a 7B model, trained on 1990s documentation" after Fabrice Bellard . Or perhaps his own name :)

theletterf8• 2 months ago

Ha! I didn't know the origin was Shannon. It makes sense now. Big fan of my namesake Bellard. :]

shye• 2 months ago

What I've learned from reading this is how much of my own writing style was influenced by late 1990s MSDN.

ethanlearns• 2 months ago

Angle: Eval angle: fine-tuning docs prompts is exactly the kind of prompt change that needs a regression gate before it ships. what eval harness, if any, guards the doc-generation prompt?

mybbor2• 2 months ago

For the last few weeks, I've been running an agentically created AI design publication where each day's issue gets re-imagined in a new design direction. Today the agent used this post as inspiration and built the issue as a Windows 98 help viewer, a nice nod to the manuals in your training corpus.

I jumped in and added working minimize/maximize/close buttons, a draggable window, and a Start menu, because of course. Brought back memories of young me learning Visual Basic to make AOL add-ons.

https://artdirectiondaily.com/issues/2026-06-05-docs-find-a-...

theletterf8• 2 months ago

The Windows 3.1 theme of my blog says Hi: https://passo.uno/win31/fine-tuning-docs-llm/

mybbor2• 2 months ago

Hi! Love it. The sound effects are a great touch. :)

selfhoster1312• 2 months ago

Too bad the author left out the most interesting part: not how to produce slop books, but how to properly OCR original text in 2026?

Fine-tuning an LLM to write docs like it's 1995

Related Stories