How LLMs Work, Explained Without Math

https://blog.miguelgrinberg.com/post/how-llms-work-explained-without-math

by kdamica • 2 years ago

215 87 2 years ago

blog.miguelgrinberg.com

thefz• 2 years ago

On the other side, given the propensity of LLMs to hallucinate, I wouldn't trust any workflow in which the LLM produces output that goes straight to end users without verification by a human.

Yep. Nice article, though!

astrange4• 2 years ago

The assumption that most people make is that these models can answer questions or chat with you, but in reality all they can do is take some text you provide as input and guess what the next word (or more accurately, the next token) is going to be.

These two things cannot be compared or contrasted. It's very common to see people write something like "LLMs don't actually do <thing they obviously actually do>, they just do <dismissive description of the same thing>."

Typically, like here, the dismissive description just ignores the problem of why it manages to write complete novel sentences when it's only "guessing" subword tokens, why those sentences appear to be related to the question you asked, and why they are in the form of an answer to your question instead of another question (which is what base models would do).

theshrike79• 2 years ago

Yeah, an LLM is not a Markov Chain. The only similarity is that they string words together with weighted possibilities. That's about it.

astrange4• 2 years ago

Well, it is a Markov chain if you do greedy sampling, which 99% of the time you do. So the weird part is why it still works so well.

If you do beam search, RAG, tool usage, etc then the whole system no longer is one.

pottspotts• 2 years ago

I came here to make this comment as well.

This line of reasoning that LLMs "only predict" the next token is akin to saying humans can only think or speak one word at a time. Yes, we use one token/word at a time, but it is the aggregate thought that matters, regardless of what underlies it.

winternewt• 2 years ago

I think the mistake people make is assuming that "probability" is a simple concept.

If there are 50K possible tokens and I don't have any other information, I could make a naive estimate that every token has equal probability and start generating text that is just gibberish. With the simple single-token Markov-chain example I would estimate probabilities based the previous token, and that probability estimate would be much better. If you use it for generating text it will look like something that is almost, but not quite, entirely unlike human speech. [1]

The difference lies entirely in how accurately you model the world and what information you have available when estimating probabilities. Models like GPT4 happen to be very good at it because they encode a huge amount of knowledge about the world and take a lot of context into account when estimating the probability. That's not something to be taken lightly.

[1] https://projects.haykranen.nl/markov/demo/

lm28469• 2 years ago

When the other camp is treating it like an oracle of truth and a sentient being it's hard to pick a side tbh.

vasco3• 2 years ago

If someone asks me "What is your name?" my reply is also simply just guessing what token would go well next in the full text of the conversation.

miningape• 2 years ago

No you "know" what your name is and you retrieve that information. It's infinitely different to rolling a dice and picking a name based on what number comes up

poniko• 2 years ago

It's really not though, generally maybe but you also give it a split second to think about if it's a good time to lie, make a joke or maybe just not reveal your name.

vrighter• 2 years ago

what if you're the only john in a world of steves? What would happen then?

input_sh• 2 years ago

If asked repeatedly, are you gonna answer that question with a different name, depending on which "memory" you randomly pick from?

edflsafoiewq• 2 years ago

That's an interesting example, since an LLM has no concept of a "self" and literally does not know who it is. It can only answer it "correctly" if you prefix it with a prompt telling it who and what it is.

photon_lines• 2 years ago

Yeah - most of the online descriptions aren't even remotely accurate nor close to explaining how LLMs like ChatGPT actually work. They are not simple 'next-word predictors' and most of the online tutorials / info don't go into fine-tuning nor the intricate details of chain of thought reasoning (which I personally believe plays a huge role in ChatGPT's amazing performance). If you want my own detailed description you can find it here: https://photonlines.substack.com/p/intuitive-and-visual-guid...

sk11001• 2 years ago

I think too many people confuse the base model (which can be called a next token predictor) with the fine-tuned chat model which is specifically modified to carry a conversation, be helpful and be as factually correct as possible.

7sidedmarble• 2 years ago

Where can you read more on that matter of it answering your question rather than asking more?

astrange4• 2 years ago

That's called instruction tuning.

https://arxiv.org/abs/2308.10792

mjburgess14• 2 years ago

If commenters wish to know what is not, "guessing the next word", let me outline it.

Compare, "I like what you were wearing", "Pass me the salt", and "Have you been to London recently?" as generated by an LLM and as spoken by a person.

What is the reason each piece of text (in a whatsapp chat, say) is provided?

When the LLM generates each word it does so because it is, on average, the most common word in a corpus of text on which it was trained: "wearing" follows, "I like what you were" because most people who were having these conversations, captured in the training data, were talking about clothes.

When a person types those words on a keyboard, the following are the causes: the speaker's mental states of recollection, preference, taste; the speaker's affective/attachement states with respect to their friend; the speaker's habitation into social cues; the speaker's imagining through recall what their friend was wearing; the speaker's ability to abstract from their memories into identifying clothing; and so on.

Indeed, the cause of a person speaking is so vastly different to generating a word based on a historical frequency, that to suppose these are related seems incomprehensible.

The only reason the illusion of similarity is effective is because the training data is a text-based observation of the causal process in people: the training data is distributed by people talking (and so on). Insofar as you cannot just replay variations on these prior conversations, the LLM will fail and expose itself as actually insensitive to any of these things.

I'd encourage credulous fans of AI not to dehumanize themselves and others by the supposition that they speak because they are selecting an optimal word from a dictionary based on all prior conversations they were a part of. You aren't doing that.

jameshart4• 2 years ago

LLMs don’t ’replay variations on prior conversations’ though.

Predicting the probable token in a conversation requires predicting the probable subject of the conversation, predicting the interlocutors’ relationship and manner of speaking to one another, predicting the state of recollection, preference and taste of the speaker, predicting the speaker’s mental model…

If the LLM isn’t predicting all of those things then it will produce poor predictions of the next word; doing it well - and humans tend to agree that in a vast array of cases state of the art LLMs do predict tokens very well - requires that prediction model to predict all that context as well.

mjburgess14• 2 years ago

Alas, no it doesn't. Language induces this sort of anthropomorphism in people, I guess, so consider images.

Suppose I take a billion images of all the coffee cups in the world, at a set of angles on the cup, and then build an associative (ie., frequency) statistical model of their pixels (ie., statistical AI). Consider generating one pixel at a time, in sequence, through the image. My associative model tells me P(col of next pixel | all previous).

Now, I can generate coffee cups images similar to any variation or combination of the images in the dataset. Now, you might say, "well you can only do that if you have a model of a coffee cup" (rather than of pixels) -- if so, just generate a coffee cup at one of the angles not in the dataset. This will not happen, because the model has not been provided with enough information to do so.

Namely, the model does not know the distance from the camera, the camera lens parameters, the angle to the coffee cup, etc. So there's literally a very very large inifinity of possible objects at unseen angles. Consider that underneath a coffee cup, the bottom might be missing entirely, etc.

Now it will appear to know all of these things, because its just generating images with these same parameters (camera, angle, distance, etc.). But as soon as you want "a coffee further away than has been seen before", or "a coffee using a macro lens", etc. the whole thing will fall over.

It is you, the view, who attributes 3D knowledge to the model because under ordinary circumstances the cause of a photo is features of a 3D environment.

civilized3• 2 years ago

humans tend to agree that in a vast array of cases state of the art LLMs do predict tokens very well

This argument is backwards. Humans don't measure the next token prediction ability of the agents they speak to, human or AI. We rate speakers on whether they seem to understand what we say in context and respond by contributing useful information and analysis.

The attributes you're saying can be inferred from known superior next token prediction ability are the things we can actually detect and measure, at least qualitatively. Next token prediction quality is not measurable by humans in any human-meaningful way. Improving test cross entropy by 50% doesn't mean anything to us. It is irrelevant except as a mechanism to train LLMs.

tiborsaas2• 2 years ago

If your point is LLM-s are different than humans, I guess that goes without saying? We are not even on the same level yet.

the most common word in a corpus of text on which it was trained

I think you are downplaying the fine grains of knowledge that can be encoded in a huge corpus of text. LLM-s are capable of taking context into account and encoding that too, not simply how often word A comes after B.

When I'm in a conversation I'm also selecting the optimal word from a predefined dictionary. That's precisely what's like speaking in a given language. Sure, I'm thinking a bit ahead and I can tap into my memory, feelings and experiences which influences everything.

But the optimal part is derived from context for me, it changes which word I use when I talk to a colleague, family or friend, but I might want to say the same thing. For stock LLM-s everything must be defined in the prompt if we are talking about zero-shot inference.

These models are opening good insight on how language works and I don't find that too dehumanising. There's plenty of room exists still for me to be human and do non AI things.

I get the notion that if we understand fully how something works the magic is gone, this always happens to AI. Are we afraid that this might happen to us too?

mjburgess14• 2 years ago

Are we afraid that this might happen to us too?

No, I'm more frustrated by the pseudoscience that models of frequency associations in text are explanations of people (or anything else). The choice isn't between a pseudoscientific behaviouralism where animals have no presence in the world, no mental faculties, and so on vs. "magic".

When I'm in a conversation I'm also selecting the optimal word from a predefined dictionary

Consider it this way: the probability dististribution over all possible world for you speaking is parameterized on space and time: Pyou(x, t; ...). And of the LLM generating text, Pllm(historical data).

So imagine plotting the live probability distributions of Pyou and Pllm for any given situation. As you think, imagine, move, recall, prefer, desire... the Pyou goes "wild" with dramatic discontinous shifts in distribution brought about by these causes.

Whereas the Pllm remains the same. It never changes. It never reacts to anything at all.

The whole distribution over all prior text tokens, Pllm is a stationary model of frequency associations. Yours is not. This makes all the difference in the world when claiming that Pllm somehow models, or is even relevant to, Pyou.

rhdunn• 2 years ago

I've experienced LLMs forgetting details (no memory). This is expecially a problem when the information is out of the context window, but I've seen it in other cases as well.

I've experienced LLMs lacking spatial awareness, such as switching locations in a description despite no indication of moving to the new location. The same applies to other concepts that have a visual/spatial component.

I've also experienced LLMs struggling to get subtext, some metaphors, etc., especially when used in casual conversations instead of as a question/answer style prompt.

LLMs are great, but need more work to fix these gaps.

throw310822• 2 years ago

When the LLM generates each word it does so because it is, on average, the most common word in a corpus of text on which it was trained: "wearing" follows, "I like what you were" because most people who were having these conversations, captured in the training data, were talking about clothes.

Yes. Now extend this to the context length of GPT-4 turbo- about 240 pages of text. So from your description, "wearing" is just the "most common word" to follow those 240 pages of (previously unseen, unique) text according to its training data. Quite simple, nothing to see here, I suppose.

chx• 2 years ago

https://hachyderm.io/@inthehands/112006855076082650

You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.
Alas, that does not remotely resemble how people are pitching this technology.

gizmo3• 2 years ago

Humans are capable of intricate complex thought that involves "mental states of recollection, preference, taste". However, LLMs have demonstrated pretty conclusively that most of the thoughts we have do not require any of that. Language turns out to be much simpler than previously assumed.

Thinking hard it makes your brain hurt. It's exhausting. Most of the work we do, including programming, is not like that. Some of the work we do is fiendishly difficult but much of it is more like word-completion based on prior experience.

Evolutionary processes optimize for energy efficiency. We don't think at 100% brain power all the time because we can't afford to. It makes a ton of sense, in retrospect, that our brains have optimized for language to the point that very little compute is required. And even so, the brain still consumes 20% of our daily calories.

Hard thinking is the exception and casual thinking is the norm. Why is it so hard to persuade people of anything on the internet? Because we mostly engage in LLM-like auto-completion. Very little actual thinking is involved and very few calories are spent.

logicallee• 2 years ago

When the LLM generates each word it does so because it is, on average, the most common word in a corpus of text on which it was trained:

But ChatGPT 4 follows instructions passably well. For example I just asked it: "Construct a sentence of at least 10 words each of which is extremely grammatically unlikely to follow the word before it. (For example "be are isn't had" as each of those words is impossible after the word before it.) Do not give any explanation of how you have arrived at your answer, reply only with your answer. However, as you construct it ensure that you cannot think of any context in which each next word would ever come after the word before it. Reply with your constructed nonsense sentence only."

Indeed it replied with a good nonsense sentence: "Dogs swimming beautifully reads soft under Wednesday during sky oranges" ("sky oranges" is unlikely, "under Wednesday" is nonsensical and ungrammatical) and when I complained that "dogs swimming" could be sensible as can "swimming beautifully" it came up with an even more nonsense sentence "Apples slowly would butter river quickly seven whenever blue music".

Do you think "Wednesday" is really the most likely word to follow "under" and "river" is really the most likely to follow "butter", or isn't it obvious that it was, for lack of a better word, "trying to" follow my prompt?

https://chat.openai.com/share/50037af6-0f3e-4de3-aff7-53a7b9...

mjburgess14• 2 years ago

Well ChatGPT is a mixture of LLMs, and no doubt a great deal more to make this work out (i'd suppose, eg., that they augment their datasets to have conversational framing, with actions/verbs etc. augumented -- or they acheive similar with models-on-top).

Nevertheless, roughly consider a dataset D for which we have an approximate stochastic model of its conditional frequency associations: P(next|previous..., D) etc.

Then if your prompt really got that reply, from this model, it would do so like this:

"Construct" is first projected to an encoding which replaces it, effectively, with a set of related words (Construct, Make, Create, Write...) all weighted by how they co-occur with construct.

Then we sample from D based on this word set, obtaining roughly, all conversations where these related words were used, call this Dc.

Next take "a sentence" and replace it with its word-set, say, (Sentence, Phrase, Words, ...) and sample conversations from Dc in which these occur, Dcs..

And so on. Since each token in your prompt actually corresponds to basically all possible words but weighted by association, each "filtering operation" actually selects vast amounts of the training data (space).

Finally, consider the reverse problem: what words could this system possibly produce from this process that weren't relevant to your prompt? Given enough data (PBs of text from all possible digitized conversations, books, etc.) then a sensible-seeming answer becomes the only plausible one to generate.

Now, I do think here PBs wouldnt be enough to generate a single statistical model that behaved this way -- so you need a mixture of them (ie., ChatGPT) and I suspect you also need a system for regulating discrete constraints such as quantities. I suspect many deployed LLMs have improved in this area due to models trained to be specifically sensitive to quantities.

danielbln3• 2 years ago

Yeah, I feel OP is ignoring the magic of in-context learning in their slightly reductive view on how LLMs work.

eggdaft3• 2 years ago

I think what this argument is missing is the emergent properties of the LLM.

In order to “predict the next word”, the LLM doesn’t just learn the most likely word from a corpus for the preceding string. If that were true, it would not generalise outside of its training set.

The LLM learns about the structure of the language, the context, and in the process of doing so constructs a model of the world as represented by words.

Admittedly the model is still limited, but it seems to me that there is something more insightful to be gleaned here: that given enough data, and sufficient pressure to learn, that excelling at scale on a relatively simple task leads indirectly to a form of intelligence.

For me the biggest takeaway of LLMs might be that “intelligence is pretty cheap, actually” and that the human brain is not so remarkable as we’d like to believe.

euroderf• 2 years ago

and that the human brain is not so remarkable as we’d like to believe.

Well, it IS pretty seamlessly integrated with a very impressive suite of sensors.

ryandvm• 2 years ago

Bingo.

When I write some 100% bespoke code that is rather hastily composed and then paste it all into ChatGPT4 asking it to "refactor this code with a focus on testability and maintainability" and not only does it do so, but it does a pretty damn good job about it, it feels rather reductive to say "it's just providing the next most likely word".

I mean, maybe that's how it works, but that statistical output clearly involves modeling what my code does and what I want it to do. Rather than make me think LLMs are a cheap trick, it just has me thinking, "shit - maybe that's all I do too."

mjburgess14• 2 years ago

Each word is taken to a distribution over words this is where the illusion of "context" largely comes from. eg., "cat" is replaced by a weighted: (cat, kitten, pet, mammal, ...) which is obtained via frequencies in a historical dataset.

So technically the LLM is not doing P(next word |previous word) -- but rather, P(associated_words(next word)|assocated_words(previous), associated_words(previous_-1), ...).

This means its search space for each conditional step is still extremely large in the historical corpus, and there's more flexibility to reach "across and between contexts" -- but it isnt sensitive to context.. we just arranged the data that way.

Soon enough people with enough money will build diagnostic (XAI) models of LLMs that are powerful enough to show this process at work over its training data.

To visualize roughly, imagine you're in a library and you're asked a question. The first word selects a very large number of pages across many books (and whole books), the second word selects both other books, and pages across the books you have. Keep going.. each more word you're ask, you convert to a set of words, and find more pages and books and also get narrower paragraph samples from the ones you have. Now finally, with total set of pages and paragraphs etc. you have to hand at the end of the question, you then find the word most probable following the other.

This process will eventually be visualised properly, with a real-world LLM, but it'll take a significant investement to build this sort of explanatory model.. since you need to reverse from weights to training data across the entire inference process.

ThinkingAgain• 2 years ago

Can it also explain the following: "An alien named abcdpqrs landed on earth yesterday. What is the name of the alien who landed yesterday on earth?"

Chatgpt answers it correctly. abcdpqrs is perhaps not in the training set. If it is we can pick some othername.

mft_• 2 years ago

How does this concept explain (for example) an LLM’s ability to provide a precis of an article? Or to compare two blocks of text and highlight differences? Or to take an existing block of code and find and correct an error?

lnenad• 2 years ago

Exactly what I am wondering, these are the complex things that "Oh it just dumps out tokens" explanations don't explain.

amanzi• 2 years ago

Stephen Wolfram has a great post on this: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

It's really long! But worth the read.

l5870uoo9y• 2 years ago

Are there any open source implementation of neural network "functions"? And the layering, transformers and attention mechanisms.

barfbagginus2• 2 years ago

Here's a walkthrough nlp.seas.harvard.edu/annotated

Here's a cool animated transformer, also open source prvnsmpth.github.io/animated

Here's the "attention is all you need paper" with links to open source implementations paperswithcode.com/paper/attention-is-all

pietmichal• 2 years ago

This was such a nice primer that inspired me to give Karpathy's series another try. Loved the explanation!

z72• 2 years ago

"I'll begin by clearing a big misunderstanding people have regarding how Large Language Models work. The assumption that most people make is that these models can answer questions or chat with you, but in reality all they can do is take some text you provide as input and guess what the next word (or more accurately, the next token) is going to be."

What separates this from the following:

"I'll begin by clearing a big misunderstanding people have regarding how the human brain works. The assumption that most people make is that the brain can think, reason, and understand language, but in reality all it can do is process electrical and chemical signals."

seydor• 2 years ago

yeah i think we are at a point of reflection in philosophy of mind, where we ponder if that our "thinking", our communication and dialogues are mechanistic continuations taking place serially in one or more brains

z72• 2 years ago

Whether brains or LLMs, claiming that "in reality all they can do" is X seems dubious, given that a system can have properties that its constituents do not have on their own. As an explanation of the emerging complexity of what "they can do" it is fundamentally unsatisfying.

alabhyajindal• 2 years ago

Great read, thanks for sharing!

How LLMs Work, Explained Without Math

Related Stories