Language Models Represent Space and Time

https://arxiv.org/abs/2310.02207

by birriel • 3 years ago

123 186 3 years ago

arxiv.org

famouswaffles12• 3 years ago

The Twitter thread is worth looking at. Very Fascinating

https://twitter.com/wesg52/status/1709551516577902782?t=3b2F...

joshuahedlund2• 3 years ago

I don't understand what the colors represent and I can't read the thread. Can you summarize?

famouswaffles12• 3 years ago

You should be able to read the thread here https://nitter.net/wesg52/status/1709551516577902782

The colors in the first tweet just represent different territories(continents)

nmarinov• 3 years ago

Seconded, it's a good summary with graphs and commentary. Also nitter to avoid the login wall: https://nitter.net/wesg52/status/1709551516577902782

kridsdale34• 3 years ago

Thank you for your service.

civilitty• 3 years ago

I think we have a new member to add to the group: Lies, damn lies, and statistics… and now neural activations. How exciting!

This reminds me of all those fMRI studies that look at brain blood flow before the dead salmon experiment came out.

somenameforme• 3 years ago

Haha, I have no idea how I was not aware of the dead salmon experiment prior. This article [1] has a nice overview for anybody else in my shoes. Thanks for the reference.

[1] https://blogs.scientificamerican.com/scicurious-brain/ignobe...

gpderetta2• 3 years ago

First of all that's an excellent article. But also:

Some people like to use the salmon study as proof that fMRI is woo, but this isn't the case, it's actually a study to show the importance of correcting your stats.

I.e. be aware of the "look elsewhere effect".

corethree8• 3 years ago

A lot of HNers were so adamant that the LLMs understand absolutely nothing and that these models are just predicting the next most likely word.

I think with this paper it becomes clear that the adamant denial was just human bias talking. LLMs crossed a certain line here.

You would think people would be amazed or in awe or react in fear at technological break through a and they often are. The weird part for LLMs was the delusion. Many people were in denial that it was any sort of breakthrough.

YeGoblynQueenne17• 3 years ago

This is a preprint that was just uploaded to arxiv two days ago. Don't be hasty and assume that it settles any matter at all. Many such claims have been made before and many counter-claims also. There is still a lively debate on the subject and it will be some time before there is agreement.

More generally, any scholarly article is a claim, and should never be read as automatically true. That's something to keep in mind.

circuit105• 3 years ago

I thought it was completely obvious that if it can predict things about the world as well as it can it has some form of world model, even if inaccurate. I’m surprised that anyone would argue that a language model doesn’t model things

andrewguenther2• 3 years ago

I think LLMs are presenting some uncomfortable philosophical questions for people about how our own brains work and admitting that there is any kind of "intelligence" (even if very basic) in an LLM is an admission that our own brains may work in a similar manner.

YeGoblynQueenne17• 3 years ago

For me, one of the most interesting things that have come out of LLMs is the confirmation that humans are very bad at reasoning and, consequently, its' a very bad idea to try and make machines that "think like humans", because that way we'll only make machines with none of the advantages of machines and all the disadvantages of computers.

For instance -I'm not trying to be mean and I'm certainly not blaming you in particular, because I've seen this very often- but the reasoning that because LLMs can generate language, and humans can generate language not only LLMs are somehow like humans but also humans are like LLMs is not sound.

For example, walls have ears, cats have ears, therefore walls are like cats and cats are like walls. That doesn't work because walls' ears are not like cats' ears and even if they were, that still wouldn't make walls cats and cats walls, it would just make them both entities with ears.

corethree8• 3 years ago

This. I think this is the reason why so many people are in denial. Is All of intelligence simply trying to find the best fit curve in an n-dimensional scatter plot of data points?

anigbrowl3• 3 years ago

Indeed. I was offering examples of how you could bootstrap cognitive processes at the beginning of the year and people were just sticking their fingers in their ears. A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things.

SiempreViernes2• 3 years ago

Who is the subject in the last bit of your last sentence? More pointedly: are you claiming that OpenAI is doing LLM research to gain political freedoms?

OneDonOne3• 3 years ago

A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things.

Could you please explain what the above has to do with a Large Languange Model?

fipar6• 3 years ago

My only experience so far is with chatgpt, and I am not an expert in AI, or even just in LLMs.

With those disclaimers, my interactions inform my opinion that the LLM behind chatgpt has no internal world model. It shows no understanding of basic facts and makes very silly mistakes very easily. I have my bias, like anyone else, but in the case of AI in particular, I should say that I don't think there's anything especially magical or sacred about conscience and the human brain (or the brain of other animals, for that matter), and I'm sure it must be possible to arrive at different forms of intelligence starting with the "inanimate" building blocks of hardware and software, but what I've seen so far in LLMs doesn't make me support your statement that they've crossed a certain line. In fact, I'm very much let down by the experience and I'm afraid once the hype goes down, we may be in for another AI winter.

As I said at the start of the comment, I'm well aware I'm not a subject-matter expert. I'm open to being wrong. I just wanted to point out that not all of us unimpressed are in denial. I'd love to be in awe at an AI breakthrough, and I kind of feel I will be in awe sometime during my lifetime, but LLMs are not (yet?) that for me.

Edit: s/are not in denial/are in denial/

ilaksh3• 3 years ago

Have you used GPT-4?

nopinsight4• 3 years ago

Current LLMs likely have multiple world models with varying qualities. Good prompts are required to activate suitable models for each task.

1. Did you use GPT-3.5 (free version) or GPT-4 (paid one) to form the judgment? Their performances on harder tasks differ significantly as shown in https://openai.com/research/gpt-4.

2. Have you tried adding "Please think step by step" to some harder requests? This simple phrase gets most current LLMs to perform significantly better. It's a bit like asking students to show their work, which forces them to think more clearly.

Current LLMs, without additional mechanisms, tend to be perform like a drunk or sleepy human, i.e. using mostly intuition or System-1 (as defined in "Thinking Fast and Slow"). The prompt such as "think step by step" asks it to think more in the System-2 style. (There are other techniques which get them to perform even better still.)

I think of current LLMs as a very-well-read, but often sleepy intern who needs strict instructions, feedback, and sometimes extra training if you want them to perform well.

pyth0• 3 years ago

People are already asking you if you have tried <insert version here> but this pretty much echos my experience and I have used both free and paid versions of ChatGPT (3.5 and 4 respectively) as well as the GPT4 API directly. I had a similar experience between all three of them, although the GPT4 based versions were certainly better in terms of output quality. In my opinion it seems pretty clear that this is a fundamental limit of the architecture and all the RLHF and increase in model sizes wont change that.

webdood902• 3 years ago

I really don't understand replies like this.

This is just the beginning. It is a tiny, tiny view into this new technology.

Nobody with an ounce of intelligence looked at the first iPhone and thought "ah well it's just another phone". It's the potential of this new technology that is so impressive.

How can you not be impressed by this? I am literally incapable of empathizing with your perspective. All I can think is that the world is going to pass you by.

tayo42• 3 years ago

Seems like the hyperbolic conclusion to make? The paper doesn't make that claim. Simple ml models can take data and find patterns, like separate red and blue, or xy coordinates. Those are a small amount of dimensions and easy to reason about.

This seems like the model found on its own another dimension to segregate data in deeper layers by generalizing and using context when learning. Its cool it can do that, but it still to me seems like drawing a best fit line just with higher dimensions.

I'm sure someone will correct me if they think I am way off.

Sharlin2• 3 years ago

It's the Chinese Room argument all over again. People hear "predicting the next token" and all they can imagine is some sort of a statistical database lookup, ignoring the fact that when you have a huge corpus of data with incredibly complex internal correlations and all that data also happens to correlate with some unknown external thing, it's almost certain that a powerful learner will end up modeling that external thing if and when doing so will cause a quantum leap in prediction performance! A model that includes the external-thing hypothesis will almost certainly be simpler, ceteris paribus, than a model that doesn't.

esafak3• 3 years ago

That is what a model is; a parsimonious explanation of observations that admits extrapolation.

gamblor956• 3 years ago

If you actually read the paper, the analysis absolutely does not support the claim that LLMs are capable of maintaining a spatial or chronological internal world model.

The LLMs simply activated when locations or times were mentioned, with no understanding of the relationship between different places or different times beyond the fact that different places and times had different tags. Or in other words, no internal model of the world.

nyrikki11• 3 years ago

The lack of generally accepted definition for knowledge and intelligence makes these conversations difficult but the above post is important to remember.

The paper has nothing to do with the common sense problem.

While it is a preprint, I do wonder how useful this will be to improve generalization.

Termination analysis on linear polynomials is undecidable over the naturals without many constraints and is undecidable ofer the rationals.

If you think about how sstrees or even rtrees work, this isn't surprising to me that space and time are useful for compression which is required for learnability.

But I think people also see 'linear' and think that means easy. When it can mean serious restrictions on how it can be used.

Hard to explain the connection I have in mind without serious math but here is a paper that may cause some issues with trying to use these internal structures outside of the learnability implications.

https://arxiv.org/abs/2307.14805

umanwizard2• 3 years ago

I think with this paper it becomes clear that the adamant denial was just human bias talking

It already becomes clear after playing with ChatGPT-4 for five minutes. But a lot of people still refuse to accept it.

OneDonOne3• 3 years ago

A bigger Chinese Room - is still a Chinese Room. We already went through this with ELIZA 60 years ago

benlivengood• 3 years ago

The next step in the research should be ablation of these time and space neurons to see how it affects accuracy on space/time completions which would help rule out the memorization of linear probes.

SiempreViernes2• 3 years ago

It would also be reassuring if they showed that they couldn't reach similar performance when they assigned random values to the target coordinates and used non-spatial words.

cratermoon• 3 years ago

No, no they don't. People looking at the output of LLMs project their own internal models of the world, time, and space onto the output and project that model as if it were the LLM's.

canjobear2• 3 years ago

The paper is looking at internal representations, not output.

klysm4• 3 years ago

You’re responding to the headline. Go look at the paper

canjobear2• 3 years ago

Much simpler models like word2vec also showed linear embeddings of geography in this way.

semitones• 3 years ago

source?

uoaei3• 3 years ago

It was something like

    Paris - France + Japan = Tokyo

I know that's not a source but you could also just google "word2vec", as I recall most of the explainer blog posts had similar examples.

alexpetralia• 3 years ago

Space and time (the abstractions) are lingisuric concepts, so this is perhaps not too surprising, but nevertheless interesting.

klysm4• 3 years ago

The space and time mapping they pull out of the LLM here is non-linguistic.

klysm4• 3 years ago

I’m most surprised that the embedding is actually linear. I wonder if that makes extrapolation perform better

nyrikki11• 3 years ago

Feed forward networks are pairwise linear in respect to inputs, they are effectively DAGs.

In theory you could represent an entire LLM as a single 2 dimensional graph of linear line segments.

It wouldn't be useful for much as the parameters are clustered in dense patches.

kridsdale34• 3 years ago

So are my synapses, and by and large my cortex is 2-dimensional as well (folded and scrunched in to a super complex shape by the limitations of my roughly spherical cranium)

xpe7• 3 years ago

The papers cites earlier papers that posit a ‘linear representation hypothesis’. Does your surprise “factor in” these papers? Do you suspect something different in play for the current paper?

klysm4• 3 years ago

Didn’t have that factored into my surprise because I’m not that informed. I just thought that the latent space representation being linear was interesting.

toxik• 3 years ago

Maybe something to do with the Transformers are mesa optimizers thing?

yieldcrv• 3 years ago

I bet if we were able to get better output from our brains we would see something similar, for those of us that have any spatial awareness capabilities

But we don't have the output to analyze aside from our language and assumption of shared experience

ilaksh3• 3 years ago

One thing that is clouding this discussion is that most people are mixing up a lot of different characteristics that animals like humans have as if they were all the same thing.

So for many people they don't really distinguish between things like "reasoning", "self-aware", "conscious", "alive", "sentient", "intelligent", "has world model". They also don't distinguish between different types or varying levels of cognitive abilities.

It seems clear that high functioning LLMs must have some type of world model. But that doesn't mean it's necessarily exactly the same type of highly grounded model that a human would have, especially if it was trained on only text. It might be less rich or different but still quite useful.

Another example, LLMs clearly don't have the same type of fast adaptation in a realtime 3d environment that animals have. (That's not to say that they can't mimic it in some rough ways).

But if you don't really break all of this stuff down carefully in your head then it can be hard to accept that LLMs are doing anything interesting. Because in that worldview, it's all the same thing, so they have to give the LLM all of the other characteristics at the same time.

xpe7• 3 years ago

Well said.

Also, the paper’s use of “world model” (instead of simply “model”) doesn’t help clarify the situation.

sebzim4500• 3 years ago

I do not see how referring to a world model as simply a "model" could possibly provide clarity.

LLM literally stands for Large Language Model, it has a 'model' in it by definition.

moffkalast• 3 years ago

It's not the first time this has been a problem to accurately specify either. Throughout history animals proved nigh impossible to classify in these terms and were considered so many things, from something that's intertwined with nature as a hive mind of sorts, to complete automatons by Descartes, and later to beings that are mostly completely identical to us by Darwin.

I suppose it's also an interesting juxtaposition, animals can do everything but talk, while LLMs can do nothing but talk.

kridsdale34• 3 years ago

So all we have to do is plug a USB cable from a device running an LLM in to the back of a monkey.

kelseyfrog2• 3 years ago

For the most part our daily life doesn't depend on contrasting minimal pairs of these attributes. Humans, if they have one, tend to have all.

AI entities, may exhibit one, or none or sometimes some and that maps poorly to our prior experience with reasoning entities.

How would self-aware, non-conscious agent respond?

How would a reasoning but non-intelligent agent respond?

somewhereoutth2• 3 years ago

If an LLM is trained on just text, then it's internal model can only be based on such text.

I think the 'conceit' of LLM as AI advocates is that this model is just as (or at least nearly as) our own internal model - that the 'simplest' model to match the text is indeed our 'real' model that matches reality as a whole. A bit of a stretch surely!

mjburgess3• 3 years ago

LLMs are a redemonstration of the lesson of sun worship: apes are easily fooled.

If the output is indistinguishably 'world modelling', so must its mechanism be.

There is no 'world' model in a hashmap from questions to answers about a world. There is only the condition that, given those questions, any old ape would accept those answers.

Science is required to determine whether a system has properties, such as a rational ecology; ie. a genuine mechanism of environmental coordination.

No engineer, no statistician, and certainly no VC HN Hype Man can determine-so. The nature of reality is no up for Big-O equivalences.

Systems have properties in virtue of their nature, not in virute of what I will pay $20/mo for

wanderingstan• 3 years ago

I’m no expert, but the word “just” is tricky. You could say mammalian brain is trained on “just” nerve impulses.

essive• 3 years ago

I think the term "world model" is a highly overloaded term. What does it really mean in your opinion?

ilaksh3• 3 years ago

A reasonably precise although not universal definition could be to take the data points from the study in relation to the use of those terms in the study.

I think the main idea though is that there is some structure in the memory that resembles a structure in the real world to some meaningful degree. One aspect of this is composition, the other is how closely it maps to the real world.

If it were literally just a lookup table where the indices did not correspond to physical proximity then it would definitely not be valid to say that this was really a world model in that context.

esafak3• 3 years ago

Some understanding of the laws of nature, as we know it.

teaearlgraycold2• 3 years ago

There was an interesting thought from a Microsoft researcher in an episode of This American Life. He had been given early access to GPT4 and found that it had gained an understanding of gravity and balancing. 3.5 would fail to describe a safe order for stacking 3 eggs, a bottle, a book and a nail. But 4 would give robust answers with logical justification added to each construction step for the tower.

The researcher remarked (paraphrasing) “The models are built to predict the next token. At some point the only way to improve token prediction is to start modeling the world and building an understanding of how things work.” Since I heard this I see it everywhere with LLMs. Load up a 30B parameter model on your computer and toy around with it. When it fails I now see that it’s reached a concept it doesn’t understand and is in more-or-less Markov Chain mode. Give the same task to GPT4 and it proves it’s modeled the concept appropriately because it can pull the right words out of the ether.

xpe7• 3 years ago

Give the same task to GPT4 and it proves it’s modeled the concept appropriately because it can pull the right words out of the ether.

I take your point, but we should be careful when using phrased like “modeled the concept appropriately” here. That implies correctness.

There are many ‘concept models’ that could work well enough. Unless we can inspect the concept model directly (inside the LLM, somehow!) we are left to reason about how the concept is represented via statistics over a set of interactions. So, if, over the course of many interactions, it seems that a model is saying reasonable things we can have more confidence that it has achieved a concept model that is “good enough”. But such an assessment is always in relationship to the test protocol. Claims to generality should be done carefully.

In some ways, my standard here is probably not that different than a rigorous human educational assessment. I’m not talking about standardized tests, I’m talking about adversarial challenges like one would hope to see during a Ph.D. defense.

teaearlgraycold2• 3 years ago

Yeah I’m setting the bar at the level of a child’s understanding. Because smaller models are well below that, just picking tokens at random. That’s how misunderstanding manifests with LLMs. They don’t say “I don’t know”, they just go off the rails.

The quality bar for different use cases might be such that an understanding of a system on par with a 6 year old is isn’t useful. But right now we clearly have two clusters and I’m drawing the boundary where it’s most useful.

nl• 3 years ago

I take your point, but we should be careful when using phrased like “modeled the concept appropriately” here. That implies correctness.

All models are wrong, but some are useful.

Perhaps we should say "modeled the concept usefully".

In some ways, my standard here is probably not that different than a rigorous human educational assessment. I’m not talking about standardized tests, I’m talking about adversarial challenges like one would hope to see during a Ph.D. defense.

Usefulness is very contextual obviously. Newton's laws aren't entirely correct but are still useful.

computerex• 3 years ago

This is the same reasoning that Ilya Sutskever uses. That next token prediction objective causes the model to learn world models. Because being able to predict the next token requires a lot of the knowledge/model of the world.

didibus• 3 years ago

Can someone explain to me how they get these graphs? What exactly are they measuring in the model execution?

ImHereToVote8• 3 years ago

Wait I had the "It's just a stochastic parrot" parroted at me ad nauseum. Was this FUD all along?

nyrikki11• 3 years ago

No, this paper doesn't change that claim at all.

This paper is about learnability and not generalization of what is learned.

From the papers summary, note that they are referencing learning and not some implications that it gets us closer to a universal quantifier.

"We have provided evidence that LLMs learn linear representations of space and time that are unified across entity types and fairly robust to prompting, and that there exists individual neurons that are highly sensitive to these features. The corollary is that next token prediction alone is sufficient for learning a literal map of the world give sufficient model and data size."

ImHereToVote8• 3 years ago

I mean parrots have neurons. I didn't realize the line referred to the actual birds "parrot". Parrots are wicked smart.

cjbprime• 3 years ago

The paper it originates from predates ChatGPT (let alone GPT-4). So I think a charitable interpretation is that it was partly wrong (and definitely unimaginative) at the time, and partly it's the case that very few people were expecting "let's just throw more parameters at it" to work and result in emergent general capabilities, which it appears to.

tomjakubowski• 3 years ago

LLMs have an internal temporal model and yet ChatGPT still can't explain to me what happened in Tenet

jsty• 3 years ago

I’ve watched Tenet and I’m not sure I can explain what happened either …

gcanyon2• 3 years ago

To be fair, Christopher Nolan made a whole 2+ hour movie attempting to explain the concept and couldn’t…

kridsdale34• 3 years ago

GPT4 explains it pretty clearly, I think:

"Tenet" is a 2020 science fiction action-thriller film written and directed by Christopher Nolan. The story revolves around concepts of time manipulation and inversion, making it a complex narrative that can be challenging to understand on first viewing.

Here's a basic summary: The protagonist (referred to as "The Protagonist", played by John David Washington) is a secret agent who gets involved in a mission to prevent World War III. He is introduced to a concept called "inversion," where the entropy of people or objects can be reversed, making them move backwards in time.

His mission leads him to cross paths with a Russian oligarch named Andrei Sator (played by Kenneth Branagh), who is collecting pieces of an algorithm that can invert the entropy of the entire world, effectively reversing time and destroying the present in favor of the future.

The Protagonist also meets Kat (played by Elizabeth Debicki), Sator's estranged wife, who becomes a crucial part of the mission. The Protagonist and his partner Neil (played by Robert Pattinson) use inversion to their advantage in several action sequences, including a car chase and a final battle at Sator's secret city, where they successfully get the algorithm and prevent the destruction of the present.

The twist at the end of the film reveals that Neil was recruited by a future version of The Protagonist, and that they have been working together for much longer than the duration of the film's events. This means that the organization "Tenet", which they work for, was created by The Protagonist himself in the future. Neil's character is seen sacrificing himself to ensure the mission's success, highlighting the theme of fate and predestination in the movie.

"Tenet" is a complex film that uses its time manipulation concept to construct a narrative that loops in on itself, with events and characters revealing their true significance only as the story progresses or even after the film ends. Nolan's movie plays with the concepts of time, fate, and free will, requiring viewers to actively engage with and decipher its narrative structure.

esafak3• 3 years ago

That's a better explanation than I can muster.

gcanyon2• 3 years ago

“Have an internal world model,” or “give the appearance of having an internal world model”?

(Disclaimer: I’ve only read the summary, not the full paper)

Recent papers about advanced Go-playing AIs have made it abundantly clear that, at best, their conceptual understanding of Go is deeply flawed, and at worst, completely absent. Yet under most circumstances they give the appearance of super-human understanding based on superhuman play.

Do we have any reason to think LLMs are different? That they understand anything beyond “this is the best word to go next here.”

To be clear, LLM output is remarkable, all the more so if the above is true.

galaxytachyon• 3 years ago

I think sometimes it is good to zoom out a little bit and look at things on a higher level. The LLMs are not an organic creature. It doesn't in any shape or way relate to a human being except the knowledge it was trained on. A bacteria shares over 50% of its genetic sequences with a human and we don't see it as anything more than a glorified bionic automaton. An LLM shares 0% with us. In fact its "genetic sequence" is coded on a completely different physics altogether. It is unreasonable to expect it to understand or think the same way an organic creature do.

But the outcome of its action is irrefutable. It plays and does things better better than us. That is all we need to evaluate it on. I know that the topic is about "sentience" but my point is we are evaluating a complete alien being using human's standards. Of course we will see that it is lacking. It isn't human.

lsy2• 3 years ago

The interpretation of the LLM having a "world model" is a big stretch in terminology. Encoding a coherent or accurate set of spatial coordinates for places is not equivalent to understanding the actuality of world space and the relationship between items in that space. By this definition an accurate spreadsheet of the lat-long of every major city would also constitute a "world model". It shouldn't be surprising to anyone that a model trained on (likely many) such spreadsheets would also encode that coordinate data. What's notable is that it still takes a team of human researchers to plot these raw numbers onto a map and to interpret them as experiential differentiators in physical space.

There's no doubt that an LLM can uncover whatever is structurally included in its training data, even if that encoding is implicit. What's less believable is that the LLM somehow achieves a grounded understanding of physical aspects of experience purely from streams of raw text.

I think the paper engages in some equivocation here, as the abstract differentiates between "an enormous collection of superficial statistics" and "a coherent model of the data generating process", without admitting that the first should well imply the second. But that doesn't then further imply a "world model" in the sense that we understand it as sentient beings. For us, an internal model is useful, but when our model or texts don't agree with the actuality of the world, the actuality of the world takes precedence. For an LLM there is no distinguishing between its trained representations and an actual exterior place, or even any sense that an exterior space exists.

nopinsight4• 3 years ago

Current LLMs clearly have much coarser models of the world than humans do. Their training data do not include as many modalities and training data with spatiotemporal dimensions are inadequate.

Up and coming multimodal models are changing this.

lsy2• 3 years ago

"Multimodal" models only promise an additional layer of indirection and interpretation from the world through sensors into the same conceptual mechanism. An image is converted to tokens, a sensor is converted to coordinates, and it is all fed back into a correlating model. But this isn't adequate either. The famous dress illusion picture has RGB values closer to gold and white, but the actual physical dress was black and blue. The difference between perception and reality is something we don't have a handle on when it comes to these models.

ComplexSystems• 3 years ago

"Encoding a coherent or accurate set of spatial coordinates for places is not equivalent to understanding the actuality of world space and the relationship between items in that space."

What do you mean by the "actuality" of world space? It certainly has memorized an enormous data set of geographic points for various important locations and can do some pretty sophisticated reasoning and inference from them, and with the code interpreter it can compute various metrics derived from them. What can this thing do that humans can't?

xpe7• 3 years ago

I’m seeking a brief almost hands-on introduction to how probing is done: in earlier papers and here. Suggestions? (Preferably with some photographs of napkin drawings.))

xpe7• 3 years ago

Perhaps the most important public discussion, at least among technical people, is making sure we really understand the experimental setup.

(I admit I could be projecting my lack of understanding; I’m not rock-solid on the experimental setup yet.)

Still to me, it matters if the linearity been described can be observed directly _in_ the model. My understanding is that it is not; rather, it is tested with an additional model. The term probing can be misunderstood; it isn’t -just- an internal probe. There are ‘derivative’ models as well.

This raises questions about what we mean when we we are talking about internal representations! In my view, a derivative of an internal representation is not the same thing.

(I welcome people who can educate, correct me, or even reframe the topics. The above thoughts are subject to change.)

flaghacker• 3 years ago

It's important to note that these probe models are very simple - they're "trained" by just doing a linear regression between the hidden activations and the desired output. This means that the probes can barely do any computation themselves, so if they work at all this is a strong indication that the signal they predict was already in the hidden activations.

For even more proof, see "Figure 5: Space and time neurons in Llama-2 models" for single neurons in LLMs that already encode this information, without even having to use a probe model to extract it.

xpe7• 3 years ago

Thanks. After some additional research, I tend to agree.

One thing I learned of note is that while a _positive_ result (i.e. a probe showing a relationship) is proof that a label is encoded somehow; a _negative_ result does _not_ prove that the network under test does _not_ encode the information _somehow_. (The information could be encoded in more complex ways that the probe does not discover).

haltist3• 3 years ago

coherent model of the data generating process -- a world model.

I'd never seen a definition of world model but this seems deficient in several ways because it does not mention anything about abstraction and logical reasoning.

wnoise4• 3 years ago

Those aren't part of the model though. The model is just a representation of the world. An abstract representation can be useful, but so can more concrete ones. A representation that is amenable to logical reasoning is also useful, but the reasoning isn't part of the model.

haltist3• 3 years ago

Then why call it a world model? It's a "data" model and not actually a "world" model. World models would need to have logical relationships between the entities being modeled which would include non-trivial logical relations, R(x,y), and properties, P(x). Current language models do not have such representations and fail miserably on any task that is not explicitly included in the training set (sudoku being an obvious example).

Language Models Represent Space and Time

Related Stories