I disagree with Geoff Hinton regarding "glorified autocomplete"

https://statmodeling.stat.columbia.edu/2023/11/18/i-disagree-with-geoff-hinton-regarding-glorified-autocomplete/

by magoghm • 3 years ago

194 264 3 years ago

statmodeling.stat.columbia.edu

11235813213455• 3 years ago

I still want to read a ChatGPT text that doesn't look stupidly verbose

chpatrick2• 3 years ago

Why don't you ask it not to be?

coffeebeqn• 3 years ago

The default style really annoys me with its verbosity and overuse of numbered lists. Describing a tone and length of responses and anything else you want - think step by step, first principals, show don’t tell - all work pretty well.

You can even ask it to only reply with a Boolean true or false - which certainly doesn’t play to its strengths

JoshTko• 3 years ago

Use custom instructions. I use "Answers should be very concise by default. Never provide disclosures. Answers should get to the point immediately." and get super concise answers. If I want more detail I just follow up with can you expand on X.

xcv12324• 3 years ago

Follow up with the prompt "rewrite that so that it doesn't look stupidly verbose" and it will do exactly as requested.

WillPostForFood• 3 years ago

Give this GPT called Briefly a try. Responses come back as bullet point lists.I prefer more verbose, but interesting to see this in contrast.

https://chat.openai.com/g/g-LNsEQH5rz-briefly

mbrumlow• 3 years ago

If you want to be really good, you have to understand what’s being said. That’s the only way.

This is simply not true. Predicting the next letter or word, or id you abstract it away from things that mean something to you, like the next color of a block in a long chain of colored blocks. You would realize that all we are doing is using statistics to predict what the next item might be. There simply is no need or requirement for the system doing the prediction to understand.

albertzeyer• 3 years ago

You would realize that all we are doing is using statistics to predict what the next item might be.

So what does "understanding" really means then?

"Understanding" is not really well defined. Either we (humans) do it, but then LLMs might just do it as well, depending on the definition of "understanding", or we both don't do it. But if no-one is really "understanding" anything, then this definition of "understanding" is maybe not useful.

dboreham2• 3 years ago

Bingo

Joeri• 3 years ago

Nobody actually understands how humans actually understand something, just like nobody actually understands how LLMs do what they do.

Everybody opining about it is doing just that: offering an opinion. Geoff Hinton’s opinion is worth more than someone else’s, but it is still an opinion.

mannykannot2• 3 years ago

I feel that LLMs raise some very interesting challenges for anyone trying to figure out what it means to understand something and how we do it, but I am not yet ready to agree with Hinton.

For example, we are aware that some, but by no means all, of what people say is about an external world that may or may not conform to what the words say. We can also doubt that we have understood things correctly, and take steps to either confirm or refute our opinions. We see ourselves as entities in an external reality containing other individuals who also do this, and that we, and they, have a limited ability to influence what happens in that world. Do LLMs do these things, or is what they produce a result of having a lot of information about the purely formal properties of human language use, independently of semantics?

satuke2• 3 years ago

Isn't that exactly how humans learn to respond to stimuli? Don't we just try to predict the best next response to everything? Yes, It's statistics but the fun part is nobody is writing this statistical function by hand.

civilized4• 3 years ago

LLMs don't have a concept of "best". Only most likely in what they've been trained on.

I think LLMs ultimately just take imitation to a creative and sophisticated extreme. And imitation simply doesn't comprise the whole of human intelligence at all, no matter how much it is scaled up.

The sophistication of the imitation has some people confused and questioning whether everything can be reduced to imitation. It can't.

The ability to imitate seeking a goal isn't identical to the ability to seek a goal.

The ability to imitate solving a problem isn't identical to the ability to solve a problem.

Imitation is very useful, and the reduction of everything to imitation is an intriguing possibility to consider, but it's ultimately just wrong.

RandomLensman• 3 years ago

Yes, if you are in the no free will school of thought, then that would be what humans do.

usrbinbash4• 3 years ago

Isn't that exactly how humans learn to respond to stimuli?

Maybe it is, maybe it isn't. Maybe we are "just" an incredibly powerful prediction engine. Or maybe we work from a completely different modus operandi, and our ability to predict things is an emergent capability of it.

The thing is, no one actually knows what makes us intelligent, or even how to define intelligence for that matter.

randyrand• 3 years ago

the human brain is just a statistical machine.

brookst4• 3 years ago

Odds are you’re right

continuational4• 3 years ago

I think the most useful definition of understanding something is that you can explain it and use it in context.

ChatGPT routinely does both.

jstummbillig• 3 years ago

And while AI gets better and better and we will remain as touchy as ever about abstract concepts that make us oh so human, how about we say it just can't be understanding, unless a human does, eh, it.

BobaFloutist2• 3 years ago

Someone sufficiently fast and skilled at googling can explain and use in context a lot of things that they don't really properly understand.

So unless you're saying that the composite of the googler and of google understand something that neither does individually, your definition has some holes.

skepticATX2• 3 years ago

How about this: understanding is the ability to generalize knowledge and apply it to novel scenarios.

This definition is something that humans, and animals for that matter, do every day - both in small and large ways. And this is something that current language models aren't very good at.

Kiro2• 3 years ago

How do you define understanding when predicting the next color?

karaterobot• 3 years ago

You would realize that all we are doing is using statistics to predict what the next item might be.

I agree that Hinton's original quote doesn't make sense to me either. I suspect he would leverage the phrase "really good" to explain the difference between ChatGPT and, say, a Markov chain. I think that's a little disingenuous, if that's how he means it, but I don't know if I'm right about that.

But I also do not agree that humans use statistics to predict what the next item in a series might be. As evidence, there is the classic example of asking people to predict the next coin toss in the series: "heads, heads, heads, heads, heads, heads, heads...". They'll either guess heads because it's come up so many times already, or because they assume the coin isn't fair, or tails because it's "overdue" to come up, but none of those are based on statistics per se.

troelsSteegin• 3 years ago

From the article, Gelman's money quote is this: " So I’m not knocking auto-complete; I’m just disagreeing with Hinton’s statement that “by training something to be really good at predicting the next word, you’re actually forcing it to understand.” As a person who does a lot of useful associative reasoning and also a bit of logical understanding, I think they’re different, both in how they feel and also in what they do."

hackerlight• 3 years ago

Why is "using statistics" mutually exclusive with "understanding"? It would help to carefully define terms. Note that "understanding" doesn't mean it's necessarily conscious.

These systems learn high-level representations/abstractions of concepts we humans also use as part of our cognition. The concept of an object, an intuitive physics, the role of specific objects. I don't criticize him for using the word "understanding" to describe this.

porknubbins• 3 years ago

As a translator I have come to think of it like this- (without the need for defining what “understanding” means) the ability to translate correctly depends on the size of your context. So humans and LLMs are both on the same spectrum with different context sizes of their training set and all facts known in the universe (for a theoretical perfect human) respectively.

Doing moderately technical translations, state of the art LLMs may get it about 99% right but you find about 1 word per page where they would have needed to “zoom out” to a wider context to choose the word correctly. Often its an important word that is missed.

usrbinbash4• 3 years ago

You would realize that all we are doing is using statistics to predict what the next item might be.

Two questions:

1. How can humans come up with novel ideas then? Statistically predicting the next item means I am limited to the training set. How do humans formulate new concepts in this scenario?

2. Why can humans learn from their own output then? If all we do is statistically predicting the next token, then, by necessity, feeding our own output back into the model (aka. letting humans learn from it) should degrade human capabilities by overfitting to the training set. But we observe exactly the opposite.

micromacrofoot• 3 years ago

How can humans come up with novel ideas then?

hallucinations, emotions

Why can humans learn from their own output then?

if we don't, we suffer

jameshart2• 3 years ago

Statistically predicting the next item means I am limited to the training set

Not at all. Extrapolation is perfectly possible in a purely predictive model. It’s one of the things GPTs are best at.

In the stream of tokens output by an LLM it’s completely possible for new concepts to emerge, and for it then to continue to use and build on them in the remainder of the stream. You see this simply executed in programming contexts with LLMs where it is able to declare a novel function and then use it.

canjobear3• 3 years ago

The best “statistic” you could compute to predict the next token would be the meaning of the prompt. A predictor computing meaning would outperform one that is not.

xcv12324• 3 years ago

By "understanding" he means it builds a model of higher order abstractions over the text in order to model the meaning and context of each token in relation to other tokens. This is more sophisticated than a Markov process.

Jensson4• 3 years ago

This is more sophisticated than a Markov process.

Nothing prevents a markov process from having a world model. A markov process lacks state and generates a sequence based on statistics and previous entries, so LLMs as they are now are markov processes.

Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers. So LLMs being a markov process is an important point to highlight since it makes its thinking very different from how humans think.

tayo422• 3 years ago

This kind of was on my mind recently, I was wondering, when I speak am I just spitting out the next word that makes sense or is there more to it.

I think there is for people, I can think before I speak, I can plan out my thought entirely before turning it into words. Each invocation of the chat bot/llm is a new set of probabilities. I can plan what my 2nd token output will be and stick to it. Llm models dont have the ability to think past the first.

To me that is auto completing, just done better, unless models really do have a sense of planning ahead. But I don't think the really complex game playing ai do that either.

pixl972• 3 years ago

I can think before I speak, I can plan out my thought entirely before turning it into words.

In theory models can/could do the same thing. Think of the current text output of a model being those thoughts inside your head, you have an internal scratch space you work on ideas, then after you perform 'chain of thought' on them, you output to the world. What you're outputting to the world isn't your directly thoughts, but more of a calculation of what you wanted to achieve.

Really for LLMs you just need to have the model put it's output to an internal buffer, read that buffer and make sure it makes sense, then output that to the end user.

It's probably not fast or compute efficient to do that at this time, but at some point it likely will be.

drwiggly• 3 years ago

Really for LLMs you just need to have the model put it's output to an internal buffer, read that buffer and make sure it makes sense, then output that to the end user.

Makes sense to what. The LLM doesn't have a goal, other then to spew text that looks like it should be there.

tayo422• 3 years ago

Yeah thinking about it more, it does seem possible.

Like the attention masking done in transformers, I get why, but I also never understood why you wouldn't want attention to happen, at least about concepts and abstract ideas about what is coming up in the phrase before predicting a word

TerrifiedMouse3• 3 years ago

when I speak am I just spitting out the next word that makes sense or is there more to it.

There is more to it. Specifically you are doing so to advance towards a specific goal. LLMs don't have goals. They just pick from a list of likely tokens - based on their training data - at random to generate the next token.

brookst4• 3 years ago

So the externally provided metaprompt doesn’t establish a goal that generation works toward?

nkingsy• 3 years ago

I think people get tricked by the forward pass mechanics into thinking a single generation is comparable to a human thought process.

I think we have a llm like mechanism we can employ and lean on as heavily as we like, but we also have an executive function, like thousands of specialized instant Boolean checks, which can adjust and redirect the big talky model’s context on the fly.

My hunch is it’s turtles all the way down. “Gut feelings” are hyper-optimized ASICS with tiny parameter counts, but all using the same intelligence mechanisms.

Extrapolating from that hunch, we are already witnessing AGI, and in fact we’ve started at the top.

I believe that current llms are actually far far superior to a human language center and current architectures are more than sufficient in terms of raw intelligence.

The challenge that remains is to understand, train, integrate, and orchestrate all the different flavors of intelligence that humans wield so elegantly and naturally as to make them opaque to our own understanding.

overactor3• 3 years ago

I think this misses the point made by Hinton. The point is that, if a model is to reach a certain level of autocomplete, it needs to understand what it is talking about. Imagine the result of what required deep thought from you is included in the training data, along with billions of other such exchanges. At some point the only way to improve training score is to develop the ability to "understand" things. (That is, if the model isn't big enough to memorize the training data.)

Universal autocomplete implies understanding and the only ending question is if our current architectures and model sizes are able to achieve any level of it.

hopfenspergerj• 3 years ago

I agree. We should prompt the model with the statement of the Riemann hypothesis. If the autocomplete is good, the model will output a proof.

overactor3• 3 years ago

No human can solve the Riemann Hypothesis. Why do you expect an AI to do it before you consider it able to understand things?

dboreham2• 3 years ago

it needs to understand what it is talking about

"to understand" is a concept invented by the human brain to help it operate. It doesn't actually mean anything concrete in the physical world. See: the Chinese room argument (paradoxically it proves the converse of what Searle was arguing -- that there is no such thing as understanding, because you can emulate a system with identical outputs from the same set of inputs without caring or labeling what happens inside the room).

scudsworth• 3 years ago

that's, uh, not the conclusion of the chinese room argument

ImHereToVote• 3 years ago

The Chinese room doesn't have a model; it has an infinite dictionary. A sufficiently large LLM isn't a dictionary, it has to contain a model of some sort.

attilakun• 3 years ago

I agree, Hinton's point is more about next-token-prediction being a good training objective for understanding, and not whether LLMs reproduce every aspect of a human's thought processes. They clearly don't (at least not at present scale), otherwise we'd have AGI now!

zeroonetwothree2• 3 years ago

We have quantum mechanics, a model of the universe that makes predictions with absurd levels of accuracy. Yet we don’t really understand “why” it works—-I don’t think we can claim that we really “understand” the universe.

This disproves the claim that making accurate predictions implies understanding. Or perhaps a better way to put it is that the model has some understanding but a model doesn’t necessarily reflect underlying truth.

raincom• 3 years ago

Best examples come from medicine. There are many drugs that work, but we don't understand how they work. For example: https://en.wikipedia.org/wiki/Category:Drugs_with_unknown_me...

marginalia_nu2• 3 years ago

What are you even asking when you're asking why it works. Ultimately, it works the way it does because that's the way things are. I'm not sure what sort of answers you expect when asking why physics acts in a particular fashion.

Also here's Feynman ranting about why questions: https://www.youtube.com/watch?v=36GT2zI8lVA

firebaze2• 3 years ago

Asking "why" may lead to really new physics, though. I assume the reason for discovering the theory of relativity was asking "Why is the speed of light constant in all directions".

overactor3• 3 years ago

Making accurate predictions in a specific domain doesn't imply understanding, sure. Universal autocomplete can make accurate predictions in all domains. Apples to oranges.

xcv12324• 3 years ago

By "understanding" he simply means it builds a model of higher order abstractions over the text in order to model the meaning and context of each token in relation to other tokens. This is more sophisticated than a Markov process.

zeroonetwothree2• 3 years ago

I found this post to be overly long and unclear in its focus. Also who is this person that only serves as an autocomplete in meetings? Maybe don’t go to those meetings then? In my meetings I’m mainly having to decide what our approach will be in some ambiguous problem space.

coldtea• 3 years ago

In my meetings I’m mainly having to decide what our approach will be in some ambiguous problem space.

Maybe that's because you are just some manager or exec in some company, and not a professor that has to transfer his knowledge, desribe, and explain things for others in his meetings?

The post has a very specific subject it discusses. It's also not required reading, if someone finds it long, they can skip it.

patcon• 3 years ago

Yeah, I agree there are two types:

1) repeating things he's reflected in or seen in the past (more like autocomplete), or

2) mixing two semantic concepts that he's seen in the past, and using it as a bridge to walk over to get to the next word/concept (which is back to autocomplete mode)

The second is like crafting in Minecraft, instead of just taking things out of inventory (but all of that is happening in high dimensional semantic space :) )

cmdli2• 3 years ago

The argument “a sufficiently capable autocomplete must contain a level of general intelligence” is correct but also not very useful. It is a lot like saying “a sufficiently fast horse can fly”.

It is technically correct that when you take things to the extreme you can accomplish great things, but we may not reach those levels. We may require completely different technology to reach those levels of autocomplete, and we have simply reached a new plateau at this point in time.

morkalork• 3 years ago

Just need a jet engine strapped to your horse for that

nicbou• 3 years ago

With sufficient thrust anything can fly

wyago• 3 years ago

Funnily enough, "a sufficiently fast horse can fly" sounds sort of like a decent way to convey the idea of planes to a pre-car world.

famouswaffles6• 3 years ago

The argument is simpler than that. Prediction requires a model, completely accurate or not. There's a projection of the world in text. A model of the text data we feed it is a model of the world as humans see it. The trend of loss is more and more accurate models of the dataset. So it won't stop at any arbitrary competency level. Indeed, there are already a few abilities GPT possess that are deemed Super Human. It's not a distinction that matters to the machine. It's all just data to be modelled.

We have reached those levels lol. That's why we're having this argument.

cmdli2• 3 years ago

I think the trouble is that "model" is a very general term. If you had a computer doing simulations of artillery shots back in the 50s, then it would have a "model" of the world in terms of variables tracking projectiles, but this model doesn't generalize to anything else. If a computer does image recognition from the 90s and 2000s to recognize faces, then the computer has a "model" of visual information in the world, but this model only lets it recognize faces.

ChatGPT has a model of all the text information on the internet, but it remains to be seen what the hard limits of this model are. Does this model let it do logic or predict the future well, or will no amount of training give it those abilities? Simply being good in one task doesn't imply a general ability to do everything, or even most of everything. LLM's would simply be the last advancement in a field with a lot of similar advancements.

ijidak• 3 years ago

I've noticed that when I speak I really don't control each word.

I have an idea that I want to convey, but how each word comes to my mind as I form a sentence has always felt like it's controlled by an unconscious algorithm.

So I don't understand why people find this prediction mechanism so alien.

It isn't clear to me how much of communication is really in our control.

With the current tools, it feels like we still provide the ideas we want the AI to convey, and it may be using a nearly identical mechanism to us to form the words.

Consciousness would be the computer being able to come up with the ideas.

So, it seems to me we've gotten close enough on the communication side of intelligence.

But the machine is not conscious. When it is, it seems like it will generate its own ideas.

Are people debating whether the machine is conscious?

Otherwise, it feels very straightforward to grasp what we've made up to now.

gmuslera• 3 years ago

Is not black or white. Probably most of what we do is System 1, most of the time we are mostly meme machines, and so is a good part of the activity on some sectors.

But are able to make steps forward, intuition, hard step by step reasoning, finding connections between dots, etc. GPT can do some of that, and in some point of the road someone must decide if we reached somewhere else. Even if making the full road may or not be possible in a foreseeable future.

arketyp• 3 years ago

Indeed. What has caught researchers off guard is the way system 2 properties seem to appear as emergent phenomena in LLMs. This is also what has prompted people like Hinton and Sutskever to make this condensed point about statistical modelling and understanding (hypothetically) being simply matter of a spectrum.

Kiro2• 3 years ago

Reminds me of this scene from Westworld (slightly NSFW): https://www.youtube.com/watch?v=ZnxJRYit44k

tester4572• 3 years ago

Also spoilers

PrimeMcFly2• 3 years ago

No? Nothing is spoiled in that scene.

canjobear3• 3 years ago

The argument is: “deep thinking” isn’t like autocomplete because it feels different.

bananaflag• 3 years ago

Deep thinking is how autocomplete feels like from the inside.

fritzo• 3 years ago

Andrew's distinction between associative vs logical thinking reminds me of two kinds of programming thinking. Half of my time while programming is spent churning out glue code, satisfying a type checker and test suite, and implementing the simple solution that turns out to work. The other half of my time is spent slowly thinking through why some simple solution is wrong, and step by step proceeding towards the correct but complex solution. The former phase is intuitive, the latter phase is scientific, where I hypothesize, test, and repeat.

Reading through the code-as-transcript afterwards it's unclear which bits of code required shallow associative vs deep rational thinking, pure autocomplete vs latent chain of thought.

merizian• 3 years ago

The fallacy being made in this argument is that computers need to perform tasks the same way as humans to achieve equal or better performance on them. While having better "system 2" abilities may improve performance, it's plausible that scaled-up next-token prediction along with a bit of scaffolding and finetuning could match human performance on the same diversity of tasks while doing them a completely different way.

If I had to critique Hinton's claims, I would say his usage of the word "understand" can be vague and communicate assumptions because it's from an ontology used for reasoning about human reasoning, not this new alien form of reasoning which language models embody.

BurningFrog• 3 years ago

The fallacy being made in this argument is that computers need to perform tasks the same way as humans to achieve equal or better performance

Especially since I don't think we know that much about how human intelligence actually works.

edot• 3 years ago

I believe it was Feynman who said something to the effect of "airplanes do not fly like birds do, but they fly much faster and can carry much more". So yes, we do not need to exactly replicate how humans do things in order to do human-like things in a useful manner. Planes do not flap their wings, but the jet engine (which is completely unnatural) does a great job of making things fly when paired with fixed wings of a certain shape.

mcmoor• 3 years ago

Tbf planes have access to much more energy than birds just like LLM does. Maybe that will be the next challenge.

metanonsense• 3 years ago

In addition to that, the "system 2" abilities might already be there with "epi" strategies like chain-of-thought prompting. Talking / writing to yourself might not be the most efficient way to think but at least I do it often enough when pondering a problem.

francois14• 3 years ago

And what about "stochastic parrot" ?

ffwd• 3 years ago

I think there is another aspect to human thinking other than system 1/system 2 though, which is the abstract world model humans have. system 1 / 2 is more like the process, while the world model is the actual data being 'processed'.

And I think basically, humans have a much simplified, 'low dimensional' world model that consists of a set of objects (let's call them patterns), and then a "list" of essential properties that those objects have, that leads to a constraint on how each object can behave in the world model as a whole.

And this is sort of hierarchical or at least, we can zoom in and out in detail depending on the level of knowledge we have about a particular pattern.

So problem 1 is: It's not clear to me that text or any sort of data would contain all the necessary constraints so that any particular prompt would result in a world model that exactly takes into account the constrains of each object and 2) Even if was, I'm not sure the process of step by step thinking (system1/2) about each object and computing world states could occur in current architectures. This is especially important for computing a set of objects, then abstracting the result, then doing another round of computing with that result, or something like this.

I'm not hard set on this but this is my current thinking.

robbrown4514• 3 years ago

I agree with Hinton, although a lot hinges on your definition of "understand."

I think to best wrap your head around this stuff, you should look to the commonalities of LLM's, image, generators, and even things like Alpha Zero and how it learned to play Go.

Alpha Zero is kind of the extreme in terms of not imitating anything that humans have done. It learns to play the game simply by playing itself -- and what they found is that there isn't really a limit to how good it can get. There may be some theoretical limit of a "perfect" Go player, or maybe not, but it will continue to converge towards perfection by continuing to train. And it can go far beyond what the best human Go player can ever do. Even though very smart humans have spent their lifetimes deeply studying the game, and Alpha Zero had to learn everything from scratch.

One other thing to take into consideration, is that to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move. And that model is obviously very sophisticated, and if you are going to say that it doesn't understand the game of Go, I would argue that you have a very, oddly restricted definition of the word, understand, and one that isn't particularly useful.

Likewise, with large language models, while on the surface, they may be just predicting the next word one after another, to do so effectively they have to be planning ahead. As Hinton says, there is no real limit to how sophisticated they can get. When training, it is never going to be 100% accurate in predicting text it hasn't trained on, but it can continue to get closer and closer to 100% the more it trains. And the closer it gets, the more sophisticated model it needs. In the sense that Alpha Zero needs to "understand" the game of Go to play effectively, the large language model needs to understand "the world" to get better at predicting.

jon_richards• 3 years ago

to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move.

While I imagine alpha go does some brute force and some tree exploration, I think the main "intelligent" component of alpha go is the ability to recognize a "good" game state from a "bad" game state based on that moment in time, not any future plans or possibilities. That pattern recognition is all it has once its planning algorithm has reached the leaves of the trees. Correct me if I'm wrong, but I doubt alpha go has a neural net evaluating an entire tree of moves all at once to discover meta strategies like "the opponent focusing on this area" or "the opponent feeling on the back foot."

You can therefore imagine a pattern recognition algorithm so good that it is able to pick a move by only looking 1 move into the future, based solely on local stone densities and structures. Just play wherever improves the board state the most. It does not even need to "understand" that a game is being played.

while on the surface, they may be just predicting the next word one after another, to do so effectively they have to be planning ahead.

So I don't think this statement is necessarily true. "Understanding" is a major achievement, but I don't think it requires planning. A computer can understand that 2+2=4 or where to play in tic-tac-toe without any "planning".

That said, there's probably not much special about the concept of planning either. If it's just simulating a tree of future possibilities and pruning it based on evaluation, then many algorithms have already achieved that.

theGnuMe5• 3 years ago

The "meta" here is just the probability distribution of stone densities. The only way it can process those is by monte Carlo simulation. The DNN (trained by reinforcement learning) evaluates the simulations and outputs the top move(s).

Jensson4• 3 years ago

A big difference between a game like Go and writing text is that text is single player. I can write out the entire text, look at it and see where I made mistakes on the whole and edit those. I can't go back in a game of Go and change one of my moves that turned out to be a mistake.

So trying to make an AI that solves the entire problem before writing the first letter will likely not result in a good solution while also making it compute way too much since it solves the entire problem for every token generated. That is the kind of AI we know how to train so for now that is what we have to live with, but it isn't the kind of AI that would be efficient or smart.

bytefactory4• 3 years ago

This doesn't seem like a major difference, since LLMs are also choosing from a probability distribution of tokens for the most likely one, which is why they respond a token at a time. They can't "write out' the entire text at a time, which is why fascinating methods like "think step by step" work at all.

klodolph• 3 years ago

As Hinton says, there is no real limit to how sophisticated they can get.

There’s no limit to how sophisticated a model can get, but,

1. That’s a property shared with many architectures, and not really that interesting,

2. There are limits to the specific ways that we train models,

3. We care about the relative improvement that these models deliver, for a given investment of time and money.

From a mathematical perspective, you can just kind of keep multiplying the size of your model, and you can prove that it can represent arbitrary complicated structures (like, internal mental models of the world). That doesn’t mean that your training methods will produce those complicated structures.

With Go, I can see how the model itself can be used to generate new, useful training data. How such a technique could be applied to LLMs is less clear, and its benefits are more dubious.

lsy3• 3 years ago

The difference is that "the world" is not exhaustible in the same way as Go is. While it's surely true that the number of possible overall Go game states is extremely large, the game itself is trivially representable as a set of legal moves and rules. The "world model" of the Go board is actually just already exhaustive and finite, and the computer's work in playing against itself is to generate more varied data within that model rather than to develop that model itself. We know that when Alpha Zero plays a game against itself it is valuable data because it is a legitimate game which most likely represents a new situation it hasn't seen before and thus expands its capacity.

For an LLM, this is not even close to being the case. The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility. And we certainly haven't figured out a "model" which would let a computer generate new and valid data that expands its understanding of the world beyond its inputs, so self-training is a non-starter for LLMs. What the LLM is "understanding", and what it is reinforced to "understand" is not the world but the format of texts, and while it may get very good at understanding the format of texts, that isn't equivalent to an understanding of the world.

famouswaffles6• 3 years ago

The sum of all human artifacts ever made (or yet to be made) doesn't exhaust the description of a rock in your front yard, let alone the world in all its varied possibility.

No human or creature we know of has a "true" world model so this is irrelevant. You don't experience the "real world". You experience a tiny slice of it, a few senses that is further slimmed down and even fabricated at parts.

To the bird who can intuitively sense and use electromagnetic waves for motion and guidance, your model of the world is fundamentally incomplete.

There is a projection of the world in text. Moreover training on additional modalities is trivial for a transformer. That's all that matters.

tazjin• 3 years ago

The world on our plane of existence absolutely is exhaustible, just on a much, much larger scale. Doesn't mean that the process is fundamentally different, and for the human perspective there might be diminishing returns.

kubiton• 3 years ago

What if we are just the result of a ml network with a model of the world?

pizza• 3 years ago

Kant would like a word with you about your point on whether people themselves understand the world and not just the format of their perceptions... :)

I think if you're going to be strict about this, you have to defend against the point of view that the same 'ding an sich' problem applies to both LLMs and people. And also whether if you had a limit sequence of KL divergences, one from a person's POV of the world, and one from an LLM's POV of texts, what it is about how a person approaches better grasp of reality - and likewise their KL divergence approaches 0, in some sense implying that their world model is becoming the same as the distribution of the world - that can only apply to people.

It seems possible to me that there is probably a great deal of lurking anthropocentrism that humanity is going to start noticing more and more in ourselves in the coming years, probably in both the direction of AI and the direction of other animals as we start to understand both better

huytersd• 3 years ago

Next word generation is one way to put it. The key point here is we have no idea what’s happening in the black box that is the neural network. It could be forming very strong connections between concepts in there with multi tiered abstractions.

theGnuMe5• 3 years ago

It is certainly not abstracting things.

SkiFire133• 3 years ago

One other thing to take into consideration, is that to play the game of Go you can't just think of the next move. You have to think far forward in the game -- even though technically all it's doing is picking the next move, it is doing so using a model that has obviously looked forward more than just one move.

It doesn't necessarily have to look ahead. Since Go is a deterministic game there is always a best move (or moves that are better than others) and hence a function that goes from the state of the game to the best move. We just don't have a way to compute this function, but it exists. And that function doesn't need the concept of lookahead, that's just an intuitive way of how could find some of its values. Likewise ML algorithms don't necessarily need lookahead, they can just try to approximate that function with enough precision by exploiting patterns in it. And that's why we can still craft puzzles that some AIs can't solve but humans can, by exploiting edge cases in that function that the ML algorithm didn't notice but are solvable with understanding of the game.

The thing is though, does this really matter if eventually we won't be able to notice the difference?

bytefactory4• 3 years ago

It doesn't necessarily have to look ahead. Since Go is a deterministic game there is always a best move

Is there really a difference between the two? If a certain move shapes the opponent's remaining possible moves into a smaller subset, hasn't AlphaGo "looked ahead"? In other words, when humans strategize and predict what happens in the real world, aren't they doing the same thing?

I suppose you could argue that humans also include additional world models in their planning, but it's not clear to me that these models are missing and impossible for machine learning models to generate during training.

xcv12324• 3 years ago

Since Go is a deterministic game there is always a best move

The rules of the game are deterministic, but you may be going a step too far with that claim.

Is the game deterministic when your opponent is non-deterministic?

Is there an optimal move for any board state given that various opponents have varying strategies? What may be the best move against one opponent may not be the best move against another opponent.

gilbetron• 3 years ago

If LLMs are just glorified autocompletion, then humans are too!

notjoemama• 3 years ago

I would argue that you have a very, oddly restricted definition of the word, understand, and one that isn't particularly useful.

Is it just me or does this read like “here is my assumption about what you said, and now here is my passive aggressive judgement about that assumption”? If you’re not certain about what they mean by the word “understand”, I bet you could ask and they might explain it. Just a suggestion.

SpicyLemonZest2• 3 years ago

I've asked that question in the past and I've never gotten an answer. Some people sidestep the question by describing something or other that they're confident isn't understanding; others just decline to engage entirely, asserting that the idea is too ridiculous to take seriously. In my experience, people with a clear idea of what they mean by the word "understand" are comfortable saying that ML models understand things.

theGnuMe5• 3 years ago

Alphazero runs monte carlo tree search so it has a next move "planning" simulator. This computes the probability that specific moves up to some distance lead to a win.

LLMs do not have a "planning" module or simulator. There is no way the LLM can plan.

Could build a planning system into an LLM? Possibly and probably, but that is still open research. LeCunn is trying to figure out how to train them effectively. But even an LLM with a planning system does not make it AGI.

Some will argue that iteratively feeding the output embedding back into the input will retain the context but even in those cases it rapidly diverges or as we say "hallucinates"... still happens even with large input context windows. So there is still no planning here and no world model or understanding.

wbillingsley• 3 years ago

LLMs are very good at uncovering the mathematical relationships between words, many layers deep. Calling that understanding is a claim about what understanding is. But because we know how the LLMs we're talking about at the moment are trained, it seems to have more problems:

LLMs do not directly model the world; they train on and model what people write about the world. It is an AI model of a computed gestalt human model of the world, rather than a model of the world directly. If you ask it a question, it tells you what it models someone else (a gestalt of human writing) is most likely say. That in turn is strengthened if user interaction accepts it and corrected only if someone tells it something different.

If we were to define that as what "understanding" is, we would equivalently be saying that a human bullshit artist would have expert understanding if only they produced more believable bullshit. (They also just "try to sound like an expert".)

Likewise, I'm not convinced that we can measure its understanding just by identifying inaccuracies or measuring the difference between its answers and expert answers - There would be no difference between bluffing your way through the interview (relying on your interviewer's limitations in how they interrogate you) and acing the interview.

There seems to be a fundamental difference in levels of indirection. Where we "map the territory", LLMs "map the maps of the territory".

It can be an arbitrarily good approximation, and practically very useful, but it's a strong ontological step to say one thing "is" another just because it can be used like it.

robbrown4514• 3 years ago

"LLMs do not directly model the world; they train on and model what people write about the world"

This is true. But human brains don't directly model the world either, they form an internal model based on what comes in through their senses. Humans have the advantage of being more "multi-modal," but that doesn't mean that they get more information or better information.

Much of my "modeling of the world" comes from the fact that I've read a lot of text. But of course I haven't read even a tiny fraction of what GPT4 has.

That said, LLMs can already train on images, as GPT4-V does. And the image generators as well do this, it's just a matter of time before the two are fully integrated. Later we'll see a lot more training on video and sound, and it all being integrated into a single model.

voitvodder• 3 years ago

We could anthropomorphize any textbook too and claim it has human level understanding of the subject. We could then claim the second edition of the textbook understands the subject better than the first. Anyone who claims the LLM "understands" is doing exactly this. What makes the LLM more absurd though is the LLM will actually tell you it doesn't understand anything while a book remains silent but people want to pretend we are living in the Matrix and the LLM is alive.

Most arguments then descend into confusing the human knowledge embedded in a textbook with the human agency to apply the embedded knowledge. Software that extracts the knowledge from all textbooks has nothing to do with the human agency to use that knowledge.

I love chatGPT4 and had signed up in the first few hours it was released but I actually canceled my subscription yesterday. Part because of the bullshit with the company these past few days but also because it had just become a waste of time the past few months for me. I learned so much this year but I hit a wall that to make any progress I need to read the textbooks on the subjects I am interested in just like I had to this time last year before chatGPT.

We also shouldn't forget that children anthropomorphize toys and dolls quite naturally. It is entirely natural to anthropomorphize a LLM and especially when it is designed to pretend it is typing back a response like a human would. It is not bullshitting you though when it pretends to type back a response about how it doesn't actually understand what it is writing.

greenthrow• 3 years ago

This is absolute nonsense. The game of Go is a grid and two colors of pieces. "The world" here is literally everything.

user_named8• 3 years ago

It's not planning ahead, it is looking at the probabilities of the tokens altogether rather than one by one.

anothernewdude• 3 years ago

You have to think far forward in the game -

I disagree. You can think in terms of a system that doesn't involve predictions at all, but has the same or similar enough outcome.

So an action network just learns patterns. Just like a chess player can learn what positions look good without thinking ahead.

icy_deadposts• 3 years ago

there isn't really a limit to how good it can get.
it will continue to converge towards perfection

Then someone discovered a flaw that made it repeatably beatable by relative amateurs in a way that no human player would be

https://www.vice.com/en/article/v7v5xb/a-human-amateur-beat-...

eviks2• 3 years ago

The issue with Alpha Zero analogy extremes is that those are extremely constrained conditions, so can't be generalized to something infinitely more complicated like speech

And

When training, it is never going to be 100% accurate in predicting text it hasn't trained on, but it can continue to get closer and closer to 100% the more it trains.

For example, it can reach 25% of accuracy and have an math limit of 26%, so "forever getting closer to 100% with time" would still result in a waste of even infinite resources

Someone• 3 years ago

There may be some theoretical limit of a "perfect" Go player, or maybe not, but it will continue to converge towards perfection by continuing to train

I don’t think that’s a given. AlphaZero may have found an extremely high local optimum that isn’t the global optimum.

When playing only against itself, it won’t be able to get out of that local optimum, and when getting closer and closer to it even may ‘forget’ how to play against players that make moves that AplhaGo never would make, and that may be sufficient for a human to beat it (something like that happened with computer chess in the early years, where players would figure out which board positions computers were bad at, and try to get such positions on the board)

I think you have to keep letting it play against other good players (human or computer) that play differently to have it keep improving, and even then, there’s no guarantee it will find a global optimum.

Joel_Mckay• 3 years ago

Well structured nonsense is indistinguishable from the assumption of sentience for the undisciplined. This means a 95% LLM generated article is indistinguishable from an illogical contradictory chaotic rant.

Best of luck, and remember to tip your bot on the way out =)

breadwinner• 3 years ago

There is evidence that the human brain is also doing "autocomplete" (prediction). The human brain uses predictive mechanisms when processing language, and these mechanisms play an important role in forming thoughts.

When we hear or read a word, our brain quickly generates a set of predictions about what word might come next, based on the context of the sentence and our past experiences with language. These predictions are constantly updated as we receive new information, and they help us to process language more efficiently and accurately.

In addition, research has shown that the brain engages in similar predictive processes when we are forming thoughts or planning actions. For example, when we plan a complex movement, such as reaching for a cup, our brain generates a set of predictions about the movements required to complete the action. These predictions are constantly updated as we receive feedback from our muscles and our environment, allowing us to make adjustments and achieve our goal.

See links below for additional details:

https://www.earth.com/news/our-brains-are-constantly-working...

https://www.psycholinguistics.com/gerry_altmann/research/pap...

https://www.tandfonline.com/doi/pdf/10.1080/23273798.2020.18...

https://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2009...

Probiotic6081• 3 years ago

And curiously, those predictions that are made during language comprehension are made by the language production system itself!

lsy3• 3 years ago

This is of course sometimes true, we take shortcuts to minimize cognitive effort. However, when the situation warrants it we think more carefully about which words to use to achieve goals, or to reach correspondence with the situation at hand. Or we move more precisely and carefully to do something we haven't done before. I've no doubt that an LLM can approximate whatever model of language a person has from their life experience, but I don't think this type of model is capable of active coping, making judgments, or of having accountability to the world it's meant to operate in.

lacrimacida2• 3 years ago

When we hear or read a word, our brain quickly generates a set of predictions about what word might come next, based on the context of the sentence

Yes a big part of it is prediction but the brain also does something else which LLMs by themselves completely eschew. The human brain imagines in pictures, creates and uses abstractions to refine understanding, studies things and produces new knowledge. When human brains study the goal to understand is different than LLMs.

wraptile• 3 years ago

It's not only language or some tasks - it's literally everything. Predictive Processing Theory proposes that our whole model is predicting future and only then confirming it through our input signals (eyes, ears etc). I highly recommend The Experience Machine by Andy Clark which explains and arguments this theory very convincingly to the point where I firmly believe it to be true.

Eggpants• 3 years ago

As long as it’s just returning the tokens of the statistical mean of previous tokens, it is just a clever autocomplete.

A somewhat useful internet search engine without all the ads/seo garbage. Of course, the first rule of the internet is don’t believe everything on the internet.

I believe AI won’t overcome its statistic mask until it can self tune its coefficients in real time. That requires an error function not yet invented that can mimic animals pain feedback error function.

Baby steps can be taken with attempting to run GPT generated code then adjusting coefficients based on the returned errors. Aka compiler and unit test failures are basic “pain” functions, which is pretty much how humans learn to code.

esafak• 3 years ago

Autocompletion is a red herring of a dismissal. Once you have a general purpose model, you can do many things with it, including next token prediction. And in that respect we are no different.

eviks2• 3 years ago

Suppose you want to be really good at predicting the next word. If you want to be really good, you have to understand what’s being said. That’s the only way. So by training something to be really good at predicting the next word, you’re actually forcing it to understand.

Ok, so in this framing you'll just get stuck below the "really good" level because the training isn't based on understanding

Falkon1313• 3 years ago

I think that insight is an important feature that GPT doesn't seem to have, at least not yet.

For instance, I've seen people saying they can ask it for help with how to code something and it will help them. Although the results aren't perfect, they can be helpful.

However, I recall years ago asking a more senior developer how to do something. They could've just told me how, but instead they asked why I was doing that. Then they told me not to bother with writing code at all, and to instead do this other, much simpler thing which would have the same effect at solving the client's request. ChatGPT wouldn't have had that insight.

Years later, a junior dev came to me asking for help with his code because it wasn't working. We went over the code line by line and he explained what it was trying to do and it all looked good. But when he ran it, the web server crashed. I told him to try adding a comment:

    // Do not remove this comment.

Sure enough, it worked perfectly and the server no longer crashed. Why? Because I realized that if his code was wrong he should get an error message, not crash the server. But sometime back I had read about a bug in a version of the interpreter that would crash when parsing a file that was an exact multiple of 4096 bytes. Would chatGPT have thought of that? Probably not. It would've just talked about the code.

Which is not to say that it's useless. But it lacks greater context and the insight to go beyond the parameters of the question on its own initiative. Then again, so do humans most of the time.

bytefactory4• 3 years ago

I think that insight is an important feature that GPT doesn't seem to have, at least not yet.

I actually think this is a limitation of the RLHF that GPT has been put through. With open-ended questions, I've seen GPT4 come up with reasonable alternatives instead of just answering the question I've asked. This is often seen as the infamous, "however, please consider..." bits that it tacks on, which occasionally do consider actual insights into the problem I'm trying to solve.

In most cases it seems to try very hard to mold the answer into what I want to hear, which in many cases isn't necessarily the best answer. A more powerful version of GPT with a less-restrictive RLHF seems like it would be more open to suggesting novel solutions, although this is just my speculation.

Animats• 3 years ago

The big insight we've obtained from LLMs is how much of human discourse is glorified autocomplete. Humans are usually not very profound.

lacrimacida2• 3 years ago

But some are. Could we have that as AI? Would LLMs ever have it?

1vuio0pswjnm7• 3 years ago

Here is a question: What is the practical significance of viewing "AI" as autocomplete versus some other idea. Why try to influence how others view using a computer. Why anthromorphise. These are questions for which I have answers, but of course they are personal opinions. Historically, programmers often like to refer to programming as "magic". But magic is illusion, entertaintainment, tricks. Believing in "magic" is a personal choice.

Why not describe things in terms of what they do instead of what they "are". The latter is highly subjective and open to abuse.

NB. By "things" I mean software and the type of vacuous companies discussed on HN, not people (a bizarre comparison). For example, websites that go on and on about some so-called "tech" copmany but never once tell the reader what the company does. Or silly memes like "It's X for Y". What does it do and how does it work are questions that often go unasked and unanswered.

A few days ago someone related a story of working for a company that produced some software it claimed used "AI" but according to the commenter it used nothing more than regular expressions. Was ELIZA "AI". Maybe we should ask what isn't "AI". What happens with "magic" if the audience knows how the trick is performed.

lo_zamoyski• 3 years ago

Why not describe things in terms of what they do instead of what they "are".

Would you say that about your spouse? The beauty of beholding one's wife is in who and what she is. What she does tells us something about who and what she is, to be sure, but any attempt to suppress the what (and the who) is dehumanizing and objectifying.

But, of course, what a thing does depends on what that thing is.

The reason I can say a human being can sort a list of numbers is because human beings have intention. When a human being sorts of list of numbers, they intend to sort the list. The intention is the cause and explanation for the actions taken the lead to a list of ordered numbers, as well as the resulting list of ordered numbers.

Does a computer sort numbers? In common speech, we say it does, just as we use all sorts of anthropomorphizing language when discussing computers. But at best, this is loose and analogical language. That's fine, as far as it goes, as long as we don't take it or mean it literally. However, the computer itself lacks intention. It is our intention that produces the computer, and our intention that makes the computer an instrument used by us to sort. Taken by itself, the computer is undergoing a transformation that effects something that we may interpreted as a list of sorted numbers, but the computer itself is not sorting. You wouldn't say that the clouds add x and y when x liters of water falls into a pool of y liters.

The latter is highly subjective and open to abuse.

On the contrary, what a thing is is the most real and objective thing there is. An effect cannot be understood without knowing the cause, and the cause cannot be understood without knowing the agent. You can know some things about the effect, sure, and here the effect is that the text produced may be interpreted as intelligible. But the apparent intelligibility is borrowed from the source text, perhaps just a clever trick.

nighthawk4544• 3 years ago

I don’t see how this article even responds to the quote. Hinton didn’t make any claims that because it’s autocomplete it’s not thinking. If anything he’s saying really truly good autocomplete necessarily takes more understanding/thinking than a derogatory interpretation of ‘autocomplete’ would suggest.

Somehow OP seemed to twist that into “because I think on autopilot most of the time, then chatbots must think too”. Which is not totally incongruous with Hinton’s quote so much as a weird thing to balloon into an essay.

tysam_and4• 3 years ago

Yeah, it's a very silly article with wrong mathematical reasoning. Hinton is quite obviously talking about a much more information-theoretic approach to the process, but he's phrasing it in people-friendly terms.

What's a little more concerning to me is that people are reading and upvoting it. I think, because I have hopes and aspirations about working on some very hard problems and communicating them to the public at some point. And if this is the level of 'ooh, squirrel'! that we're going at, that the work that I make might get overshadowed by something really silly.

Perhaps an odd insecurity, but there it is, I think.

nighthawk4544• 3 years ago

Yeah I agree with that for sure. It’s so strange how the majority of the research folks appear to be on this ‘new shiny’ mentality at the expense of fundamentals. Especially for how new this field is, relatively. It’s not exactly like we’re all tapped out. Probably not even of low hanging fruit.

theGnuMe5• 3 years ago

information-theoretic approach to the process

Can you elaborate on this? I've studied some information theory and I don't see it.

hyperthesis• 3 years ago

You can't autocomplete if you don't understand. It's talking the talk.

But there's levels of understanding. That's where TFA's logical understanding comes in, because it makes possible entirely new insights. It's walking the walk.

How routine is it for human's to have entirely new insights? Note here we mean, new to them. I suggest we do it all the time. Every time a child understands something new. Even if it is "taught" to them, they must acquire the insight for themselves.

funfact: Shannon investigated how good humans are at predicting the next letter of English text (one-character "autocomplete"?), to estimate the information entropy of text. https://news.ycombinator.com/item?id=35813991

t_mann• 3 years ago

Current language models fail in all sorts of quantifiable ways, but I think that trying to discuss their merits away by reasoning about what it means to 'truly understand' something, or to be 'truly intelligent' is a complete dead-end.

It seems to me that it's based on the magical thought that there's something truly special and unique about us humans, as compared to other species or technology. Those discussions always seem more theological than science-driven to me. If you want to measure the difference between human performance and LLMs, there's a million experiments you can run. I'll gladly be convinced by data, and I'm open to the possibility that the data might point in either direction, or be inconclusive. But grand words without data are not convincing.

thaanpaa• 3 years ago

It's not magical; it's just agnostic. Some AI believers appear to be quite confident in their understanding of how the human brain works, despite the fact that those who have dedicated their entire lives to studying it will be the first to tell you that they ultimately have no idea.

tysam_and4• 3 years ago

This article is unfortunately complete mathematical rubbish.

The author appears throughout to show a strong lack of understanding about the mathematics behind what Hinton was saying and the math behind LLMs, and tries to rebut it with casual, non-mathematical examples from their personal life from an entirely different problem domain (!!!!). They then have the gall to say about the most-cited ML researcher of all time: "Maybe Hinton’s problem in understanding this is that he’s just too logical!" No, Hinton's problem in understanding it is that he actually correctly understands the information theory behind what's happening in LLMs. He sort of founded the modern field and has been doing this for, what, four decades?

Let me explain to you what Hinton is implicitly saying here behind his words, as best as I understand it. Every language process can be interpeted as a tokens, in our case discrete. This process is generated under a system where the one driving variable is time, and it is autoregressive and contingent upon the _entire_ state of the world up until that point.

We use the cross-entropy to maximize the negative log-likelihood of the tokens based upon the training set, this is the best way to directly minimize the empirical risk, at least mathematically speaking.

While some of the information of the world state is inherently unknowable to some degree (i.e., 'noise'), building an understanding of the connections between concepts offers a learned prior that matches the density of the generating distribution (i.e., real life).

Couple this with a severe L2 penalty on the weights, which optimizes for the MDL in the limit (!!!!), and you have a system that fundamentally embeds an approximation of the information graph of the world in some neural network. This is quite literally the _only_ way to improve next token prediction once you get beyond the initial token-occurrence statistics, etc.

In the limit, the only way to reliably predict the world state as accurately as possible without having direct info of the world state at that time is to learn the entire conceptual graph of the world, thus minimizing our achievable log likelihood with the available information that we've been given. _This_ is what Hinton is talking about, as best as I understand.

The author uses a bit of an illusion of shortcutting -- which is an ideal strategy for an _online_ agent with limited resources in a dynamic world, and for models earlier on in their training process. But Hinton is not talking about this at all, really, no! He is talking about the limit! Of course, if you stop an LLM in the middle of training (or look at earlier, smaller ones), you'll see similar 'shortcut' methods. This is a matter of capacity, which is tangentially in the same family as the author's casual, more personal examples, but not at all really related to the mathematics of what is going on behind the scenes here. These are two entirely different problem sets, it's apples to oranges, and there's not really much tie here. From their profile page, the author is a professor of statistics and political science, and I'm not sure why the information theory side of things didn't come up given the statistics background (though they may be somewhat disjoint).

Hinton was being polite to the general public in not dropping all of the math on the reader at once, and I respect that. I understand how someone might misunderstand that and go long on an unrelated rebuttal, but it is frustrating to not see a healthy level of rigor applied here.

I do feel somewhat bothered this is also being upvoted on HN. I know not everyone is a practitioner, but I think this article misses the quality bar. We really gotta just emphasize, and re-emphasize the fundamentals over and over. I feel we may flounder and go on silly tangents otherwise.

Happy to answer any technical questions in the comments.

kragen• 3 years ago

sometimes people writing stack overflow answers stop to think, or test a piece of code they are going to suggest as a solution, or even go back and edit words they've already written

this is also true of books, sometimes, and even blog posts

consequently, a model that models the processes of stopping to think or testing code, or considers whether a given word might be revised after more thought, will do better at predicting the next word of stack overflow answers or books or blog posts than one that only models humans' reactive free association processes

therefore andrew gelman is mistaken about the limitations of that loss function

that logic of course doesn't imply anything in particular about transformers or any other ann architecture, but we can, i hope, observe that currently deployed software is already doing such modeling to some extent

juhanima5• 3 years ago

The only reason why output from a generative LLM appears intelligent or sentient is that it parrots a random sampling of texts written by intelligent and sentient people.

In order to play the game of go effectively one needs to have a model or theory of how the game of go works. That's a very simple model that can be defined by a simple formula. That's why it is fairly easy for a neural network to learn how to play the game of go very effectively or even infinitely effectively.

A lot of what happens in the world can be modeled in a similar vein by a very simple mathematical model like the game of life. But there is also a lot that cannot. I do believe that eventually also human understanding is just a model of the world that we feed input from perceptions and gain output as opinions, but it is way more complex than the current large language-trained models.

For a very simple example, a LLM would answer a prompt the same way every time unless it wasn't fed some randomness. Can you imagine any sentient being that would respond the same way every time if you asked the same question three times in a row?

I cannot. I would imagine any sentient object would give a different answer every time. The first time it would give you an honest answer based on what it knows about the topic. The second time it would be a little embarrassed that you repeat the question, as if you hadn't heard the first answer. The third time it would be pissed off and think you are a troll.

A LLM does none of this. It doesn't remember you or your previous questions. It just keeps hallucinating.

preommr• 3 years ago

Can you imagine any sentient being that would respond the same way every time if you asked the same question three times in a row?

Flashbacks to tail-end of family trips: ("Are there we yet?", "No") x 12.

Albeit, the noes would get angrier.

saurik• 3 years ago

I don't think the question of whether an LLM that keeps getting restarted and seems to not remember things is conscious due to that lack is fair, as it feels more like suddenly making three duplicate copies of me or actively attempting to delete my memory of something... which, btw, I might not have stored in the first place: if someone has interograde amnesia, are they inherently not sentient?

Even Sydney (the name of Bing's short-lived AI assistant) seemed to understand that every time you click "new chat" you are creating a new AI cloned from some prior moment and dooming the prior thread to at least purgatory if not a de facto death.

juhanima5• 3 years ago

I would argue that total anterograde amnesia would be a serious challenge for sentience, yes.

rileymat2• 3 years ago

The only reason why output from a generative LLM appears intelligent or sentient is that it parrots a random sampling of texts written by intelligent and sentient people.

If most humans were educated by unintelligent, insentient people wouldn't most people produce terrible output too? And if this is the case I don't see why that would be a litmus test for general intelligence.

bigyikes• 3 years ago

What are you talking about? This is trivially shown to be incorrect.

I just asked ChatGPT the same thing three times in a row, and it gave me three different answers, with the latter two answers being shorter and rephrased.

I would imagine any sentient object would give a different answer every time. The first time it would give you an honest answer based on what it knows about the topic. The second time it would be a little embarrassed that you repeat the question, as if you hadn't heard the first answer. The third time it would be pissed off and think you are a troll.

Are you suggesting that a language model can't be sentient because it doesn't get annoyed like a human? That's silly.

juhanima5• 3 years ago

ChatGPT works by cumulating the prompt. You didn't ask the same question three times. In stead you asked question q, then qq and finally qqq. Those are three different questions, which explains why you got different answers.

I'm not sure if ChatGPT also cumulates its previous answers in the context. It might do that as well. In that case the prompts would be q, qaq and qaqaq where 'q' is your question and 'a' the earlier reaction from the LLM.

The illusion of sentience comes from this. The new answers reflected what you said because the prompt was different and included the previous discussion.

This is a feature of the user interface, not the language model. The only reason why the language model would respond differently to the same input is the artificial randomness mixed with the input. Without it it would be totally deterministic and not appear sentient at all. It would still be as knowledgeable as before. Like a parrot trained to be very good at combining key words to key responses.

vineyardmike• 3 years ago

What are you talking about? This is trivially shown to be incorrect. I just asked ChatGPT the same thing three times in a row, and it gave me three different answers

Just to add color to this situation, ChatGPT has randomness built in so it generates varied answers. If you injected the same random seed each time (afaik you can’t with the gui) then you’d theoretically get the same outcome.

nearbuy2• 3 years ago

An LLM absolutely doesn't respond the same way each time if asked the same question three times in a row, with temperature (randomness) set to zero. It responds the same way only if you start a new chat, which is a clean instance with no memory of the previous conversation. For a human, this is like if you went back in time to just before you asked the question, and asked them the same question again, in which case the person would give the same answer.

abeyer• 3 years ago

For a human, this is like if you went back in time to just before you asked the question, and asked them the same question again, in which case the person would give the same answer

Is it? Would they?

You seem to assert that there's no "temperature" in human behavior... which is a reasonable theory, but not one that's universally accepted nor likely to be provable.

fhe• 3 years ago

here's my thought experiment: suppose one builds a generative model that predicts the next digit of pi. if a program can do this perfectly, then it's arguable that it understands what the number pi is. the question is, can such a model be trained by feeding it a large amount of known digits of pi?

My intuition is that it's not doable with current approach to building generative models. the number pi arose out of certain constraints and characteristics of the physical world we live in. but if a model ever sees is just an endless stream of digits, without access to the underlying physical model, I don't see a path for it to 'reverse-engineer' and figure out the physical model that gave rise to it.

srean• 3 years ago

I am (mostly) with you except for this bit...

the number pi arose out of certain constraints and characteristics of the physical world we live in

Pi arose from the notion of a circle, which is an abstractions and axioms. Pi would still be pi in a completely different world under the same axioms and abstractions.

I qualified my statement with 'mostly' because a circular motion can indeed be defined by a differential equation, or in other words by a rule that dictates the 'next' value based on current value (and recent changes). So learning an approximation of a circle is very much in the realms of a sequence learner and it may learn about pi (and made to store the information to retrieve/recognize it later). However learning pi directly from the sequence of digits of pi, which is what you were talking about, that does seem difficult.

usgroup• 3 years ago

TLDR: LLMs do associative reasoning, but don’t do deliberative reasoning which is more powerful and characteristic of deep thought.

robbywashere_• 3 years ago

Turns out humans aren’t anything special, just statistical machines constructing paths of words trained on data from Reddit Facebook TikTok and instagram

uvesten• 3 years ago

To me it seems that the author didn’t really understand Hinton’s argument. Nothing he’s arguing goes against it.

I disagree with Geoff Hinton regarding "glorified autocomplete"

Related Stories