The Norvig – Chomsky debate (2017)

http://web.cse.ohio-state.edu/~stiff.4/cse3521/norvig-chomsky.html

by rrampage • 3 years ago

102 74 3 years ago

web.cse.ohio-state.edu

mkoubaa• 3 years ago

I don't know but I hope that Chomsky is less wrong. Because if statistical methods reach an asymptote, we will have no choice but to try to better understand the principles and foundations.

If statistical methods do not reach an asymptote, I don't think we will have the incentive to reach a deeper understanding

rfrey• 3 years ago

This reminds me of something I heard Geoff Hinton say .. that it was a shame and a sorrow that neural networks as currently used worked so well.

LudwigNagasena2• 3 years ago

I think it should be marked as being from 2017. Also, I don't see much point in this article. It just butchers Norvig's article into a bunch of quotes even though his article is quite accessible and not very long.

bsaul3• 3 years ago

Not sure about the article itself, but from an epistemological point of view, i believe this debate will remain one of the most famous of the 21st century (provided NN still keep giving new fantastic results, and doesn’t stop there).

Never in history did we manage to achieve so much about a given complex problem (let’s say producing meaningful text) while at the same time understand so little (chatgpt didn’t provide any single useful result to the linguistic science department).

If this approach spreads to other fields, it will lead to an immense scientific crysis.

narag2• 3 years ago

Actually this explains a lot. Language might be the stuff dreams are made of, but not the stuff consciousness is made of, so a specialized form of perception, a layer on top of visuals and hearing, with just another layer of logic on top of it.

We still have a coarse understanding of brain processes, but if they use parallelism, they could be probabilistic in nature, so these language models would be more similar to ours than it seems.

hackandthink6• 3 years ago

I agree. Start with reading:

https://norvig.com/chomsky.html

pafje• 3 years ago

Actually this seems to be from 2013.

https://github.com/joshuaeckroth/cse3521-website/blob/master...

nuc1e0n10• 3 years ago

I think the thing to note about today's large language models is that they aren't purely statistical. The topology of the neural networks behind them has been explicitly defined by the creators of those systems. They are not 'tabula rasa' as some might suppose them to be.

bsaul3• 3 years ago

Yet their structure being generic for all kind of problems, it doesn’t tell much in itself about the things it managed to « understand ». Much like studying einstein’s brain biology can’t teach you much about general relativity.

nuc1e0n10• 3 years ago

But none of them do have a generic structure. For example, GPT-3 can't produce images from text prompts, and stable diffusion cannot generate language. The possible relationships of words are written into GPT-3's code, in python, by its developers. In a way all this proves is that that written language can convey meaning to people.

jsenn7• 3 years ago

This is true, and interesting, but doesn’t address Chomsky’s concerns. While a LLM has structure, it’s still not as structured—or structured in the same way—as the human language faculty. This is easy to see by observing that LLMs can and do just as easily learn to produce things that are not human language as things that are. For something to count as a model of human language, it has to be able to produce language and not produce non-language.

alecst2• 3 years ago

What do LLMs produce that counts as non-language?

webmaven2• 3 years ago

For something to count as a model of human language, it has to be able to produce language and not produce non-language.

You're arguing that the underlying (untrained) architecture cannot be more flexible than a specific and specialized human brain faculty in order to consider the trained model to be relevant to that specific faculty.

But that ignores the fact that if you destroy many specific brain areas (including Broca's area, which is a nexus for language) early enough the rest of the brain will reroute around the damage and use other less specialized parts to fulfill the function.

The brain may have many optimizations that predispose certain parts for specific functions, but it is actually rather general purpose, at least regarding many of the more evolutionarily recent "higher" functions, including language.

foobarqux10• 3 years ago

Chomsky has already addressed Norvig's objections before he even made them as anyone how has actually listened to what Chomsky has said would know.

Norvig's reply: He agrees, but engineering success often facilitates scientific success.

There has been virtually no progress in understanding of the human language faculty from probabilistic models or LLMs.

Norvig's reply: Science is both description and explanation; you can't have one without the other; in the history of science, the laborious accumulation of data is the usual mode of operation.

Probabilistic models don't describe anything in a way that leads to understanding (as the fact that no progress in understanding has been made shows).

people actually generate and understand language in some rich statistical sense (maybe with statistical models several layers deep, like the modern AI models of speech recognition).

They do not; there are studies which Chomsky cites involving trying to learning "impossible" non-structural languages that give strong evidence that this is not the case.

Norvig's reply: Certain advances in statistical learning methods provide reason to believe that such learning methods will be able to do the job.

It has nothing to do with the human language faculty.

My conclusion is that 100% of these articles and awards are more about "accurately modeling the world" than they are about "providing insight," although they all have some theoretical insight component as well.

If you have a black box machine and you write a paper that says the black box reproduces some natural phenomenon with a billion of its knobs to turned to these specific settings you have wasted everyone's time.

Norvig illustrates that rules about language do not always capture the right phenomena. (i before e)

The fundamental character of human language has nothing to do with spelling.

[Norvig]: so any valid criticism of probabilistic models would have to be because they are too expressive, not because they are not expressive enough.

Yes Chomsky has explicitly said this. Any model that accepts virtually everything is a bad model.

I don't have time to go through the rest.

morelisp3• 3 years ago

As a connoisseur of the Chomsky-Foucault debate summarized as one intellectual giant completely missing the other's point, it's quite funny to see Chomsky in the opposite role here. The more operationalist you are, the more depressingly wrong you get - but also it seems the more likely you are to win debate club antics.

voidhorse• 3 years ago

Unfortunately I think the person who presents the simpler theory, even if it's less correct or less powerful, typically wins debates on theory.

In some sense, theorists are working at different levels of abstraction across temporal dimensions. Foucault was concerned about deeper structures and a longer time horizon than Chomsky was at the time they debated, Chomsky is concerned about deeper issues and a longer time horizon than Norvig. This is why they wind up talking past each other.

alecst2• 3 years ago

They do not; there are studies which Chomsky cites involving trying to learning "impossible" non-structural languages that give strong evidence that this is not the case.

I was looking for these studies. I found some similar stuff by Jennifer Culbertson, along these lines https://www.annualreviews.org/doi/pdf/10.1146/annurev-lingui..., but didn't quite know what to Google. Can you point me to something?

foobarqux10• 3 years ago

I believe Chomsky mentioned two studies in one of the 4 episodes in the "Closer to Truth" series, you'll have to search the transcript for the exact timestamp.

The first is an fMRI study that shows that the brain doesn't engage the language processing centers when trying to understand a made-up non-structural language (i.e. a "statistical language") but does when trying to learn a made-up structural language.

The second is about a man who had brain damage except in the language processing centers. A similar study showed that he could learn made-up structural languages but not "statistical" languages.

Poverty of stimulus arguments might also be relevant. There might be an energy argument in his book "Why Only Us" as well.

fock• 3 years ago

The weirdest thing to me is that a person with a high personal, financial involvement in the subject went and took that quote from an old man (which to your and my understanding only states that these things are not linguistic "science", but they solve problems alright), then created a strawman from thin air (points A-E) to then say "oh, all my arguments are void, statistical models are great, don't you dare criticizing me, you old fool".

And then he went and took "Science", aka the epitome of publish or perish academia and tried to argue that all this looks the same as the thing he does. Oh well, who would have guessed...

The looks on this are weird, even more so as GPT nowadays works wonders, but still doesn't help explaining why and how language evolved (which seems to be the idea of linguistics, no?).

hackandthink6• 3 years ago

Norvig has strong arguments, but this in bad faith:

"Chomsky has a philosophy based on the idea that we should focus on the deep whys and that mere explanations of reality don't matter. In this, Chomsky is in complete agreement with O'Reilly." (O'Really stands for mythmaking, religion or philosophy)

Chomsky is no mysticist - he is an old fashioned scientist looking for a parsimonious theory. Maybe there is no simple theory for language with great explanatory power but there should be some people looking for it.

foobarqux10• 3 years ago

Norvig has strong arguments

What was one of those arguments? I didn't see any.

YeGoblynQueenne2• 3 years ago

> Chomsky is no mysticist - he is an old fashioned scientist looking for a parsimonious theory.

When did that become old fashioned?

LudwigNagasena2• 3 years ago

Have you read the original question by Pinker and the response by Chomsky (http://languagelog.ldc.upenn.edu/myl/PinkerChomskyMIT.html)? It doesn't look like a strawman, though it's a bit hard to get what he was gesturing at as the answer was impromptu.

dang2• 3 years ago

I'm not sure who you're slurring worse here but can you please make your substantive points without personal attack and generally not post in the flamewar style? We're trying to avoid that here.

We detached this subthread from https://news.ycombinator.com/item?id=34857541.

vouwfietsman• 3 years ago

This is cool, especially because its already 6 years old and I think not much has changed. Can anyone here speak to the current SOTA of explaining whats going on inside a neural net?

If we go: problem -> nnet solution -> explanation of nnet -> insight into problem that would still be very significant to the scientific process.

hackandthink6• 3 years ago

The debate is mostly about:

Are opaque probabilistic models scientific?

David Mumford's stance:

"This paper is a meant to be a polemic which argues for a very fundamental point: that stochastic models and statistical reasoning are more relevant i) to the world, ii) to science and many parts of mathematics and iii) particularly to understanding the computations in our own minds, than exact models and logical reasoning"

https://www.dam.brown.edu/people/mumford/beyond/papers/2000b...

foobarqux10• 3 years ago

The debate is mostly about: Are opaque probabilistic models scientific?

No the debate is about whether probabilistic models are scientific when applied to the human language faculty (they aren't).

Probabilistic models are scientific when they tell you something about the natural world. In some cases they do and in others they don't.

hackandthink6• 3 years ago

Yes, the scope of the Norvig - Chomsky ist the human language faculty.

But I think Mumford's more general discussion is relevant. And Mumford explicitly refers to speech:

"This approach denies that statistical inference can have anything to do with real thought ...

The new applications of Bayesian statistics to vision, speech, expert systems and neural nets have now started an explosive growth in these ideas."

jsenn7• 3 years ago

Interesting paper, thanks for linking. I think this is slightly tangential to the Norvig-Chomsky controversy though. What Mumford is saying is that probability and statistics is a more useful basis for modelling natural phenomena than classical logic. I don't think Chomsky would disagree with this! What he disagrees with is the idea (also raised by Mumford) that merely reproducing surface-level aspects of a given natural phenomenon (like human language) with no insight or understanding is sufficient for a scientific theory. It isn't sufficient: one also has to show that the theory cannot produce phenomena that are not natural, and give some insight into what's going on. In fact, it's not even necessary: Galileo advanced physics by imagining a frictionless plane. This doesn't and can't exist in the real world, but it helps us understand what does happen in the real world.

As an example, Mumford talks about how particle filters work much better than any "classical AI" technique for certain tracking tasks. This is true, and particle filters are still important in engineering for this reason, but it's not a theory of how humans accomplish this for the simple reason that particle filters can just as happily do other things that humans don't/can't.

dang2• 3 years ago

Related. Others?

Debunking Statistical AI – Noam Chomsky, Gary Marcus, Jeremy Kahn [video] - https://news.ycombinator.com/item?id=33857543 - Dec 2022 (19 comments)

Noam Chomsky: Where Artificial Intelligence Went Wrong (2012) - https://news.ycombinator.com/item?id=30937760 - April 2022 (2 comments)

On Chomsky and the Two Cultures of Statistical Learning (2011) - https://news.ycombinator.com/item?id=16489828 - March 2018 (12 comments)

On Chomsky and the Two Cultures of Statistical Learning (2011) - https://news.ycombinator.com/item?id=11951444 - June 2016 (102 comments)

Norvig vs. Chomsky and the Fight for the Future of AI - https://news.ycombinator.com/item?id=5318292 - March 2013 (2 comments)

Noam Chomsky on Where Artificial Intelligence Went Wrong - https://news.ycombinator.com/item?id=4729068 - Nov 2012 (177 comments)

Norvig vs. Chomsky and the Fight for the Future of AI - https://news.ycombinator.com/item?id=4290604 - July 2012 (147 comments)

Norvig vs. Chomsky and the Fight for the Future of AI - https://news.ycombinator.com/item?id=2710733 - June 2011 (4 comments)

On Chomsky and the Two Cultures of Statistical Learning - https://news.ycombinator.com/item?id=2591154 - May 2011 (107 comments)

ribit• 3 years ago

What’s really interesting is that entire Chomskian syntax worldview is fairly pseudo-scientific in nature. Most of these papers are about taking an essentially Turing-complete computation system and tweaking it until it can solve a specific riddle. Rinse and repeat. Most of the arguments (like the poverty of stimulus) are purely authoritarian as well.

qikInNdOutReply• 3 years ago

So, if the people rise up in rebellion against a believe system, like sovjet-socialism and the defenders of that believe system thus decide to decline those people - or even regions agency over there own destiny. Does that make those defenders of the believe system pseudo-scientific, because they fail to update there heuristic models?

The Norvig – Chomsky debate (2017)

Related Stories