Norvig vs. Chomsky and the Fight for the Future of AI

http://www.tor.com/blogs/2011/06/norvig-vs-chomsky-and-the-fight-for-the-future-of-ai

by fogus • 14 years ago

211 147 14 years ago

debacle5• 14 years ago

They have two different definitions of "artificial intelligence," which is where the schism seems to be arising from.

Chomsky takes the academic approach - artificial intelligence is the simulation of humanlike (or even possibly mammalian) intelligence.

Norvig is taking the engineering approach - artificial intelligence needs only to pass the Turing test.

They're both right, both approaches have value, and they both are bound by our limited technology at the moment.

In the end, though, Norvig will lose out. Sure, he'll make the finish line first - an AI capable of 'passing' the Turing test, but in order to have real intelligence you need an analytical engine (or brain, if you will) that can prioritize data without fiddling with bits. In the Norvig solution, someone will always have to be fiddling with the bits.

Chomsky's approach, on the other hand, will result in a 'true' artificial intelligence, the way neurologists understand it. It's just going to take a lot longer to get there.

phuff• 14 years ago

Having studied Chomsky a fair bit in grad school, and also studied cognitive linguistics a fair bit in grad school, I think the idea that Chomsky's models will ever win anything is just wrong.

Chomsky's central problem is that his modeling is not based on anything biological at all. His models don't correspond to reality. Some of them were based on some assumptions about how the brain works that were untestable in the 50s and 60s when all of his linguistic models were developed, and have since become testable and are not particularly evident in the way we currently understand the brain to work.

Given this fact, I think your current best bet is Norvig as a modern approach to AI or anything linguistic-y. But this is only because it is slightly more grounded in reality rather than being something that Chomsky (who is a very smart guy) came up with on his own without the benefit of actual biological models of the brain.

In the end, I think there will be (eventually, a long time from now) an actual model of how the brain processes language based on actual observations of working brains that throws away much of what Chomsky has proposed but probably uses some of it and that doesn't use huge Google-esque lookup tables but is highly influenced by statistics.

But until we get to that point, statistics are probably your best bet since at least they're grounded in reality (unlike much of Chomsky's work).

sp3324• 14 years ago

Children learn language much faster than it seems possible for a "blank" neural network to learn. It seems that there is some "circuitry" hard-wired into the human brain that helps learning language. So the question is: can a computer learn language as well as a human, without simply hard-coding language into it?

jan_g3• 14 years ago

But what exactly is true artificial intelligence? For example, I consider Google search and Wolfram alpha very intelligent. They can do math, answer questions, rank information, follow current events, ...

debacle5• 14 years ago

They're still computers in the traditional sense - they only do what someone told them to do.

waterlesscloud• 14 years ago

I think statistical models are an 80% solution. They get a long way down the path very quickly, but then they hit a wall and don't advance much further. Search, translation, probably also autonomous vehicles. They get to a point with statistical approaches and then they stop advancing.

The rapid success at first may be leading to a dead end.

Having said that, there's a lot of value in these technologies as assistance to human intelligence, but I'm skeptical they're ever going to lead to full-on autonomous intelligence.

snowwrestler• 14 years ago

I think "artificial intelligence" is a total misnomer for we're talking about here.

There is a lot more involved than just language. It seems possible to solve language and still not create "strong AI." See: Watson kicking ass at Jeopardy. That is pretty sophisticated language comprehension; it even got clues that involved puns and wordplay.

From that it would seem totally possible to adapt this sort of intelligence to taking an IQ test. Just provide the right corpus to search and give the algorithms time to learn and be tuned. What if Watson tests out at a 150 IQ? Is he "intelligent"? After all it is an "intelligence quotient."

I think most people would say no, the issue is the Turing Test. That involves language, but I think the real point is an artificial personality: a computer that can hold a conversation and evince a discernible will and opinion. It seems to me that to do that, the machine must have emotions. I'm no expert in the field, but from my layman's position I don't see nearly as many stories about studying and replicating emotions as I do about human language.

Which is funny because any dog owner knows that dogs, despite having almost no formal language, are clearly willful, independent, emotional living beings. Can we even simulate a dog's emotional state with a computer? How about a bug's emotional state? I haven't heard much about that. But there is a ton of heat around language.

olalonde• 14 years ago

Isn't there a consensus that "passing the Turing test" <=> "true artificial intelligence"? Otherwise, what test do neurologists propose?

power• 14 years ago

“Norvig is taking the engineering approach - artificial intelligence needs only to pass the Turing test.”

Passing the Turing test and the simulation of intelligence are supposed to be the same thing. Turing came up with the test to sidestep the definition of intelligence.

I’m not sure what you mean by “but in order to have real intelligence you need an analytical engine (or brain, if you will) that can prioritize data without fiddling with bits. In the Norvig solution, someone will always have to be fiddling with the bits”. Do you mean bits as in pieces or as in ones and zeros ? If the former, haven’t Chomsky’s models required the addition of new parameters as exceptions to his rules are found ? If you mean the latter, how is a computer to prioritize data if it can’t look at bits ?

There seems to be the misconception that Norvig does not use simple models. He does, just ones that use statistics for training and to learn. His approach strikes me as elegant, simple and more robust to changes in language over time than Chomsky’s.

netcan• 14 years ago

I'm going out on a limb because I don't really know much about neurology & I may be wrong about facts but.. I think an issue here is that we don't really know what "natural intelligence" is.

For a significant part of the Scientific age we knew about genes in some sense without knowing much about them. We called them traits, observed & measured them. We got to know some "rules" about their inheritance. But it wasn't until genetics got to be a little better understood that we got to know their physical manifestation. We can explain the diference between genetic & cultural (memetic?) inheritance in these terms. A descendant's cooking habits are memetic and her hair colour is genetic.

When it comes to neuroscience I think we're where we were a century ago in biology. Emotions, thoughts, memories. We don't know what their physical manifestation is. We dont know how they work. Since we don't know much about how natural intelligence works I think our common sense definition of intelligence is, to a certain extent: "stuff we can do that computers can't."

I think that if we had a definition that was more functional than observational, you wouldn't be hesitant at all to use "mammalian" in your definition. Whatever processes result in observed human intelligence are almost certainly shared with other species. We'd probably also know what species to draw the lines at: reptiles? invertebrates? fungi?

If apes have intelligence, goldfish don't but octopi do that suggests there multiple versions of natural intelligence.

_delirium4• 14 years ago

A significant proportion of the "academic approach" is actually a third one: artificial intelligence is the analysis and implementation of rational decision-making. That approach tends to care neither about biological accuracy, nor believability in a Turing-test sense. Rather, it cares about whether its decisions are correct based on evidence available to the decision-maker. That's the kind of attitude you most often find in both statistical and logic-based AI circles.

Actually, Russell & Norvig's AI textbook has a nice summary of these different approaches to AI in its intro chapter.

JabavuAdams6• 14 years ago

Too bad humans aren't rational.

bugsbunnyak• 14 years ago

> Chomsky's approach, on the other hand, will result in a 'true' artificial intelligence, the way neurologists understand it. It's just going to take a lot longer to get there.

High-level behavioral impressions taken by a neurologist are a convenient abstraction. That this high-level behavior is useful in monitoring mental state (outputs) says very little about the underlying 'hardware'. In fact, this is the `fundamental` debate in cognitive science: from whence does intelligence arise? Theories generally fall under two headings: 'top-down' and 'bottom-up', which roughly correspond to 'pre-programmed' and 'emergent'. The canonical bottom-up approach is the neural network, approximating cells with various equations that govern behavior (outputs) based on aggregate input (there are various levels at which this can be done). There are a variety of top-down approaches, a typical approach would take the form of logic engines (think Prolog), or generative rules (Chomsky)

Statistical modelling approaches are closer to bottom-up, but depending on the model they may still incorporate domain knowledge that is emergent from the model input.

Statistical approaches have momentum these days due to considerable success - thanks largely to Moore's law. However, they also have biological support: what is a neuron? It's an FPGA with a lot of electrical and chemical inputs. Small neural circuits can behave statistically, and it's an open question whether this gives rise to high-level behavior. A big reason it's an open question is that we don't yet have the spatial or temporal resolution to measure enough signals.

That said, there is plenty of room for what I consider a happy medium: locally statistical behavior, but globally (and generationally) top-down organization driven by genetics.

btilly• 14 years ago

I personally believe that it will be possible to simulate a working brain in sufficient detail that it is able to "think" like a human does before humans understand said brain.

In fact I doubt that the cognitive capacity of a human brain is enough to truly understand an operating human brain.

rm9992• 14 years ago

This is exactly my take on it. They are talking about AI in different contexts, and therefore aren't really arguing with each other as much as past each other. Anyone who has any interesting in building something today would take a Norvig approach, and anyone who pictures AI 100 years from now should hope that the Chomsky approach eventually won out.

SudarshanP• 14 years ago

Many confuse the Chomsky approach v/s Norvig approach as who is right and who is wrong. But that is an unnecessary distraction. Consider planes. The wright brothers studied planes without mastering Fluid dynamics. Their planes flew. That was awesome. But plane makers kept exploring and inventing. We achieved super sonic speeds and even made rockets. As time passed, we learnt more and more about fluid flows. While on the one hand atoms jiggling around with various velocities is all that we are studying. But the fluids demonstrate very rich and complex behaviours. Only a fool would stay contented that the aeroplane flew. Our understanding of fluid dynamics comes in handy to model wind turbines and weather patterns and a whole host of awesome areas. A lot of those great discoveries came from investments in aeronautical research made by profit hungry entrepreneurs. The science v/s technology debate is usually just funding politics. It is silly to argue whether the eye runs the marathon or the leg runs the marathon. Let us just accept that "the runner runs the marathon" and just move on.

Chomsky may or may not be right. Just like aristotle was not right about many things. He was a genius and he contributed as well. He may have stirred up some trouble also. Over time if we are fortunate enough we may get a clear understanding of the core ideas like we understand the behavior of a fluid. If we have to settle with flying planes but don't understand wind, we cannot go supersonic or reach the moon.

PaulHoule• 14 years ago

Well, in the big picture, Chomsky created an activity which keeps liguists very busy. His approach, however, has contributed very little to language engineering.

_delirium4• 14 years ago

Do you mean specifically of human languages? Because Chomsky's approach has contributed pretty extensively to programming language engineering, as the foundation of parsing theory and the whole formal language hierarchy (context-free, context-sensitive, regular, etc.).

I do agree it's been less successful for its original intended purpose, but things often find new life, which seems okay.

slurgfest8• 14 years ago

Yes: the Chomsky hierarchy is a fundamental of computer science, one of the great intellectual achievements of humanity. And in that respect also important to AI.

Chomsky is incredibly strong on anything that does not require empirical data.

But UG has no legs and Chomsky's analysis of syntax has very limited applicability and after many many IOUs, pretty much no empirical claims have panned out in any significant way.

If you take away the application of those basic computer science concepts to language, you unfortunately take away most of what Chomsky has written regarding linguistics, psychology and AI. Because of the sheer volume of output, that leaves a number of contributions. My point is that it is necessary to be discriminating rather than making Chomsky into the Pope, as certain fields have done for some time.

mootothemax2• 14 years ago

Isn't this basically an argument over John Searle's Chinese Room thought experiment?

It supposes that there is a program that gives a computer the ability to carry on an intelligent conversation in written Chinese. If the program is given to someone who speaks only English to execute the instructions of the program by hand, then in theory, the English speaker would also be able to carry on a conversation in written Chinese. However, the English speaker would not be able to understand the conversation. Similarly, Searle concludes, a computer executing the program would not understand the conversation either.

http://en.wikipedia.org/wiki/Chinese_room

sp3324• 14 years ago

The argument is whether a computer can learn language (well) from scratch, or whether some capacity for language must be built into the computer manually. https://en.wikipedia.org/wiki/Poverty_of_the_stimulus

mootothemax2• 14 years ago

Ooh, I see, interesting, thanks! :)

debacle5• 14 years ago

The problem with Searle's assertion is that he is making a distinction between the computer and the program. We, as human beings, are not our computers, we are our programs.

bluekeybox3• 14 years ago

Absolutely. Also, according to Searle's argument, submarines cannot swim.

knowtheory7• 14 years ago

Just as a note, the space of possible responses to Searle's argument have been pretty well enumerated here: http://plato.stanford.edu/entries/chinese-room/

I'm of the opinion that the room has an understanding entity inside of it, in talking about infinitely-sized books with an infinitely-sized index allowing any mechanical process to map an input to a correct output, you've hypothesized something complicated enough that it should be said to be an entity capable of understanding/meaning.

zeteo• 14 years ago

Is a robot capable of running? Let's say you had one, then take his legs and give them to an amputee. The amputee can operate the legs by pressing a button. However, pressing a button is not running. Similarly, the robot would not really be running either.

pessimizer• 14 years ago

>However, pressing a button is not running.

No True Scotsman here, which I'm assuming that you're implying exists in Searle's argument. But that puts you in a position of saying that the English speaking operator of the Chinese room understands Chinese because he responds in a convincing way to Chinese speakers, which is pretty explicitly false.

It also relies on the concept of "running" being monolithic instead of constructed. Under a part of most people's concepts of running lie things that a running robot does, other parts of the concept for some people involve particular types of clothing, and/or nipple chafe cream. I would think that most people would think of what the amputee does with the robot legs as running, yet they wouldn't want him in the Olympics.

Is Searle's room, or the assemblage of Searle's room and its operator, intelligent? Whether you can answer that question easily depends on which part of the concept of intelligence is important to you in that context. Is the designer and implementor of Searle's room intelligent? Unambiguously. Are the assemblage and the designer intelligent in exactly the same way?

nessus42• 14 years ago

> Isn't this basically an argument over John Searle's Chinese Room thought experiment?

No, this debate is completely and utterly different from Searle's Chinese Room argument! Searle's argument is a philosophical one for the assertion that a computer could never be a person. He concludes that this is true even if we were to eventually believe that we completely understand intelligence in the manner than Chomsky is lobbying for, and then fully implement that understanding in a computer.

For Searle, no amount of understanding of intelligence in any form will ever let us make an intelligent computer. For Searle, intelligent beings must be made out of flesh and bone. Or at least not out of anything digital and computer-like.

orbitingpluto2• 14 years ago

I know a card counter. I showed him how to condition probabilities to determine how to best play. He went for the full Monte Carlo method and he lets his simulation run for a week before he starts using it "just to make sure". It's frustrating because he doesn't get that his results are statistically significant after about 30 seconds of runtime. He still makes money doing it. The results are tangible, but he's still just mucking about.

'Quantum mechanics is certainly imposing. But an inner voice tells me that it is not yet the real thing. The theory says a lot, but does not really bring us any closer to the secret of the "old one." I, at any rate, am convinced that He does not throw dice.' --Einstein

Statistical methods can work but they are unsatisfying to the scientifically curious. You're not really a scientist if you create something that works and you don't really know why. (Not to say that the method doesn't have value. Sometimes you have to play with your Lego before you grow up.)

bluekeybox3• 14 years ago

> You're not really a scientist if you create something that works and you don't really know why.

According to your logic, the only true "science" is mathematics. If you test the workings of your "creation" using scientific method, you're still a scientist. Scientific method is also about testing your claims empirically, and it has been successfully applied for more than a century to study of biological organisms, climate, and other complex systems that we do not "really" understand. Not to berate understanding of underlying mechanism which is always preferable, just to point out that there is more than one way to skin the cat.

orbitingpluto2• 14 years ago

I notch you up a point good sir for discovering my bias. (mathematics)

robg• 14 years ago

This is one of those rare moments in intellectual life where being in the room and now seeing the debate develop, it becomes clear that the resulting hype isn't (wasn't) loud enough.

This distinction marks the real turning point in AI from abstract, grand claims with highly restrictive evidence toward engineering that simply works. Who cares about the ontology when we can recreate? It's like saying airplanes don't properly explain flight because they don't replicate how birds do it. Who cares? We can fly (and translate and soon reason) artificially.

It's clear that Chomsky and Universal Syntax has held back the entire field of AI (and at MIT). There isn't one algorithm in the human mind to decode all of our mental capabilities. That's mistaking subjectivity for objective lessons. Trying to recreate that Phantom has led to rule tables in AI, constraints on how the mind must operate. Instead, by allowing those fuzzy boundaries to accumulate with evidence, statistical approaches win in the long-term of our lives and in this debate.

Kuhn knew what happens to dinosaurs.

_delirium4• 14 years ago

I don't think I would take that strident battle-against-dinosaurs view, in part because I think intellectually understanding things is useful in and of itself, not some kind of "just build it and shut up" anti-intellectual view; and also because I don't think it's an accurate summary of the history of AI. There's no particular reason we can't both build and study things in various ways, and the history of AI has been full of people doing many takes on both.

In particular, statistical approaches have been used for a long time, but were not practical until fairly recently; it was the lack of "big data" computing power holding them back more than anything. Statistical machine translation and parsing experiments have been tried on and off for decades, but with 1950s-era data they produced total garbage as output, even worse than the (also bad) symbolic approaches. Hence why Shannon's work on text processing didn't produce practical NLP or NLG systems. It took Google-sized data to produce statistical translation that was actually usable.

What numerical approaches were possible on computers of the time were fairly extensively investigated when they became possible (e.g. the 1980s focus on "sub-symbolic AI", with perceptrons, neural networks, numerical regression methods, etc.). Some were shelved for years because they just didn't work as well, e.g. symbolic game-tree search massively outperformed machine learning in board games in the early experiments, which is why Samuels's 1950s ML-based checkers player was theoretically intriguing but not considered very practical.

jerf• 14 years ago

I often feel bad for computer science researchers in the 50s and 60s. It isn't that they were any less smart than today, it's just that where they had kilo-X, we have giga-X, if not tera-X or even peta-X.

One of my evolutionary computation professors said that he had a deck of cards that he carried from job to job, university to university, running bits of his evolutionary computation experiment on spare cycles, an experiment that runs in fractions of a second on hardware at the time he told the story in 2001. Presumably now it would run in even more fractional fractions of a second.

slurgfest8• 14 years ago

Intellectually understanding things is useful. There's no anti-intellectualism here.

It's become increasingly clear that Chomsky's approach to language is not going to generalize to AI successes or explanations of many domains of human performance, as was promised. That approach has not yielded fruitfully anywhere except the narrow realm of syntax (in particular typologies of syntax). While machine learning has kicked butt left and right.

There is no reason people can't go on studying syntax at the same time machine learning expands. But it would be dishonest to ignore the extreme and exclusive claims laid down by Chomsky's school at the outset of the cognitive revolution. These claims were given a very liberal benefit of the doubt for decades. A lot of good work has been done in syntax. But now those claims about AI and psychology-in-general are clearly threadbare, since they have not yielded the promised fruits either in AI or in explaining human functions. They have not even yielded plausible IOUs. Remember, this was supposed to explain pretty much everything. Not even language acquisition has been explained.

That is empirical inadequacy. Who cares if Chomsky thinks it's beautiful?

And it's equally clear that learning of various kinds has to occur and also gives us the more parsimonious and elegant solutions to problems (no-solution and no-explanation is not elegant even if it is simpler).

Machine learning has a great deal of conceptual beauty - if you study it rather than pooh-poohing it because of some facile abductive argument to UG.

Computing power isn't a problem. The scale of the human brain is enormous and we are still nowhere near reaching it with our computational resources.

pnathan• 14 years ago

"Who cares about the ontology when we can recreate? It's like saying airplanes don't properly explain flight because they don't replicate how birds do it. Who cares?"

Well, I do. Being able to understand something from first principle is very different than being able to model it and gain an approximation of it.

One might point out that this debate crops up in a different form in developer circles, but reframed as "Learn assembly & cs theory vs. use IDE & ignore assembly and theory".

fusiongyro• 14 years ago

> It's clear that Chomsky and Universal Syntax has held back the entire field of AI

That may be true, but there is value to understanding language for the sake of language outside the practical goals of improving AI. There is plenty of evidence for parameters playing a role in human language whether or not a parametric implementation of NLP is possible, desirable or necessary. It's certainly true that a statistical approach that bears plenty of fruit in AI applications is going to be very strained to provide anything of value back to linguistics or developmental psychology.

JabavuAdams6• 14 years ago

> There isn't one algorithm in the human mind to decode all of our mental capabilities.

Careful here. There could be one algorithm, but we may not be able to express its parameters in the way we like to for simple models.

These statistical approaches are algorithms. It's the fact that we can't make sense of the parameters that seems to lead people to believe that they don't explain anything.

The fact that we can't categorize all life in an unambiguous way that makes sense to humans doesn't mean that the elegant and simple algorithm of evolution by natural selection is wrong.

Model vs. initial conditions. Knowledge vs. search.

slurgfest8• 14 years ago

You are right, these statistical approaches are algorithms. And they themselves are very simple and do not necessarily require a lot of initial state. And you are also right that it is extra hard to reverse-engineer these.

But they amount to learning and search. And because of the politics of Chomsky's movement, it has always fought the concept of learning tooth and nail in favor of innate knowledge or ready-made models. This is why we talk about language "acquisition" and why we read Chomsky saying that contemporaries who studied learning were facilitating totalitarianism.

It's also worth mentioning that Chomsky has stated that he does not think the brain has evolved.

So however many algorithms there are (and however they are employed to get the vast array of intelligent human actions) many things you have said here are totally anathema to Chomsky's school.

It was pushed too hard, it overextended and only now has any challenge come to it - from the quarter of AI. (Up until a few years ago, fighting it in academic Psychology meant you were OK with killing your career, and anyone less respectable than Norvig would get torn apart - look at what they are saying about Norvig now).

meric• 14 years ago

"It's like saying airplanes don't properly explain flight because they don't replicate how birds do it."

The analogy doesn't hold. We don't want planes to fly like birds, but we may want machines to thinking like humans.

eyko• 14 years ago

Also, the statistical approach would be to imitate flying creatures until we can fly, rather than understanding aerodynamics in order to create "vessels that float".

knowtheory7• 14 years ago

It's a little bit frustrating to read a rehash of an argument that was cutting edge maybe back in the late 90s, especially one that is so poorly written, and framed as a battle between two intellectuals.

Chomsky's past his heyday. He has been seminal in his field, but he's no longer doing research which pushes at the boundaries of our understanding of language, how to model it, or what the fundamental nature of language understanding systems is. (as one might infer, I come from a non-chomskyian school of linguistics).

Given that we have actual data and research about large scale systems that do interesting things (including the massive artificial neural network that google built last month, see: http://www.wired.com/wiredscience/2012/06/google-x-neural-ne... ) reporting as substance free and obfuscating as this is, is a real frustration, when we could be talking about more interesting things, such as what a solid operational definition of meaning is, or how exactly heuristic/rule based systems actually differ from statistical mechanism, and whether or not all heuristic systems can (or should) be modeled with statistical systems.

The framing of this article is particularly galling because there are so many non-chomskian linguists out in the world who operate fruitfully in the statistical domain. Propping Chomsky up as somehow representative of all linguists is pretty specious and a bit irritating.

gbog2• 14 years ago

> we could be talking about more interesting things

Please do.

mcguire7• 14 years ago

Non-Chomskian linguists? I was under the impression that, post-Chomsky, linguistics was defined as Chomskian; everyone else had left the field for whatever related discipline most closely matched what they wanted to do.

knowtheory7• 14 years ago

I mean... there are core things at the basis of what everyone agrees on in linguistics (language is structured, learned, and there is a defined syllabary of sounds that humans make and use in language, etc), but Chomsky's grand unified theory of language, and his ideas about how those mechanics function are not universally agreed upon.

Even within syntax, which is really the sort of core of what Chomsky has been interested in, there are other formalisms to represent syntax which differ from Chomsky's theoretical framework. You can read more about Chomskian transformational grammar on wikipedia with links off to other sorts of formalisms as well: http://en.wikipedia.org/wiki/Transformational_grammar

saurik• 14 years ago

West Coast Functionalism gave birth to various continuations that are still practiced fervently at various universities (including UCSB, as a specific example I can point to that I have personal experience with).

jholman• 14 years ago

Elaborating on what knowtheory posted, in modern linguistics, "Chomskian" doesn't usually refer to "everything Chomsky said". For example, Chomsky was probably the foremost advocate of the overthrow of behaviourist theory in linguistics, and now no-one takes extreme behaviourism a la BF Skinner serious (as an explanation of linguistic behaviour), but this rejection is not currently labelled as "Chomskian". It's just linguistics.

So, not that I'm 100% clear on the details, but Chomskian refers to more specific claims. For example, some claims about what aspects of language knowledge are innate (Chomsky claims a great language engine with a relatively small number of tuning parameters). As another example, at least at one point, I think Chomsky was rejecting syntax rulesets with productions that were not binary? I think?

gbog2• 14 years ago

> linguistics was defined as Chomskian

IOPOTW (In Other Parts Of The World) Chomsky may have a much lower relative influence on linguistics. For instance in France, you have Ferdinand de Saussure and André Martinet. I read a bit of the later, and it seem to me more solid on some grounds than Chomsky, because more humble (the first act of reason is to acknowledge its limits).

I'd bet in continental Europe the political Chomsky is deemed more interesting than the linguist.

morsch2• 14 years ago

Not sure how you got that impression, but a lot of stuff has happened in linguistics since Chomsky, some of it quite explicitly contrary to his arguments. None of my advanced linguistics classes dealt with Chomskian stuff.

ntoshev• 14 years ago

It's a rehash of http://norvig.com/chomsky.html and the talk by Chomsky that led to it (both recent and relevant). I can't wait to read the continuation of that article that Norvig promised in comments.

mcguire7• 14 years ago

You know, the discussion of Chomsky made me realize something. I would bet that Chomsky's take on Norvig's approach (to put specific names on the ideas) is that Norvig is a resurrection of Skinner. Specifically, behaviorism as "study texts, make statistical correlations, but don't make up any grand theories."

brudgers2• 14 years ago

Intellectually, there seems to be something as wrong with avoiding anthropomorphism when discussing human endeavors (such as language) as there is with anthropomorphic explanations of erosion or chemical reactions. Skinnarian approaches to language may leave people unsatisfied because there is no story, just clinical observation.

Norvig's approach (as characterized in the article) takes the the "Artificial" in "Artificial Intelligence" to include the mechanism by which an intelligence makes decisions. Chompsky's aesthetic of linguistics applied to AI would treat "Artificial" as a description of the platform in which an intelligence is embodied (i.e. non-biological) while requiring the platform to operate linguistically on the same principles as a "natural intelligence."

Norvig's approach (as characterized in the article) is essentially a better Eliza (or Ford's faster horse).

If one takes the Turing Test as scientifically meaningful rather than an engineering standard, then one falls in one camp or the other and the Norvig Chompsky debate is over a pseudo-problem. "Artificial Intelligence" is in that sense metaphysical jargon.

slurgfest8• 14 years ago

Skinner's book Verbal Behavior was mostly unsatisfying because it didn't have a lot of data; it really just laid out a research program which had not been carried out in any significant way (and now, never will be). Of course it is also unsatisfying that Skinner does not appeal to our sense that we already understand everything important about psychology and language "from the inside" and don't really need any stinking data.

The reason most people are unsatisfied with Skinner's approach to language is that they did not read Verbal Behavior, but rather Chomsky's review; and because Chomsky chose it (as among Skinner's weakest work) and reviewed it in the most uncharitable way possible, without understanding any of the basic concepts or motivations to Skinner's approach.

So, for example, he successfully associates Skinner directly with Watson, and makes it out that "radical" behaviorism is radical not for its rejection of premises of classical behaviorism but for being even more crazy.

That review is a masterpiece of propaganda and it effectively prevented Skinner's basic ideas from even being seriously evaluated ever again.

brudgers2• 14 years ago

Just to clarify, my reference to Skinner was to be behaviorism in general or even more generally, radical empiricism in regards to human activities.

phaedrus• 14 years ago

I spent about ten years working on Markov based chat programs. I gave up on themwhen I realized that no matter how sophisticated your statistical model it will never be more than a statistical analysis of text, unless it includes some rich rule based model of mental processes and mental objects. It may be that such a model of mental processes must itself be fuzzy and probabilistic, but it must exist. Therefore I come down firmly on the side of Chomsky in this debate: we should pursue theories of intelligence, and stastical models without any theory do not advance our scientific understanding of AI, however practical their application may be at the present time. This is not to say statistical methods do not work, of course they work, what I am saying is it is not a path that leads to true understanding of intelligence any more than spectral analysis of the EMF emissions of a running computer would lead to a theory of computation.

bhickey• 14 years ago

The fact that Markov chains are by definition memoryless isn't an argument in favor Chomsky or magical thinking. Sure, if you want to improve your output you can use (n+1)grams instead of n-grams, but the curse of dimensionality is going to quickly catch up with you. Language smoothing will help for a little while. Over a long enough horizon all Markov chain output is jibberish. None of these obvious limitations are an argument against statistical models.

Where is the data that statistical methods don't 'advance our understanding'? What does an EEG tell us about the brain works?

slurgfest8• 14 years ago

Running a Markov chain model as-is to generate text produces gibberish. You correctly point out that the gibberish can be much higher quality.

Fundamentally it is gibberish not for any simple algorithmic reason, but because generation is occurring without any respect to context or meaning beyond what randomly emerges from the graceful juxtaposition of randomly chosen words. It is purely about the combinations of words (in that sense, syntax). This shouldn't be surprising - who ever actually expected that generating a kind of syntax model would result in coherent thoughts? At most it can generate texts like weird dreams, it shouldn't be surprising that the result is not a cogent discussion of current events.

This does not mean that the same information cannot be used in more sophisticated ways. But these wouldn't be a Markov chatbot. The Markov model would effectively be a component in a larger system that needed to use words. It isn't at all clear that the Markov model is the best possible one, but it is just groundless dogma to insist that learning can't have anything to do with real performance.

knowtheory7• 14 years ago

Just because you chose Markov chains as your modeling mechanism doesn't mean that there is no statistical modeling method that is capable of developing something passing for what we'd call "meaning".

This is the same argument that was used against artificial neural networks. Neural network of type A can't do X, therefore neural networks will never do Y.

Language is immensely complex, and real human language involves things which are not encoded in text (and i'd remind you that you were trying to infer meaning from text specifically, not the full multi-channel robustness of humans communicating), we don't even have a full handle on what all of the cognitive processes and factors are that go into the production and understanding of language (although we've developed a lot of interesting work to those ends).

So hearing folks give up claim that Chomsky is correct because our current tools aren't up to the job is a bit puzzling, because we don't even have a complete understanding of what sort of thing language is or what sorts of things we are as systems which can use language.

Chomsky has opinions (and some facts) about what language is, and we are, but he does not have solid proof to confirm his specific conjectures. Is human language context free? context sensitive? Something else? (Chomsky's minimalist program uses movement along a tree to preserve referentiality and a bunch of junk, alternative syntactic frameworks such as HPSG uses directed graphs as the basis of their language modeling. Still others do weirder things like higher order combinatoric logics. And unfortunately none of the theoretical frameworks appear to be without their drawbacks)

neilk• 14 years ago

I am not a specialist, but as far as I know, Chomsky's argument here was that the existence of recursion showed that a Markov approach had to be wrong. Surely a similar argument can be made for statistical approaches? There is no way to represent a reference to some other part of the statement in a purely statistical method. If they work they happen to work basically by accident.

Just blue-skying here, but it seems to me that if I knew enough about how a statistical program worked, I could craft a sentence that would utterly confuse it, even though it was perfectly intelligible to a normal English speaker. A putative strong-AI program could not be fooled in this way.

JabavuAdams6• 14 years ago

Two experiences have made it clear to me that humans don't understand language that well, without context:

(1) Raising a child. My dad often remarks that he's surprised that my daughter knows how to use a word in just the right context. I'm not, because this is a natural product of mimicry: if you copy what others say, you usually use the words in the correct context. As with computer-generated text the exceptions are often hilarious.

(2) Song lyrics. I had a very clear experience where I just could not understand the refrain of Gold Guns Girls. It sounded completely unintelligible until I read the lyrics. After that, it sounded crystal clear. Why would reading the lyrics make the song sound different? Context.

bwanab• 14 years ago

It strikes me that the theories are not mutually incompatible at all. They have very different purposes. Chomsky is trying to understand meaning and intelligence at a deep level. Norvig is trying to build models that help people right now (and incidently, help his company to make more money). Any new insights from either path will help refine the other.

So it sounds like what you decided was that you wanted to explore one path and not the other. Nothing wrong with that, but it's a very different statement.

dave_sullivan• 14 years ago

As far as language modeling, this is a recent paper that models language on the character level rather than word level and can track long term dependencies and even generate plausible sounding non-words from time to time: http://www.cs.toronto.edu/~ilya/pubs/2011/LANG-RNN.pdf

The state of the art is improving a bit, although this method still knows nothing of meaning so it can often generate some strange sentences. Still, I wouldn't write off the whole field yet--just because something didnt work with tools years ago doesn't mean it isn't possible.

slurgfest8• 14 years ago

There is no valid argument leading FROM your disenchantment with Markov based chat programs, TO a conclusion that machine learning is invalid. Markov chatters are a toy.

Did you produce a better chat bot based on UG? - No? Then on what basis are you junking machine learning?

Machine translation is far from a solved problem. But Chomsky's school claimed they were going to solve it in the 60s or perhaps the 70s. Do you know what is the basis for the most successful current approach to machine translation?

Analysis of text. (But not the kind of simplistic junk one does in a Markov chatter)

All you have done is suggest that some beautiful perfect text model exists natively in every person. (Presumably this evolved somehow - or if you are Chomsky, it just developed like a crystal for no apparent reason). This isn't an explanation of anything unless you actually find that model instantiated in the brain. But this is just not happening. So either our instruments are still too crude to detect it, or it's not really there.

Appealing to an as-yet unknown perfect universal text model does not build a better chat bot or a better explanation of human behavior.

True understanding of intelligence must incorporate an understanding of how learning occurs. Because anyone who watches children sees learning occurring, and only doctrinaire Chomskyists deny that it occurs (because it is not beautiful enough and some abductive argument is claimed to show that it is not sufficient).

guscost2• 14 years ago

There are two possible ways to invent this kind of theory, the way I see it:

1.) Measure data, then make an educated guess. 2.) Make an uneducated guess.

Jun86• 14 years ago

OK, let me start with two facts, one objective, one personal: (i) Noam Chomsky is a genius with many contributions to linguistics and computer science (ii) I think his overall influence had been damaging to linguistics.

Here's a summary of Chomsky's career in layman's terms: As everyone knows, Chomsky first came to prominence with his critique of Skinner (who, as everyone also knows, was a total psycho). He pretty much created linguistics as we know it (at least in the US, there were some numbskulls in Europe who still doubted the new order), starting from the main thesis of linguistic universals, which can be summarized as the fact that all humans possess the same language faculty, i.e. the wide range of linguistics differences between, say, English and Mandarin are just on the surface. This was a welcome relief against the Sapir-Whorf mumbo-jumbo which held that Eskimos had hundreds of words of snow and language constrained how we think. Chomsky has also been very active in politics (he's actually much better known to the general world by his political books), pointing out the evils especially of the American brand of capitalism (is there any other kind?) and its corrosive influence on the world, e.g. Iraq, Afghanistan, etc. He also points out errors in certain approaches in Economics, e.g. see http://en.wikiquote.org/wiki/Noam_Chomsky#Capitalism, without holding a degree in the field, but everybody does that.

Chomsky's greatly damaging influence to linguistics is due to the fact that his speculative and simplistic (at least originally) views on how the brain processes and learns language has stifled research in promising fields by decades. The main problem I have with him is that the cause of the shortcomings of his theory seems to be not lack of knowledge (very little was known about cognition in the 60s), which, of course handicaps all pioneers of science, but politics (I detest politically motivated scientific theories). AFAIK, his universalist views were motivated from his political beliefs.

Luckily, starting in the 90s, Chomsky's chokehold on linguistics has slipped somehow. Researchers, such as Leda Cosmides, have ventured into research on linguistic relativity (http://en.wikipedia.org/wiki/Linguistic_relativity). Skinner's theories are making a comeback in academic circles (http://www.theatlantic.com/magazine/archive/2012/06/the-perf...).

So, what does all this mean for the current debate? I think it's time to retire and the "old guard"! Let us acknowledge their breakthroughs, their contributions, but also their limitations and move on.

why-el• 14 years ago

You cant label something as a fact then proceed to say "I think". (ii) is poorly defended. You set out to outline Chomsky's long and engaging political life just so that you can say that it influenced his linguistic theory. Chomsky himself have said many times that there is no clear correlation between his linguistic work and political activism. This is not new. He has been asked this in a lot of talks.

with this said, there were attempts at bridging the creativity exhibited by linguistic behavior and that of political freedom, such as in his "Language and Freedom" book, but this happen way after he established his theory.

Jun86• 14 years ago

Labeling (ii) as a fact was a bit of joke, but not overtly marked, which may lead to misunderstanding, so you are correct.

My gripe is not against Chomsky, the person, but against Chomsky-ism (although he does have a very aggressive, know-it-all tone when discussing many subjects, that I find at odds with scientific humility). This is common: A pioneer comes up with a new way of thinking, then it is accepted as dogma by the community. Although one may think otherwise, this is also very common in science. Scientific revolutions usually don't happen with paradigm shifts but through the demise of old age of the new theory's opponents.

What's shocking is that a field like linguistics, which some may argue is more scientifically rigorous than its sister humanities disciplines, operated on what, to me, are speculative assumptions for close to 40 years.

icandoitbetter• 14 years ago

> I detest politically motivated scientific theories

Then you detest every scientific theory ever devised.

JabavuAdams6• 14 years ago

This is a bizarre claim. My motivations for doing science are to understand, to create, and to stroke my own ego.

Jun86• 14 years ago

If you comment was not in jest, can you elaborate on why you think so? One can argue that all scientific theories are motivated to a larger or lesser extent by the prevalent world view of their day. But arguing, e.g. the Standard Model in physics is influenced by the politics of its day (~70s) is indeed bizarre.

slurgfest8• 14 years ago

I am interested in how you come to the conclusion that Skinner was a total psycho. You actually seem to suggest that Skinner was an evil, mentally ill person and I would be interested to know what evidence there ever was for that.

Jun86• 14 years ago

Sorry for not being clear: The second paragraph was meant as a somewhat humorous summary what an informed non-expert would think about Chomsky and his work.

But even in academic circles his name had a bad vibe, like Anakin, who started out with so much promise but then went to the dark side. This is partly due to the huge backlash in the 70s (see page 2 of the Atlantic article).

zzzeek3• 14 years ago

I'm completely ignorant here - is it widely established that there's no link between how we think about things versus the languages we speak ? It seems intuitive that our ability to conceive of concepts would be dependent on having/creating language that can describe them, even in our own minds, but I have no idea how anyone could really know one way or the other.

Jun86• 14 years ago

As you point out, this is a hard thing to test. Add to this the fact that the question may be a sensitive one, similar to differences between men and women.

How can one go about testing the effect of language on thinking? Consider this example: English has an explicit grammatical structure for counterfactuals (CFs), e.g. "If I were a rich man ..." whereas some languages, e.g. Chinese (Mandarin) do not (they do have some other means but not as overt). One can then think presenting stories containing complex CF situations to native English and Chinese speakers, and somehow test how quickly they grasp them. This exact experiment was performed by Alfred Bloom in 1981 and indeed showed some differences. Later researchers noted some points that might have affected the results. You can see why this research may be sensitive, it may be mistakenly used to argue that Chinese speakers are somehow linguistically deficient.

morsch2• 14 years ago

I think it's widely accepted in linguistics (IA kind of AL) that language influences conceptualisation (sometimes called the weak Sapir-Whorf hypothesis) but that it does not determine conceptualisation (the strong version).

See http://en.wikipedia.org/wiki/Linguistic_relativity

rdtsc• 14 years ago

> Chomsky's greatly damaging influence to linguistics is due to the fact that his speculative and simplistic ... views on how the brain processes and learns language has stifled research in promising fields by decades.

By decades. Really? I think those claims would need better substantiation. Looking back we have a 20/20 hindsight and maybe can say where we took the wrong turn. But can you point out to other budding and promising theories that Chomsky somehow squashed because of, I don't know, his great power or character.

As for politics. Can you explain better and provide some more information. So far it seems, this is just you disagreeing with his politics and then attacking his influence on linguistics as an obvious extension of his political views.

> I think it's time to retire and the "old guard"!

No. You retire, theories that have been proven wrong and bring about theories that explain the reality in a better, more complete, or universal way. You don't retire the "old guard" because you disagree with their food tastes or politics (which seems to be in this case)

guscost2• 14 years ago

I think that his philosophy damaged linguistics (and science in general) rather than his theory or experiment. I think that he genuinely does not know where one should end and the other should begin, especially in his problem domain, especially in 2012.

forkandwait• 14 years ago

> ... of Skinner (who, as everyone also knows, was a total psycho)

Not so fast cowboy. Skinner's theories were far more limited than he or his disciples believed (ditto Chomsky and pretty much every psych or social theorist), but the behaviorists added a great deal to psychological understanding.

It is interesting to contrast the soft sciences (for lack of a better term) with the hard sciences and how they relate to their history. Nobody would say "Newton was an utter psycho" because his theories didn't account for thermodynamic behavior, or because his dynamics were proven to be approximate by Einstein, or because in his later years he spent a lot time analyzing word patterns in the bible. Rather, what he said that was true gets incorporated into the textbooks and we move on. Sometimes Skinnerian dynamics apply, and we should use them in those situations, and learn about them, and toss them when not appropriate. Duh.

I am not sure why the soft/social sciences are so dysfunctional in this respect.

In anthropology this is extremely evident, but they are all psychos there...

http://www.theatlantic.com/magazine/archive/2012/06/the-perf...

fauigerzigerk• 14 years ago

I think you are wrong about Chomsky's theories being politically motivated. I don't have time right now to look up quotes but many years of debates with intellectuals on the left have made one thing very clear to me: The left hates talk of anything innate, particularly if it is even remotly connected to meaning.

I don't want to dumb this down because there are many different opinions on all sides of the political spectrum, but the left has a tendency to see everything as learned, learnable and socially induced - nurture not nature. Chomsky's theory is seen as a form of biological determinism. There are few things less popular on the left than that.

disgruntledphd2• 14 years ago

No offence (I like most of your post), but BF Skinner was not a psycho. He was an amazing scientist and researcher. You should really read some of his books before you judge him (Beyond Freedom and Dignity is very good, and short). Skinner was totally wrong on language, yes, but he was right on a lot of other things. He essentially invented operant conditioning, which is the basis for slot machines and much of modern gamification. In addition, he's responisble for such signs as "thank you for not smoking" (that was in Beyond Freedom).

To summate, he was an extremely interesting thinker and a fine researcher (and really loved pigeons). His theories were wrong, but all theories tend towards wrong in the limit.

aidenn0• 14 years ago

It's a question about engineering vs science. Before Kepler, people actually could predict the motion of the stars and planets through the sky; perhaps not as elegantly or accurately as after Kepler, but to a certain degree, so what?

The AI case is clearly a point where the theories from linguistics are insufficient for engineering purposes. Watson could not have been built today based off of Chomskian linguistics. Maybe the statistical models will advance the theory of linguistics, maybe not. Either way they will give us useful tools now which is better than elegant tools later.

psb• 14 years ago

Where is eyudkowski when we need him?

rm9992• 14 years ago

I've been in machine learning/AI for ten years now - from undergraduate research, to graduate school, to industry - and I find debate like this fascinating. My take on it is that our understanding of what we will be able to do in the future is very unclear, and what we will want to do is very open-ended. So the debate is worth having, but it won't really resolve anything.

Statistical models may (in my opinion probably will) end up being an "AI" dead-end, eventually falling into other fields such as algorithms, like game trees and logic-based agents did. That's not to say the current statistical approach is a bad idea; on the contrary, I think these techniques are useful and simple enough that they will become fairly ubiquitous in CS.

On the Chomsky side of the argument, AI researchers have consistently been frustrated in the past 50 years, to the point that studying AI today makes you sound like a joke. But their goal is a noble one. Anyone can understand how great it would be to have a human-level intelligence on a chip - this would fundamentally change the World. The fact that we haven't dented this problem doesn't mean the problem isn't worth solving, it just means our understanding of what it takes to build this kind of AI is in its infancy.

I almost feel like Norvig and Chomsky are arguing in parallel. They are both right, but their arguments are valid on different time scales. Today, the Norvig approach will easily win out; Chomsky has nothing and is largely irrelevant. But Chomsky is, IMO, correctly predicting what will need to happen to move beyond an eventual roadblock in a much grander AI.

VikingCoder3• 14 years ago

I picture Chomsky as Kepler, trying to build orbits out of Platonic solids.

Until Kepler had access to Brahe's data, he was not going to be able to come up with his theories of planetary motions.

Worse than that, the laws of planetary motion present a simplistic view of the universe: what happens when a bunch of small objects orbit a very massive object. I think they wouldn't help you out at all, in trying to understand planets moving in a binary star system.

There is no analytic solution to the N-body problem. We can only simulate the motions of a group of massive bodies by iteratively applying the laws of gravitation that we have deduced. Knowing the mathematical properties of how objects behave in a gravitational field, and actually understanding HOW GRAVITY WORKS are two enormously different things. Newton was frustrated with the theory of Gravity, because it was, as Norvig's models, just a model - with no explanation of why. But the model allows you to make falsifiable predictions, and understand how the universe will behave. Looking for the Higgs Boson is awesome - but there is potentially no equivalent in the linguistic world.

Chomsky asks us to ignore F = G * m1 * m2 / r^2, because there's no WHY attached to it.

PS - this understanding of the history of science is brought to you by Carl Sagan's Cosmos TV series. I have no deeper insight than that.

OmegaHN• 14 years ago

I think it is the other way around. Chomsky is trying to find the underlying structure of intelligence (just like gravity underlies planetary motion), and is saying that others are simply trying to generate a model of intelligence (through statistical methods) with no understanding of why the intelligence behaves that way. Gravity is the why, planetary motion is the model produced by data (acquired by Brahe).

VikingCoder3• 14 years ago

What will prevent Chomsky from having an Earth-centric model of the solar system, with epicycles to explain all of those weird little ticks like dropped pronouns?

The only thing that could possibly break you out of that way of thinking is massive amounts of observational analysis to show you that your foundation is flawed.

Seeing the moons of Jupiter revolutionized physics. Chomsky says that observing the heavens is unnecessary, and a distraction from his studying of the motion of billiards balls.

He's got trigonometry down cold, but he'll never come up with calculus that way. And quantum mechanics would never fall into Chomsky's way of thinking, in my analogy.

jal278• 14 years ago

But there is a WHY associated with F= G* m1 * m2 / r^2 -- an explanation for why gravity exists, which although isn't yet unified with quantum mechanics, is that mass deforms space. The statistical formula is cool on its own right though too of course -- because of its elegance.

However, a concise formula without an explanation is less interesting, and an enormous statistical model without any elegance or insight (perhaps driven by big-data) is less interesting still, although it may have practical applications.

There may be general principles to intelligence, how it evolved through natural selection, and what in general will make for an intelligent robot. These are some of the most interesting unmade discoveries. But it isn't a sharp dichotomy between "Chomsky" and "Norvig," although that is one way to frame the question.

We should explore all reasonable approaches to AI and make sure one method is not dominating the others, but otherwise let everyone explore what they will. Who really knows what actually will lead to AI?

VikingCoder3• 14 years ago

First, Norvig is not rejecting Chomsky's approach - you can't rely only on stats and probabilities. Chomsky is absolutely rejecting Norvig's approach. So, you apparently agree with Norvig.

Second, you don't know WHY mass deforms space. You don't know why some particles have mass, and others don't. You don't know how space is deformed by mass.

You wouldn't even have that explanation, "mass deforms space," without starting from the mathematical models. The mathematical models came about by studying the observed data.

Remember, people thought that cannon balls moved in a straight line, until they ran out of energy, and then they fell straight down. That was a perfectly valid view of the universe, given the observations available at the time. It even allowed you to make predictions about how far a cannonball would go, if you gave it different amounts of energy. It was also wrong.

We then cleverly figured out that the cannon ball moves in a parabola, with gravity as a constant force, pulling the cannonball down. Also wrong.

Now we know that the gravitational force exerted on the cannonball by the Earth is inversely proportional to the square of the distance between them. Meaning that you can put a cannonball in orbit, or you can even get it to escape Earth's gravity well.

Knowing "mass deforms space" doesn't help you make any of those predictions. How it deforms space does. And we, in fact, figured out that it deforms space, before we had any idea WHY, or by which mechanism.

Again, my problem with Chomsky is that he thinks we will come up with "mass deforms space," and Kepler's laws of planetary motion, before we even make any observations about the natural world.

Or rather, given our ability to measure huge quantities of text, it seems absurd that Chomsky would have us ignore the corpus, and go back to first principles as the exclusive and only way to gain any insight, be able to make any predictions, before we build any applications.

fat_clown• 14 years ago

It is an interesting debate, though I think it's being shone in the wrong light.

According to the article, it almost sounds like Chomsky believes a statistical approach to AI is a disservice to the field. The point he's missing is that research in statistical based AI is just that - statistics research.

Chomsky and Norvig deal in two different fields, which happen to have similar applications. Norvig does research in statistical and machine based learning. Success in this field comes from a new model that can make more accurate predictions, or a proof that it is impossible to make valid predictions about X with only Y as input. Applications of this field include technologies which rival AI systems as envisioned by Chomsky, but the essential point is that this field focuses on statistics research, not AI research.

Chomsky is wrong in dismissing this as a disservice. I do agree with his main point, that AI research and knowledge is not necessarily furthered by statistics research, but that is simply because they are different beasts entirely.

Maybe one day, when the biology has caught up with us and we have a solid understanding of the brain, will we be able to create a highly intelligent computer. Until then, statistics research is most likely to yield fruitful results.

slurgfest8• 14 years ago

Chomsky sees a threat to a politics-based academic hegemony, so he responds politically. If he hadn't made so many broadly quantified claims (i.e. had stuck to syntax as Norvig sticks to AI and machine learning) then those claims would not be in such jeopardy from other fields. Chomsky has always been a top-class warrior and this is more of the same.

SlipperySlope• 14 years ago

I am a entrepreneur/researcher working to create artificial intelligence. My approach follows Turing's suggestion that one should create a child mind and proceed to educate it. I employ Construction Grammar in my English dialog system - not a statistical parser/generator. Operating on a smartphone, I use available statistical speech recognition engines to transform speech to text, but from that point onwards the server-side processing in Construction Grammar is symbolic, thus engineered from first principles. Likewise, for English generation, my discourse planner emits structured RDF that the bi-directional Construction Grammar generator transforms into a text utterance. That symbolic text is then input to an available statistical text-to-speech engine available on the smartphone, to speak to the user.

As an example of the power of symbolic approaches, my parser has a complete symbolic analysis of English auxillary verb constructions, producing unique, meaning-rich, RDF-compatible semantics for:

I am learning about computers.

We are learning about computers.

We will be learning about computers.

I could be learning about computers.

I have been learning about computers.

I better learn about computers.

I had better learn about computers.

I dare learn about computers.

I did learn about computers.

I do learn about computers.

He does learn about computers.

I had learned about computers.

He has learned about computers.

I have learned about computers.

He is learning about computers.

I need learning about computers.

I ought to learn about computers.

I ought to be learning about computers.

I used to learn about computers.

I was learning about computers.

We were learning about computers.

Because of the so-far limited success of my work, I am inclined to agree with Chomsky's AI argument despite using a modern grammar opposed to his linguistic principles.

An artificial intelligence will use both statistical techniques and symbolic, e.g. procedural techniques, I think. With the most useful intelligent behavior being symbolic. E.g. an AI designing, writing and testing software.

aangjie• 14 years ago

Just for the record, i consider this a simple model. And it's from norvig. http://norvig.com/spell-correct.html

azakai• 14 years ago

First thing, please read the actual article by Norvig, it is excellent,

http://norvig.com/chomsky.html

Second: I found it astounding that the article never mentions Skinner. Surely this article is trying to do to Chomsky what Chomsky did to Skinner in 1959 ("A Review of B. F. Skinner's Verbal Behavior", http://www.chomsky.info/articles/1967----.htm ).

Chomsky basically marked the beginning of modern era of cognitive psychology with that essay, displaing the previous paradigm of behaviorism. Norvig's article has similar form in some ways to that article, and similar goals (to argue for a new paradigm over an older one). As I was reading it, I was sure Norvig had that context in mind. So I was surprised to read

> So how could Chomsky say that observations of language cannot be the subject-matter of linguistics? It seems to come from his viewpoint as a Platonist and a Rationalist and perhaps a bit of a Mystic

Well, no, Chomksy explained very well why he opposed observations being the subject matter of linguistics in his 1959 essay. Skinner's behaviorism looked only at observations and experience, and did away entirely with internal mental states. That might seem bizarre to us today, and the reason is in large part the shift heralded by Chomsky's article from behavioral psychology to cognitive psychology. In the latter, the goal is to understand the internal processes that are involved in psychology (or specifically language).

Statistical language models are not behaviorism. But they do share a lot with it, they are based primarily on raw empirical observations as opposed to deep models, so it is natural for Chomsky to oppose them on similar grounds (and not due to Platonism or Rationalism, although I suppose you can speculate that those motivated his 1959 essay too).

Side note, we can speculate that if Skinner had today's computers and statistical modelling methods, the shift from behaviorism to cognitivism might never have happened, seeing as the statistical approach is so successful.

frobbin• 14 years ago

AI research, including speech recognition and machine vision, are currently ENGINEERING disciplines trying to make artifacts that do interesting things. Success is an artifact that works.

Several basic science disciplines are trying to understand how brains work. There is mostly tremendous amounts of experimental facts, difficult to put together, and some theory and modelling to go with it.

Norvig would be confused if he thinks that engineering AI systems automatically counts as models useful for understanding the brain. If there is application to understanding brains it is a welcome accident. It happens that there are signals in basal ganglia that look like the temporal difference error signal from reinforcement learning. So maybe RL research can help understand some brain circuitry in that case.

But in general the engineers are trying to get stuff to work, and they are deluded if they think they are simultaneously making progress in understanding how brains work.

EDIT:

For example: why does speech recognition use hidden markov models and N-gram language models? Because they're the best model of how brains understand speech? No! Not at all. HMMs and N-gram models are above all computationally tractable. Easy to implement, not too slow to run.

We have algorithms (such as baum-welch and N-gram smoothing techniques) to get them work work well in engineering applications. Nothing more. Might they help us understand brains? Maybe, but not at all necessarily so.

sireat• 14 years ago

There must be some analogies made to the much smaller field of chess computer programs.

From 1950s to about 1980 or so it was thought that the best computer chess program would approximate the way a human would think about the game. Botvinnik in particular was adamant that such an approach would be the right one.

However, most of the progress was made through brute force. Modern chess programs select moves in a way that is far removed from the way a good chessplayer selects moves, yet they can now produce games that seem very "uncomputer" like and "human".

mcguire7• 14 years ago

Historically, AI has been divided into two related but different approaches. "Strong" AI is interested in understanding and creating Minds; figuring out what intelligence is, how it works, how we do it, and how it could be done in general. "Weak" AI is interested in doing things that couldn't be done before; things that we do not have good algorithms for, or don't have any algorithms at all.

Those two are not opposed. Any advance on either side helps the other. In this argument, Norvig is representing an extreme version of weak AI since he seems to be arguing that it's possible that statistical methods are all there is. (I suspect that he isn't actually making that argument, though, but that strong AI's models are currently too simplistic to capture what statistical approaches can do.) Chomsky, on the other hand, seems to be caricaturing strong AI by saying that anything that doesn't directly shed light on the Grand Theory is worthless.

mbq• 14 years ago

The main problem with Chomsky's approach is that it is quite likely that human intelligence mechanics are just incomprehensible for a human intelligence, and not because of some crazy construction tricks but simply plain old brute size and complexity it imposes. Judging from much simpler (thus deeper investigated) biological systems like some bacteria metabolisms we can see that there is no grand design there, only trivial primitive core and numerous layers of less or more subtle modifiers of modifiers. IMO there is no reason why the same can't work for the brain and thus the "transition to sentience" is way more continuous than we would like to expect.

ilaksh• 14 years ago

I know that everyone has been careful not to mention Chomsky's political beliefs, but I am suspicious that this is actually partly about Chomsky's political beliefs, which I think are more in line with reality or at least more egalitarian than Norvig's must be, since Norvig has been running one of the hegemony's greatest tools recently. I see a parallel between the general derisive dismissal of Chomsky's academic views as being simplistic with the type of dismissal commonly given to a Chomskyish geopolitical viewpoint. I see this disagreement as a surrogate for the very different geopolitical worldviews.

I doubt that Chomsky is really so hard line about his old approaches to AI as we are led to believe, although he is probably farther behind the times than Norving.

I actually think that even Norvig is just applying recent contemporary AI to AI problems, but still is part of an old or establishment guard himself as far as AI goes. I think that the real cutting edge AI research is called AGI (artificial general intelligence) research.

The generation/category of AI research or machine learning that Norvig is tied into is much newer and steps beyond the earlier traditional AI that Chomsky might have been involved with, but the AGI researchers are a step beyond Norvig's clique. And the AGI researchers are, by the way, very optimistic about the Singularity or at least the likelihood of human-like and probably super-human artificial general intelligence in the short or medium term.

I mean the Norvigish machine-learning stuff isn't completely disconnected from the AGI stuff and completely behind and I assume it will result in extremely capable AIs relatively soon, but the AGI approaches will probably prove to be more powerful and more humanlike since they are closer to human models.

Take a look at what Brain Corporation is doing, or Numenta, or the OpenCog project. That stuff is beyond Norvig and friends' approaches.

no_more_death• 14 years ago

One myth I want to debunk:

Copernicus's theory did NOT do away with epicycles. Search on Google for "copernicus epicycle" and the first article demonstrates my point. The one who did away with epicycles was Kepler. Copernicus believed orbits had to be perfectly circular; Kepler recognized that the data fit better into an elliptical model.

It's not 100% clear whether the author believed the "myth," but hopefully I can set some people straight in this forum.

6ren• 14 years ago

It's true that Engineering at times leads Science. But, from a scientific view, what's the point of a model if you can't understand it? After all, we already know how to create intelligence without understanding it.

While it's conceivable that intelligence is too complex for a human to ever understand (e.g. if not amenable to hierarchical decomposition), that would be very sad news for science.

yters• 14 years ago

Norvig is only trivially right. Sure, with enough stats you can infer a lot of the structure of all the information we humans have created, and thus replicate the structure, as Google is doing with its suggest service. However, this does not explain how humans created the structure in the first place. Such a form of AI will forever being playing catch up to humans.

stcredzero• 14 years ago

> If the solar system’s structure were open for debate today, AI algorithms could successfully predict the planets’ motion without ever discovering Kepler’s laws, and Google could just store all the recorded positions of the stars and planets in a giant database

I'm sorry, but this bit is half wrong and simply numerically illiterate. We can store all of the recorded positions of the planets and other bodies in the solar system, but we need models to predict their future positions. This is an important distinction, since we might use such models to save the human race one day.

ecolak• 14 years ago

When Einstein heard about Quantum Mechanics and the idea that everything is a probability, he said: "God doesn't roll dice". He meant that even though Quantum Mechanics does give us many answers about the world of the tiny, it doesn't truly explain it. I believe that a similar analogy can be made to this case.

Norvig vs. Chomsky and the Fight for the Future of AI

Related Stories