The meeting of the minds that launched AI

https://spectrum.ieee.org/dartmouth-ai-workshop

by fremden • 3 years ago

173 53 3 years ago

spectrum.ieee.org

johndhi• 3 years ago

My father and his friends were academic computer scientists working on AI back in the 60s. I don't know that there's a straightforward path between what they were doing and the popular LLMs today, but I do applaud more stories on what old school comp sci researchers were up to.

empath-nirvana• 3 years ago

There's some value in putting a flag in the ground. Even if most of those people there were in the symbolic camp, a lot of their critiques of neural networks as they existed were well-founded and were really only proved obviously _wrong_ after many many rounds of moore's law.

marcosdumay• 3 years ago

The criticism from the beginning was of a fundamental theoretical nature, and died at the 90's when people proved and demonstrated that neural networks were powerful enough to run any kind of computation.

In fact, I don't recall people criticizing neural networks from being too small to be useful. Ever. There was a lot of disagreement between wide and deep network proponents, that deep won by demonstration, but "how large a network we need to handle X" was always more of a question than a "see, we'll never get there". (Even more because the "we will never get there" is obviously false, since the thing practically no limit on scaling.)

fnordpiglet4• 3 years ago

LLMs of today display amazing abductive abilities but are limited in inductive and deductive abilities, as well as other optimization techniques of classical AI and algorithms. These abductive abilities are unique and exciting because we’ve typically done really poorly with ambiguous and complex semantic spaces like this. However I think the excitement has obscured the fact it’s just a piece of a larger machine. Why do we care that LLMs are mediocre chess players when we have machine models using more traditional techniques that are the best chess players on earth? Why do we care they fail at deductive reasoning tests? At mathematical calculations? Those are really well understood areas of computing. Somehow people have fixated on the things we’ve already done that this new technique fails at, but ignore the abilities LLMs and other generative models demonstrate we’ve never achieved before. At the same time the other camp only sees generative AI as the silver bullet tool to end all other tools. Neither is correct.

dr_dshiv2• 3 years ago

How are LLMs bad at induction? I thought they were great at induction. This paper doesn’t go into measurements of it, but helps lay out the nature of reasoning well.

https://aclanthology.org/2023.findings-acl.67.pdf#page15

famouswaffles5• 3 years ago

but are limited in inductive and deductive abilities

LLMs are great at induction.

In a broad sense, they are also very good at deduction.

"I define a new word, the podition. A podition is any object that can fit on a podium. Is a computer a podition ? Why ?"

A correct answer is deductive.

LLMs eat these kind of questions for breakfast. Even the OG 2020 GPT-3 could manage them.

You really do have to stretch deduction to heights most people struggle with to have them falter majorly.

TradingPlaces• 3 years ago

Summer camp for mathematicians

aborsy3• 3 years ago

Other than Minsky, I don’t think others (who are nevertheless scientists in their respective fields) are considered to have made significant contributions to modern machine learning or AI. McCarthy’s work around this topic culminated in LISP, leading to Emacs, a text editor!

From that period, Rosenblatt’s work was instrumental to modern AI.

abecedarius• 3 years ago

Solomonoff's https://en.wikipedia.org/wiki/Solomonoff%27s_theory_of_induc... is about as basic to the theory of intelligent agents as anything gets.

(He's in the pic and I'd guess this article was by a relative.)

astrange2• 3 years ago

If I was an intelligent agent, I would prefer to be based on a theory that was computable without time travel, which this one isn't.

daveguy• 3 years ago

Claiming that the creator of LISP did not have a significant impact on AI is not a defensible position.

JamilD• 3 years ago

People forget for how long Lisp had an impact on AI, even outside GOFAI techniques; LeCun's early neural networks were written in Lisp: https://leon.bottou.org/publications/pdf/sn-1988.pdf

aborsy3• 3 years ago

The story goes as, John McCarthy was applying for an assistant professorship position at MIT. MIT told him, but we have here Norbert Wiener who was a renowned mathematician at the time and had published cybernetics some time ago, in which he talks about agents interacting with the environment and feedback control, sort of modern computation-based AI. McCarthy changed the name from cybernetics to AI, and focused on symbolic systems and logic. The approach was generally not successful.

Some people consider that the logic-based approach to AI pioneered in this conference contributed to an (what we now call) AI winter. People like John Pierce of Bell Labs, a very influential figure in government, defunded research in computation-based AI such as for speech recognition (he wrote articles, saying, basically, researchers pursuing these techniques are charlatans).

There is no major algorithm or idea in undergrad machine learning textbooks named after these people. There are other people from that era.

fipar• 3 years ago

Emacs is so much more than a text editor! But I need to stay on topic...

I believe your assessment of LISP (and therefore of MacArthy)'s impact on AI to be unfair. Just a few days ago https://github.com/norvig/paip-lisp was discussed on this site, for example.

dwroberts2• 3 years ago

Shannon did foundational work on the theory of computers being able to play chess. That stuff might seem ‘obvious’ but you have to remember that would have seemed futuristic if not impossible when it was first proposed. That work in itself is fundamental to a lot of subsequent AI research (esp just the concept of ‘game playing’ as a model for testing approaches)

aborsy3• 3 years ago

If I recall, Shannon returned a proposal to McCarthy for research during his stay on the cellular automata (perhaps inspired by von Neumann), not machines playing games.

Shannon did foundational research in information theory, communications, cryptography, digital relay circuit design, and gambling. Crediting him for impact on AI and machine learning is a bit of a stretch (even though the IEEE Spectrum and Bell Labs published articles claiming credit, indicating that Shannon played a role).

He was an avid chess player, and demonstrated machines doing various things: playing, finding their ways in a maze, juggling etc. It wasn’t based on a foundational theory of computers playing games (and the approach is different from the reinforcement learning or optimization of the models).

currymj• 3 years ago

Shannon was, I believe, the first person to define an n-gram probabilistic language model of English text and then sample from it!

simonw• 3 years ago

My favourite detail about that 1956 meeting is this extract from the conference proposal:

An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.

I think this may be one of the most over-ambitious software estimates of all time.

The whole proposal is on https://en.wikipedia.org/wiki/Dartmouth_workshop

madrox• 3 years ago

I forget where I read it, but there was a similar assessment of the time needed for computer vision. Overall, quite an optimistic time.

BHSPitMonkey• 3 years ago

The promise was "a significant advance in one or more of these areas", not the Holy Grail. Not unreasonable!

kaycebasques• 3 years ago

I just learned about this conference a couple weeks ago while watching the Computer History Museum video on AI: https://youtu.be/NGZx5GAUPys?si=aVDZAmpR2ziKq4x9

(Video is from 2014)

shon• 3 years ago

67 years later: https://aiconference.com

swyx• 3 years ago

and https://ai.engineer/summit ! as this industry builds out there need to be multiple forums for different groups to get together

Animats• 3 years ago

They tried so hard.

I went through Stanford CS in the 1980s, just as it was becoming clear that the predicate-calculus / expert systems approach to AI wasn't going to do much outside of very well defined problem spaces. It was very frustrating. I'd already done machine proof of correctness work, but that's in a problem space you can formalize. So I had a sense of the limits of the approach, and was rather surprised by the broad claims made for it at Stanford.

I took McCarthy's "Epistemological problems in artificial intelligence" class. One day he was describing the missionary-and-cannibals problem (amusingly, in a less violent form). He started out with it as a word problem, and then he set it up so that his new "circumscription" approach could be applied. Turn the crank on that algorithm, and an answer pops out. The word problem to formalism conversion struck me as the step where, as an old New Yorker cartoon showing a mathematician explaining something put it, "a miracle occurs". Formalizing the problem is the hard part.

The classic critique along those lines is "Artificial Intelligence meets Natural Stupidity", by Drew McDermott.[1] That was written in 1976, and basically describes why that class of AI systems hit a wall in the 1980s. The "AI Winter" followed, the 1980s AI startups went bust, and not much happened for 15 years. Well-known history.

[1] https://cs.fit.edu/~kgallagher/Schtick/Serious/McDermott.AI....

dontreact• 3 years ago

This group of people may have been the first to mention the words “AI” prominently in academia but is this flag planting or are they truly foundational to the work with the same name today? If none of these people had done anything, would we really be far behind?

My sense is that modern AI has more to owe Fukushima’s neocognitron, Hubel and Wiesel, and the connectionists than any intellectual descendant of the work mentioned here.

p-e-w2• 3 years ago

Modern AI depends on one thing above all else: The unimaginable compute power provided by today's GPUs.

All theoretical foundations are trivial by comparison. The basic math can be understood by an interested high school student, but unless you are able to do billions of matrix multiplications per second, it doesn't mean anything. AI was going nowhere before Nvidia.

Calling these people "the minds that launched AI" is like calling Archimedes the father of hot-air balloons, because he recognized the principle of buoyancy.

gumby2• 3 years ago

Are you trolling? Minsky, McCarthy, Newell and Simon all went on to win Turing awards for their work, (as later did several other AI luminaries over the decades). And Claude Shannon?

In the mid sixties Minsky and Papert published a paper/book called "Perceptrons" which explored the limit of perceptrons, though it said those limits could be overcome by multilayer networks, if they were ever computationally feasible. And, thanks to Moore's law, they now are.

Almost every one of those attendees is a well known scholar whose work you depend on every day outside AI (as well as within it, though nobody really depends NNs yet). Of them, the least known in computing, Ray Solomonoff, might be the smartest of the bunch, though he was never univerity affiliated as far as I know.

p-e-w2• 3 years ago

And, thanks to Moore's law, they now are.

Not thanks to "Moore's law". Thanks to the countless engineers who made that happen, and whose work is much more important for today's AI systems than the theory that was cooked up in the 60s and 70s.

Your comment is a typical example of the hero-worship towards theorists, and the casual disregard towards engineers, that is so common in today's science culture.

Any above-average grad student could reinvent the perceptron network from scratch. Good luck having a grad student (or even a Nobel laureate) redesign the H100 GPU from scratch.

_0ffh2• 3 years ago

You got that wrong, the problem with multilayer networks at the time was not about scaling, but how to train such a network at all.

Using error backpropagation for training multilayer networks was what overcame that problem, not Moore's law, or anything else.

fodmap• 3 years ago

For those interested in reading Peter Milner's autobiography, here's the pdf.

https://www.sfn.org/-/media/SfN/Documents/TheHistoryofNeuros...

ggm2• 3 years ago

As a 7 year old, I met two of them at IFIP68 in Edinburgh. I don't have much recall of Minsky, but McCarthy was nice. We went out to turnhouse airport and he flew a light plane around. At dinner he commented we did the washing up by hand and when he offered to send us a dishwasher my mother (of course) said "no no John don't be silly" but she said to me in the kitchen later on: "if he ever offers again accept"

We also ate at a chinese restaurant across town and he was surprised when me and my brothers and sisters said birds-nest soup was disgusting.

Susy McCarthy stayed with us for a while. She was a typical Californian teenager, ate her cereal with the wrong hand holding the spoon and her feet up on the chairs and we younger kids were told in no uncertain terms not to emulate her. She refused to go to school and there were exasperating phone calls about what to do. In the end she went somewhere else. Johns wife died climbing Mt Everest. it was really sad. I think it was the first all-women's climbing team to try the route (or even Everest's pinnacle)

Minsky and McCarthy were quite happy to respond to emails in the 80s and 90s. I asked Minsky about writing with Harry Harrison and he said he was frustrated at the ending of the book they did ("the Turing Option") and McCarthy talked about his ideas of long term information ontologies in a notation or language he called "Elephant"

McCarthy was a notable USENET curmudgeon and would be oppositional to any anti-nuclear, anti-oil, anti-plastic agenda he saw. He also said he cried when he learned the bombs had been dropped, and that the invasion of Japan was off, because he knew he wouldn't die on a beach in an opposed landing. (It is possible I have this mixed up with memories of what Paul Fussel wrote, but I am pretty sure John said this)

John played "Alice's Restaurant" with a guitar, he had a very nasally whiny voice.

blast• 3 years ago

As a 7 year old, I met two of them at IFIP68 in Edinburgh

Can you fill in some background? What was IFIP68 and how did you end up there as a 7 year old?

ggm2• 3 years ago

IFIP68 was the 1968 conference of IFIP (The International Federation for Information Processing) held in Edinburgh.

My dad was one of the conference organisers. Local host maybe? He was the foundation chair of the computer science department at the University of Edinburgh. As he was involved in the academic programme committee he had the principal keynote speakers back for socials, dinners and the like. I went to the trade show which was held alongside the conference, I got two cardboard mock pdp-8 computers from the Digital Equipment stand, they had just announced the PDP8/I configuration and were handing out cardboard blanks to prospective buyers. My memory is that this was in the old exhibition space next to Waverley station, which had a roof with gardens and glass insets, it was removed in the 70s. It was the kind of space which had the ideal home show, that kind of thing, but packed out with hardware vendors trying to sell mainframes.

I think I had the only 2 node parallel cardboard pdp-8 in existence.

joshu• 3 years ago

iirc simon and newell arrived with working code despite not yet having a working a computer at CMU

The meeting of the minds that launched AI

Related Stories