The Bitter Lesson (2019)

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

by pierre • 2 years ago

45 2 2 years ago

mjburgess• 2 years ago

The question comes down to what the research project is. Are we modelling intelligence, or are we simply trying to find a solution to a specific problem?

I take intelligence to be "what you do when you do not know what you're doing", or equivalently, a class of algorithms which build conceptualisations of an environment, of one's capacities, (and so on), which "furnishes the mind" with the appropriate building blocks to begin the traditional narrow/weak AI-style algorithmic techniques.

Few, if any departments are actually researching intelligence in this sense (I'd say a few peripheral areas in zoological neuroscience, etc.).

In this "specific problem case" brute-forcing a well-defined outcome space is obviously superior to any intelligence-based strategy. Intelligence, in this sense, is an extremely expensive waste of time.

This isn't much of a bitter lesson, except to those who naively think intelligence is actually a superior method of solving problems. In general, it's a terrible method. Only of course, when the problem has effectively been solved by being able to provide a well-defined outcome space.

Recently, Yann LeCun said (iirc via linkedin) that the frontal lobe engages in conceptualisation and the sensory-motor system is action-planning. If that's your view, then you still havent learned this bitter lesson.

The situation is exactly the opposite. Cognition, as performed by the frontal lobe, is what you do when you have the right "furniture of the mind" (aproximate outcome spaces, etc.). It is the body, via the sensory-motor system, which must expend great expense to get you there.

This is why computer scientists are every disappointed by brute-search. They're in the wrong research project.

sgt101• 2 years ago

The standard reading is : "hardware wins".

But there is a second part.

"The second general point to be learned from the bitter lesson is that the actual contents of minds are tremendously, irredeemably complex; we should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries. All these are part of the arbitrary, intrinsically-complex, outside world. They are not what should be built in, as their complexity is endless; instead we should build in only the meta-methods that can find and capture this arbitrary complexity. Essential to these methods is that they can find good approximations, but the search for them should be by our methods, not by us. We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done."

This is proving quite tendentious at the moment. Do we want agents that discover like we can? The clattering impact of rogue models, even when they are being used just to make pretty pictures - as opposed to engineering diagrams, patient care plans, fraud investigations.... is making me, and many others wonder. In fact some of the projects that are supporting the discovery of precision like alphafold might be the most promising?

The Bitter Lesson (2019)

Related Stories