Characterizing emergent phenomena in large language models

https://ai.googleblog.com/2022/11/characterizing-emergent-phenomena-in.html

by fofoz • 4 years ago

77 57 4 years ago

ai.googleblog.com

djoldman• 4 years ago

Paper: https://openreview.net/forum?id=yzkSU5zdwD

xpe4• 4 years ago

Wikipedia has a fine definition of what _emergent_ means:

In philosophy, systems theory, science, and art, emergence occurs when an entity is observed to have properties its parts do not have on their own, properties or behaviors that emerge only when the parts interact in a wider whole.

The linked article uses this definition:

we discuss the phenomena of emergent abilities, which we define as abilities that are not present in small models but are present in larger models

The concept in the paper has to do with capabilities / abilities that grow non-linearly as a function of model size. This is distinctly different from _emergent behavior_ in systems theory.

<opinion>The authors and reviewers could find a better word for their concept. There is no need to muddle the concept.</opinion>

Furthermore, the idea that networks of certain sizes are necessary for certain kinds of representational abilities is not new. Perhaps a term exists already?

xpe4• 4 years ago

This comment says it more eloquently than I did: https://news.ycombinator.com/item?id=34051845

mjburgess9• 4 years ago

we discuss the phenomena of emergent abilities, which we define as abilities that are not present in small models but are present in larger models

Reading anything by major researchers in AI feels like an adversarial battle where they're trying to misuse as much technical scientific and philosophical language as possible and we adjacent are trying to hold the line.

In philosophy and esp. the philosophy of science, emergence is a relation between a whole and its parts such that a property of the whole does not obtain just in virtue of properties of its parts taken in isolation. "Emergence" has this prior positive, semi-magical, scientific association which confuses the issue in this case.

No properties of the LLM obtain from its parts differently as parameters scale, the mechanism is the same. The performance differs not due to emergence, but due to the "modelling gap" present between the statistical structure of free text and that of mathematics. With enough examples, the gap closes... indeed, you can model the addition function (add(x, y) = x + y) just by an infinite sample of its domains.

A better technical term here might be "scale-dependent capabilities". For LLM, simple arithmetic is extremely scale dependent, whereas basic text generation is less-so. The reason for this seems obvious, as given above... so the use of the term "emergence" here I interpert as more PRish mystification.

xpe4• 4 years ago

Well said. How did the authors and reviewers miss this?

mjburgess9• 4 years ago

The transition from useless to useful ML models no doubt often seems magical to researchers. But it follows just from the distribution of the training data and from the degree of its compression by the function approximation algorithm they're using.

What's "magical" is not their system, but rather that the vast library of text they use for training has useful properties which can be approximated.

What researchers are observing is more like the illusion of a "phase transition" in the quality of approximations. This illusion arises because we have discontinuous standards for the approximations.

Ie., when assessing free text prediction by LLMs there's very very many ways for them to generate an acceptable answer. For mathematics, there's only one acceptable way.

If we applied the same standard/goal to both, no such apparent "quality transition" would occur. LLMs would be exposed as equally good, or equally bad, at prediction regardless of scale.

baandang• 4 years ago

From Introduction To The Theory Of Complex Systems:

• Complex systems can exhibit a rich phase structure and have a huge variety of macrostates that often cannot be inferred from the properties of the elements. This is sometimes referred to as emergence.

This is the term and as common a term in complex systems as there is.

"scale-dependent capabilities" implies inference from elements.

I think some people just don't like the very idea even though it is not unlike the concept of stochastic process. I would think the same reasons to not like the concept of emergence applies to stochastic process. Murray Gell-Mann though couldn't even raise complex systems above arguing if emergence is magical thinking so it is probably a lost cause.

Such an interesting field that always ends up as this conversation.

PartiallyTyped• 4 years ago

that a property of the whole does not obtain just in virtue of properties of its parts taken in isolation

Thank you. I have been expressing variants of this for a while. A paper that comes to mind is OpenAI's hide and seek. They claim that cooperation is emergent behaviour, but each agent is playing it's own version of prisoner's dilemma, and thus learn to cooperate.

visarga5• 4 years ago

That model was not learning from language, it was learning from a simulation. When you can use a simulation to produce training data it is possible to have a model discover new abilities all on its own, like AlphaGo.

seydor2• 4 years ago

Eh eh . imagine how neuroscientists feel

mjburgess9• 4 years ago

Well, of late, NS's have been fond of misusing "hallucinate" likewise which means a non-veridical perception. And they're using it to mean a veridical constructed perception.

Leading everyone down a path of ever-more mysticism.

It would be nice if neuroscientists spoke out against both their own mystical PR and that of AI, but I don't hear it much.

naasking4• 4 years ago

I think "scale-dependent capability" is a much more precise term for what they're describing, but I'm not sure that that term doesn't fall under the general umbrella of emergent properties. The opening paragraphs of emergent properties in philosophy [1] cites a number of examples that I would argue are comparable to LLM suddenly becoming able to do arithmetic past a certain scale.

In philosophy and esp. the philosophy of science, emergence is a relation between a whole and its parts such that a property of the whole does not obtain just in virtue of properties of its parts taken in isolation.

This is not a settled definition. I think everyone can agree that this applies epistemologically, where studying the parts in isolation cannot always yield sufficient information to predict macroscopic properties, but to claim that all properties of the whole do not reduce to the properties of its parts is controversial.

For instance, it seems unlikely that we would have predicted H2O's dipole moment and the phenomenon of surface tension just from studying hydrogen atoms and oxygen atoms in isolation, but it would be incorrect to say that surface tension is not the result of the properties of hydrogen and oxygen in isolation. We simply cannot discover all the relevant properties without studying them together.

Edit: to clarify, H2O's dipole moment seems obvious in hindsight when we have a good model for what's going on, and analogously, LLM's ability to do arithmetic seems obvious as a scale-dependent property in hindsight, but that doesn't mean it was obvious that this would happen before it was created.

[1] https://plato.stanford.edu/entries/properties-emergent/

mjburgess9• 4 years ago

Well it's scale dependence is an illusion.

It's uniformly able to model Q->SingleAnswer problems, and uniformly able to model Q-> ManyAnswer problems.

Basic arithmetic is the former kind, and it becomes useful to people at large scales, ie., accurate on basic arithmetic.

This dual condition of "useful-to-people" is the thing introducing the illusion, since it changes depending on what we're modelling. The system isnt acquring any new property.

Consider a researcher putting a book on a thin ice-sheet, and then putting a car on it. Here, they're concluding the ice has different properties in each case -- but it doesnt.

fumeux_fume• 4 years ago

Is is that big of a deal? The authors explain their definition of emergent abilities at the beginning of the paper.

mjburgess9• 4 years ago

It's a mystification of what's going on -- the term makes it harder to understand, not easier. It's prone to popular misunderstanding, and it seems even to confuse the researchers themselves.

YeGoblynQueenne6• 4 years ago

Thanks for bringing some sense into the debate. It's scandalising to see how the machine learning research community is so ready to jump on to such ... innovative uses of terminology.

Take "few shot learning", for instance. OpenAI's preprint introducing GPT-3 was titled "Large Language Models are Few Shot Learners" [1]. This was promptly adopted by the community, even though LLMs need first to be trained on millions of examples before they can accept "few", and they don't even learn from those few (because no weight updates). So it's not really "few shot" and it's not really "learning", and yet, here we are. "Large Language Models are few-shot learners" and nobody bats an eyelid anymore.

Which is not to say we have to shut up and take it. I personally draw inspiration from the tale of the little child who pointed out the King's sartorial negligence.

________________

[1] That, in itself, is a title designed to claim some groundbreaking progress not just in technological capabilities but also in scientific understanding. LLMs are few-shot learners, man! Few-Shot!

mgraczyk4• 4 years ago

I don't see the issue here. The exact same definition of "few shot learning" has been used for at least 20 years. Nothing changed with the GPT-3 paper.

The definition is something like

Given a task, a few shot learner is an algorithm that generalizes well with only a small number of training examples for that task.

The same definition is what I'm familiar with from undergrad. Do you know of a different definition that precedes GPT-3?

maria2• 4 years ago

Why should we let philosophy define technical terms for ML? Many words have many meanings. Welcome to the imprecision of human language.

As a slight tangent, I really hate this type of comment that inevitably appears on many HN submissions. Personally, I find it distracting when the main conversation happening on an article is a pedantic discussion on whether or not a word means what the author thinks it means.

HarHarVeryFunny• 4 years ago

I'm OK with the term emergent used here - it seems the best word to describe non-trivial capabiities/properties that weren't designed in. At least prior to the first LLMs, I think most people would just expect them to be capable of doing literally what they are trained to do - predict (locally plausible) next word(s), and this is certainly all that is "designed in" if we're talking about a basic LLM (vs one with further RL-based "alignment", etc).

Of course we can also appreciate that to get REALLY REALLY good at "predict next word" would require intelligence/understanding of what is being generated, but I think the point here is would anyone - the model designers in particular - have expected that the transformer architecture (+ scale) is all it would take to become so good at this task? I don't think "attention is all you need" was really anticipating transformers reading API docs and generating code to perform requested functions! One might have expected it to take a much more elaborate and evolved architecture to achieve this level of capability!

So, to me, it seems entirely appropriate to describe these models as having emergent capabilities - things they are capable of doing that are really so far above and beyond "predict next word", that it seems churlish and inaccurate to describe them as designed it (or even just confidently predicted).

rafaelero• 4 years ago

What a waste of time to be worried about how people are using a word.

ot• 4 years ago

I believe the term is reasonably appropriate here.

The abilities being described here are "emergent" in the sense that the model was not specifically trained for them, but they show up anyway. Your example is about modeling a specific function and having its accuracy increase with model complexity, which is classical ML formulation, but this is not what is happening here.

LLMs are trained on a very simple task: given a natural text, predict the next word. But as model complexity and training set sizes increase, they start exhibiting more sophisticated abilities, such as basic forms of reasoning, and contextual memory.

In your definition, the parts of the whole are "lots of statistics about text" and the emergent property is "semantic reasoning".

Scale is inevitably a part of this: somewhere else in the thread you mention that "liquidity" is an emergent property of H2O, but if you take a handful of H2O molecules they don't behave as a liquid.

smeagull• 4 years ago

People speak in metaphors. I don't know why people get so hung up when it happens. Language isn't a formula unless you're a model. Even then, LLM do cope with some idioms.

xg15• 4 years ago

With enough examples, the gap closes... indeed, you can model the addition function (add(x, y) = x + y) just by an infinite sample of its domains.

Infinity is not a number.

Trainsets cannot be infinite and neither were the number of examples of addition that we humans saw before we became able to do addition on unseen samples.

So the question of how finitely many samples you need to generalize a math operation is a different one.

seydor2• 4 years ago

The X axis here is the training Flops but what about parameter size and how does it account for the different architectures. Comparing apples to shoelaces may not be a fruitful approach or indicative of what to expect from ever-expanding scale. Also , is it emergence or overfitting

evrimoztamur• 4 years ago

Has there been any efforts in processing calculation prompts, where instead of letting it internally 'compute', it's trained to identify equations and process them with an external calculator instead (perhaps one which outputs not only the result but the individual steps too)?

obiefernandez• 4 years ago

Been wondering the same

vutekst• 4 years ago

Yes: https://twitter.com/goodside/status/1581805503897735168

visarga5• 4 years ago

Language models with toys. The calculator, Python REPL, search engine, database, simulation, games, other AI's can easily blend with large language models lifting some weight off their shoulders.

For example, for a physics question the LM could write a small simulation, run the simulation and interpret results back to the user. That's possible when models can do code execution.

ttctciyf• 4 years ago

There's a quite accessible IAS presentation[1] from another Google researcher on Solving Quantitative Reasoning Problems with Language Models which gives some likely related background on having language models solve this type of math problem, including the "chain of thought" technique mentioned here.

I found it pretty interesting and as something of an ML skeptic was a bit surprised at the degree of coherence shown in "reasoning" examples similar to the ones in the linked article.

1: https://www.youtube.com/watch?v=qV4Ku5L4BuMt

CGamesPlay2• 4 years ago

Do these scale-dependent (I like this adjective better than "emergent") properties survive model distillation? It may be that our training/optimization processes are inefficient and require these scales to achieve, but the underlying model may not actually require the number of parameters that we are giving them. I haven't read any of the papers about distillation yet, does anyone know if this has been tested?

visarga5• 4 years ago

Good question, my guess is that you can't distill chain-of-thought or zero shot prompting in small models, they got to be 15-20B parameters or larger. Maybe someone has a link to a related paper?

lossolo• 4 years ago

For all models smaller than 62B, direct prompting outperforms CoT. The first model where CoT outperforms direct prompting is Flan-cont-PaLM 62B on BBH. For 540B models, there are more settings where CoT outperforms direct prompting, but not all of them. Also, the number can be smaller than 540B. In Suzgun et. al. 2022, the authors show that the 175B InstructGPT and 175B Codex also have better CoT performance than direct prompting. Combining all the results, we get the two numbers 62B and 175B. So yes indeed, to enter the game of scale you do need a ticket to larger models than average.

However, there are also other large models like OPT, BLOOM, and the first version of GPT-3. They all have 175B, yet their CoT performance is significantly worse, or even cannot do CoT.

source: https://yaofu.notion.site/A-Closer-Look-at-Large-Language-Mo...

Characterizing emergent phenomena in large language models

Related Stories