Let's try to understand AI monosemanticity

https://www.astralcodexten.com/p/god-help-us-lets-try-to-understand

by bananaflag • 3 years ago

350 178 3 years ago

astralcodexten.com

turtleyacht2• 3 years ago

By the same token, thinking in memes all the time may be a form of impoverished cognition.

Or, is it enhanced cognition, on the part of the interpreter having to unpack much from little?

aatd863• 3 years ago

Some kind of single context abstract interpretation maybe.

throwanem• 3 years ago

Darmok and Jalad at Mar-a-Lago.

kibwen2• 3 years ago

Shaka, paying to build the walls.

dekhn3• 3 years ago

By the same token, thinking in memes all the time may be a form of impoverished cognition.

I would recast this: any thinking is a linear superposition of weighted tropes. If you read TVTropes enough you'll start to realize that the site doesn't just describe TV plots, but basically all human interaction and thought, nicely clustered into nearly orthogonal topics. Almost anything you can say can be expressed by taking a few tropes and combining them with weights.

turtleyacht2• 3 years ago

That's what I wonder, though: mimetic thinking as a viral phenomenon. It may reshape one's perceptions, forcing a certain conformance, like prions.

If there's any room for alternatives or unconventional thought, it must now be assigned within an established hierarchy. Or, it must be shoehorned into invisible guardrails.

This is not bad in itself. Of course, useful ideas must transmit successfully to realize progression over decades.

But if one has a choice to focus attention, there also ought to exist the option to ignore it.

I don't mean to "ignore all frameworks" or "everything original must come from the self." I just mean I avoid TVTropes--yet I also embrace Dramatica.

No doubt TVTropes is a form of design patterns for writing, for which it has incredible value. It's just not something I want to "shape my mind around."

As I write this, I know it doesn't make much sense. It may just be a silly thing, and I have yet to "mature to realize the concept of" a shared, tropic mindview.

erikerikson4• 3 years ago

Before finishing my read, I need to register an objection to the opening which reads to me so as to imply it is the only means:

Researchers simulate a weird type of pseudo-neural-tissue, “reward” it a little every time it becomes a little more like the AI they want, and eventually it becomes the AI they want.

This isn't the only way. Back propagation is a hack around the oversimplification of neural models. By adding a sense of location into the network, you get linearly inseparable functions learned just fine.

Hopfield networks with Hebbian learning are sufficient and are implemented by the existing proofs of concept we have.

janalsncm9• 3 years ago

This is true. We use backpropagation not because it’s the only way or because it’s biologically plausible (the brain doesn’t have any backward passes) but because it works. Neural networks aren’t special because of any sort of connection to the brain, we use them because we have hardware (GPUs) which can train them pretty quickly.

I feel the same way about transformers vs RNNs: even if RNNs are more “correct” in some sense of having theoretically infinite memory it takes forever to train them so transformers won. And then we developed techniques like Long LoRA which make theoretical disadvantages functionally irrelevant.

bilsbie• 3 years ago

developed techniques like Long LoRA which make theoretical disadvantages functionally irrelevant.

How’s that?

erikerikson4• 3 years ago

I agree that doing effective things is effective and we should do effective things when they prove valuable. These things here are generally simplifications or even distillations which can be faster and more efficient which are excellent attributes of solutions. My objection isn't that we develop great systems but that we don't forget that other branches in the solution space exist. Particularly here because the more rich enough may help yield the next generation of solutions.

dekhn3• 3 years ago

I think it would be really exciting if somebody could show that ANNs that more resembled biological neurons could learn function approximation as well as (or better than!) current DNNs. However, my understanding of math and engineering suggests that for the time being, the mechanisms we currently use and invest so much time and effort into will exceed more biologically inspired neurons, for utterly banal reasons.

erikerikson4• 3 years ago

Yet brains remain superior in many regards. Still, by all means, let's celebrate the advances we make!

blovescoffee3• 3 years ago

Back propagation isn't a hack. It's a triumph. It's powering the revolution we're experiencing.

erikerikson4• 3 years ago

It has yield some excellent results that we should celebrate and use. Yet the consensus of the field, when I was properly informed on such things, was that we developed back propagation while under a misunderstanding that linearly inseparable functions (e.g. XOR) could not be learned by Hebbian learning in Hopfield networks. This was a correct result in the context of his assumptions but too limited and/or over applied conclusion from Minsky's work.

s1gnp0st2• 3 years ago

Shouldn’t the AI be keeping the concept of God, Almighty Creator and Lord of the Universe, separate from God-

This seems wrong. God-zilla is using the concept of God as a superlative modifier. I would expect a neuron involved in the concept of godhood to activate whenever any metaphorical "god-of-X" concept is being used.

Sniffnoy• 3 years ago

I mean, it's not actually. It's just a somewhat unusual transcription (well, originally somewhat unusual, now obviously it's the official English name) of what might be more usually transcribed as "Gojira".

s1gnp0st2• 3 years ago

Ah, I thought the Japanese word was just "jira". My mistake.

lukev• 3 years ago

There's actually a somewhat reasonable analogy to human cognitive processes here, I think, in the sense that humans tend to form concepts defined by their connectivity to other concepts (c.f. Ferdinand de Saussure & structuralism).

Human brains are also a "black box" in the sense that you can't scan/dissect one to build a concept graph.

Neural nets do seem to have some sort of emergent structural concept graph, in the case of LLMs it's largely informed by human language (because that's what they're trained on.) To an extent, we can observe this empirically through their output even if the first principles are opaque.

wahern2• 3 years ago

Neural nets do seem to have some sort of emergent structural concept graph, in the case of LLMs it's largely informed by human language (because that's what they're trained on.) To an extent, we can observe this empirically through their output even if the first principles are opaque.

Alternatively, what you're seeing are the structures inherent within human culture as manifested through its literature[1], with LLMs simply being a new and useful tool which makes these structures more apparent.

[1] And also its engineers' training choices

hprotagonist3• 3 years ago

i buy this. cf the “lossy jpeg of the web” analogy; sure there’s structure: we put it there!

TeMPOraL4• 3 years ago

Alternatively, both are the same thing. The structures embedded in human culture, which the literature captures, are themselves manifestations of structures of our understanding on the world.

Kapura• 3 years ago

This gets at the fundamental issue I have with AI: We're trying to get machines to think, but we only have the barest understanding of how we, as humans, think. The concept of a "neuron" that can activate other neurons comes straight from real neurology, but given how complex the human brain is, it's no surprise that we can only create something that is fundamentally "lesser."

However, I think that real neurology and machine-learning can be mutually reinforcing fields: structures discovered in the one could be applied to the other, and vice versa. But thinking we can create "AGI" without first increasing our understanding of "wet" neural nets is the height of hubris.

sxg• 3 years ago

AGI and human intelligence don't necessarily have to work the same way. While many of us assume that they must be very similar, I don't think there's any basis for that assumption.

mitthrowaway23• 3 years ago

We're trying to get machines to think, but we only have the barest understanding of how we, as humans, think... But thinking we can create "AGI" without first increasing our understanding of "wet" neural nets is the height of hubris.

That depends on the landscape of the solution space. If the techniques that work much better than anything else happen to be the same techniques used within our own brains, then we might find them, through aggressive experimentation and exploration, before we even realize that it's also how our own brains work too.

dboreham• 3 years ago

Hard disagre. Imho humans suffer under a delusion that their thinking is like a computer whereas it is actually much more like an LLM. So if you ask humans[1] to figure our how they think, the answer will be a) long time coming and b) wrong.

[1] possibly bhudist monks excepted.

hibikir• 3 years ago

All the efforts to get computers to play chess mostly like humans do were made to look insufficient by a large enough neural network that took zero knowledge of human position evaluation, which just played against itself an outrageous number of games.

The plane is in many ways lesser than birds, and we barely understood aerodynamics when the wright brothers gave us a working plane: While the bird is efficient, we had a whole lot more thrust. Our planes are far better than they were back then, but not because we understand bird better, but because we focused on efficiency of the simpler designs we could build. The Formula 1 car doesn't come from deep understanding of the efficient, graceful running movements of the cheetah.

Our results in medicine have been far and ahead of our understanding of biochemistry, DNA, and in general, how the human body works. The discovery of penicilin didn't require a lot of understanding: It required luck. Ozempic and Viagra didn't come from deep understanding of the human body: Resarchers were looking for one thing, and ended up with something useful that was straight out unintended.

We might be able to build AGI with what we know of brains, we might not. But either way, it's not because we need to understand the human mind better: Do we have enough compute for a very crude systems that we know how to build to overcome our relative lack of understanding?

So if you ask me, what is the height of hubris is to think that working engineering solutions have to come from deep understanding of how nature solves the problem. The only reasonable bet, given the massive improvements in AI in the last decade, is to assume that the error bars of any prediction here are just huge. Will we get stuck, the way self-driving cars seem to have gotten stuck for a bit? Possibly. Will be go from a stochastic parrot into something better than humans, the same way that Go AIs went from pretty weak to better than any human in the blink of an eye? I'd not discount it either. I expect to be surprised either way.

throwuwu2• 3 years ago

I see no reason why this is a problem. We built analogs of many things before we fully understood how they work. Most of human progress has worked this way through an empirical approach. Makes sense since the real world is the ultimate sandbox and the final judge of an idea’s applicability.

DemocracyFTW2• 3 years ago

thinking we can create "AGI" without first increasing our understanding of "wet" neural nets is the height of hubris

Already ~40 years ago I read in a popular science book that scientists had tried to apply genetic algorithms to finding a better nozzle for a given purpose (jet engine? I don't remember that). That was then done by prefabricating a number of ring-shaped metal parts the inside of which had different widths and different slopes; a stack of these could form a wide variety of shapes for the conduit. They started with an initial configuration, did a measurement, then threw the dice to determine what part should be changed; if the new measurement was better (according to their metric) than the preceding one, they kept it, otherwise reversed it. They did find a nozzle shape that was not only significantly better than any known configuration; it was also significantly weirder, seemingly completely ad-hoc.

There's no computers in this experiment, no intelligent design; people were only necessitated to procure the initial parts and setup, do the dice-throwing and so on; the design process proper was a very basic, randomized, mechanical procedure, no thinking required, no understanding necessary.

_as_text2• 3 years ago

I just skimmed through it for now, but it has seemed kinda natural to me for a few months now that there would be a deep connection between neural networks and differential or algebraic geometry.

Each ReLU layer is just a (quasi-)linear transformation, and a pass through two layers is basically also a linear transformation. If you say you want some piece of information to stay (numerically) intact as it passes through the network, you say you want that piece of information to be processed in the same way in each layer. The groups of linear transformations that "all process information in the same way, and their compositions do, as well" are basically the Lie groups. Anyone else ever had this thought?

I imagine if nothing catastrophic happens we'll have a really beautiful theory of all this someday, which I won't create, but maybe I'll be able to understand it after a lot of hard work.

mathematicaster• 3 years ago

I did a little spelunking some time ago reacting to the same urge. Tropical geometry appears to be where the math talk is at.

Just dropping the reference here, I don't grok the literature.

KhoomeiK• 3 years ago

You might be interested in this workshop: https://www.neurreps.org/

And a possibly relevant paper from it:

https://openreview.net/forum?id=Ag8HcNFfDsg

zozbot234• 3 years ago

ReLU is quite far from linear, adding ReLU activations to a linear layer amounts to fitting a piecewise-segmented model of the underlying data.

drdeca2• 3 years ago

Well, at all but a finite number of points (specifically all but one point), there is a neighborhood of that point at which ReLU matches a linear function...

In one sense, that seems rather close to being linear. If you take a random point (according to a continuous probability distribution) , then with probability 1, if look in a small enough neighborhood of the selected point, it will be indistinguishable from linear within that neighborhood.

And, for a network made of ReLU gates and affine maps, still get that it looks indistinguishable from affine on any small enough region around any point outside of a set of measure zero.

So... Depends what we mean by “almost linear” I think. I think one can make a reasonable case for saying that, in a sense it is “almost linear”.

But yes, of course I agree that in another important sense, it is far from linear. (E.g. it is not well approximated by any linear function)

jimsimmons• 3 years ago

Everything is something. Question is what this nomenclature gymnastics buys you? Unless you answer that this is no different than claiming neural networks are a projection of my soul

kdmccormick2• 3 years ago

what this nomenclature gymnastics buys you?

???

Are you writing off all abstract mathematics as nomenclature gymnastics, or is there something about this connection that you think makes it particularly useless?

seanhunter• 3 years ago

I would argue that there are a few fundamental ways to make progress in mathematics:

1. Proving that a thing or set of things is part of some grouping

2. Proving that a grouping has some property or set of properties (including connections to or relationships with other groupings)

These are extremely powerful tools and they buy you a lot because they allow you to connect new things in with mathematical work that has been done in the past. So for example if the GP surmises that something is a Lie group that buys them a bunch of results stretching back to the 18th century which can be applied to understand these neural nets even though they are a modern concept.

rlupi• 3 years ago

Could looking at NN through the lens of group theory unlock a lot of performance improvements?

If they have inner symmetries we are not aware of, you can avoid waste in searching in the wrong directions.

If you know that some concepts are necessarily independent, you can exploit that in your encoding to avoid superposition.

For example, I am using cyclic groups and dihedral groups, and prime powers to encode representations of what I know to be independent concepts in a NN for a small personal project.

I am working on a 32-bit (perhaps float) representation of mixtures of quantized Von Mises distributions (time of day patterns). I know there are enough bits to represent what I want, but I also want specific algebraic properties so that they will act as a probabilistic sketch: an accumulator or a Monad if you like.

I don't know the exact formula for this probabilistic sketch operator, but I am positive it should exist. (I am just starting to learn group theory and category theory, to solve this problem; I suspect I want a specific semi-lattice structure, but I haven't studied enough to know what properties I want)

My plan is to encode hourly buckets (location) as primes and how fuzzy they are (concentration) as their powers. I don't know if this will work completely, but it will be the starting point for my next experiment: try to learn the probabilistic sketch I want.

I suspect that I will need different activation functions that you'd normally use in NN, because linear or ReLU or similar won't be good to represent in finite space what I am searching for (likely a modular form or L-function). Looking at Koopman operator theory, I think I need to introduce non-linearity in the form of a Theta function neuron or Ramanujan Tau function (which is very connected to my problem).

blovescoffee3• 3 years ago

What? The very point of neural networks is representing non-linear functions.

SpaceManNabs• 3 years ago

deep connection between neural networks and differential or algebraic geometry

I disagree with how you came to this conclusion (because it ignores non-linearity of neural networks), but this is pretty true. Look up gauge invariant neural networks.

Bruna et al. Mathematics of deep learning course might also be interesting to you.

shermantanktop• 3 years ago

As described in the post, this seems quite analogous to the operation of a bloom filter, except each "bit" is more than a single bit's worth of information, and the match detection has to do some thresholding/ranking to select a winner.

That said, the post is itself clearly summarizing much more technical work, so my analogy is resting on shaky ground.

gmuslera• 3 years ago

At least the first part reminded me of Hyperion and how AIs evolved there (I think the actual explanation is in The Fall of Hyperion), smaller but more interconnected "code".

Not sure about actual implementation, but at least for us concepts or words are not pure nor isolated, they have multiple meanings that collapse into specific ones as you put several together

daveguy• 3 years ago

All this anthropomorphizing of activation networks strikes me as very odd. None of these neurons "want" to do anything. They respond to specific input. Maybe humans are the same, but in the case of artificial neural networks we at least know it's a simple mathematical function. Also, an artificial neuron is nothing like a biological neuron. At the most basic -- artificial neurons don't "fire" except in direct response to inputs. Biological neurons fire because of their internal state, state which is modified by biological signaling chemicals. It's like comparing apples to gorillas.

lainga• 3 years ago

Wait till you find out how we talk about evolutionary adaptation... a couple analogies and elided concepts here and there and you'd swear Lamarck had smothered Darwin in his cradle

jjw1414• 3 years ago

Yes, Lamarck is alive and well. I frequently hear colleagues in biology with years of experience making lazy statements such as, "Humans evolved to walk because..." as if evolution is a conscious act. In many cases, these people are using the term evolution as shorthand for, "Random mutations gave rise to new gene variants (alleles), and over numerous generations, natural selection leads to an increase in the prevalence of beneficial variants, specifically those that provide a reproductive advantage. The genetic variants that combined to enable bipedal locomotion provided such an advantage to the human population". Evolution is a passive, natural process driven by genetic variation, environmental changes, and the differential reproductive success of individuals with advantageous traits. This in turn results in population changes. Evolution doesn't involve a conscious choice or direction. Evolution is the result of the cumulative effects of these factors over long periods of time.

aatd863• 3 years ago

Do you mean that artificial neurons are inherently passive while biological neurons are inherently active i.e. they would act in spite of external input?

Just wondering if I understood you, I don't know anything on the subject.

baobabKoodaa• 3 years ago

That was already addressed in the comment you're responding to. They wrote that in the case of artificial neural networks "at least we know [there is no god inside the machine]". With regards to humans, we don't know.

pengstrom• 3 years ago

I'd say 'in spite'-ness is pretty spot on for life in general

Notatheist• 3 years ago

I feel anthropomorphizing is perfectly reasonable especially in this context. How would you like it described?

janalsncm9• 3 years ago

There’s so much baggage attached with it. The only thing a neural network “wants” is to minimize its loss function. That’s all.

lacrimacida• 3 years ago

Anthropomorphizing is valid in relation with laguage, human language. Other than that anthropomorphizing would be somewhat valid with other human outputs but ideally it is understood it’s just one of the lenses in the toolkit, that it has pros and cons just like any idea or any tool.

kibwen2• 3 years ago

It's one thing to deliberately anthropomorphize in order to construct an analogy.

It's another thing to anthropomorphize by accident or by illusion, as per pareidolia: https://en.wikipedia.org/wiki/Pareidolia . Just as in pareidolia, where the human brain is primed to "see" a human face in a certain pattern of light and shapes, it seems that human brains are primed to "see" a human intelligence in the output of an LLM, because our brains are pattern-matching on "things that look like human speech". But that's a reason to not anthropomorphize LLMs, precisely because people are inclined to do so without thinking.

ShamelessC• 3 years ago

Wait what are you referring to specifically? Any anthropomorphism in the article is _clearly_, clearly the author's admitted simplification due to the incredible density of the subject matter.

Given that, I honestly can't find anything too upsetting.

In any case, anthropomorphism is something I don't mind, mostly. Is it misleading? For the layman. But the domain is one of modeling intelligence itself and there are many instances where an existing definition simply makes sense. This happens in lots of fields and causes similar amounts of frustration in those fields. So it goes.

janalsncm9• 3 years ago

Humans also use neural nets to reason about concepts. We have a lot of neurons, but so does GPT-4.

I feel this is an abuse of the language. Biological neurons and ANN neurons aren’t the same or even all that similar. Brains don’t do backprop for example. Only forward passes. There’s a zoo of neurotransmitters which change the behavior of individual neurons or regions in the brain. Unused neurons in the brain can be repurposed for other things (for example if your arm is amputated).

drones• 3 years ago

All this anthropomorphizing of humans strikes me as very odd. None of these humans "want" to do anything. They respond to specific input. Maybe artificial neural networks are the same, but in the case of humans we at least know it's a simple reaction to neurotransmitter signals.

arolihas3• 3 years ago

You're probably just being tongue in cheek here, but I still wanted to add that we humans do actually "want". Desires, intentions, these are real things. You can try to represent them as loss functions or rewards in some mathematical model but that isn't the original thing. Consider this excerpt from https://scottaaronson.blog/?p=7094#comment-1947377

Most animals are goal-directed, intentional, sensory-motor agents who grow interior representations of their environments during their lifetime which enables them to successfully navigate their environments. They are responsive to reasons their environments affords for action, because they can reason from their desires and beliefs towards actions.

In addition, animals like people, have complex representational abilities where we can reify the sensory-motor “concepts” which we develop as “abstract concepts” and give them symbolic representations which can then be communicated. We communicate because we have the capacity to form such representations, translate them symbolically, and use those symbols “on the right occasions” when we have the relevant mental states.

(Discrete mathematicians seem to have imparted a magical property to these symbols that *in them* is everything… no, when I use words its to represent my interior states… the words are symptoms, their patterns are coincidental and useful, but not where anything important lies).

In other words, we say “I like ice-cream” because: we are able to like things (desire, preference), we have tasted ice-cream, we have reflected on our preferences (via a capacity for self-modelling and self-directed emotional awareness), and so on. And when we say, “I like ice-cream” it’s *because* all of those things come together in radically complex ways to actually put us in a position to speak truthfully about ourselves. We really do like ice-cream.

oivey• 3 years ago

This is the definition of anthropomorphism: “Anthropomorphism is the attribution of human traits, emotions, or intentions to non-human entities.” By definition the point you’re making doesn’t make sense.

EMM_386• 3 years ago

None of these humans "want" to do anything. They respond to specific input

Weird, because I'm pretty sure I had the choice whether to respond to this comment.

It wasn't the light waves hitting my retina from an HN post, leading to nerves firing and neurotransmitters all coming together to post this.

I posted it because I have free will. I almost didn't.

Unless you truly feel that there is no free will, and that reality is just a bizarre movie we have to experience .... well, then ... we'll disagree.

famouswaffles5• 3 years ago

None of these neurons "want" to do anything. They respond to specific input.

Yes well, your neurons don't "want" to do anything either.

Maybe humans are the same, but in the case of artificial neural networks we at least know it's a simple mathematical function

So what, magic ? a soul ? If the brain is computing then the substrate is entirely irrelevant. Silicon, biology, pulleys and gears. all can be arranged to make the same or similar computations. If you genuinely believe the latter, it's fine. The point is that "simple" mathematical function is kind of irrelevant. Either the brain computes and any substrate is fine or it doesn't.

Also, an artificial neuron is nothing like a biological neuron.

They're not the same but "nothing like" is pushing it a lot. They're inspired by biological neurons and the only reason modern NNs aren't closer to their biological counterparts is because they genuinely suffer for it, not because we can't.

Biological neurons fire because of their internal state, state which is modified by biological signaling chemicals

Brains aren't breaking break causality. The fire because of input.

airgapstopgap2• 3 years ago

Comments like this are incredibly grating. You condescend to the interlocutor for making a mistake which only exists in your own mistaken world model. Your confidence that neurons and ANN weights and «pulleys and gears» are all equivalent because there is, in theory, an intention to instantiate some computation, and to think otherwise is tantamount to belief in magic and broken causality, is just confused and born out of perusing popular-scientific materials instead of relying on scientific literature or hands-on experience.

The fire because of input.

No they do not fire because of input, they modulate their firing probability based on input, and there are different modalities of input with different effects. Neurons are self-contained biological units (descended, let me remind you, from standalone unicellular organisms, just like the rest of our cells), which actually have an independently developing internal state and even metabolic needs; they are not merely a system of logic gates even if you can approximate their role with a system of equations or an ANN. This is very different, mechanistically and teleologically. Hell, even spiking ANNs would be substantially different from currently dominant models.

So what, magic ? a soul ? If the brain is computing then the substrate is entirely irrelevant

Stop dumbing down complex arguments to some low-status culture war opinion you find it easy to dunk on.

benchaney• 3 years ago

Can you be more specific about what particular anthropomorphizing you object to? The only place the author uses the word want is in describing the wants of humans.

gremlinunderway• 3 years ago

I think you're being a bit pedantic here.

Neurons also don't "respond" to specific input either. They can't speak or provide an answer to your input.

These are all just abstract metaphors and analogies. Literally everything in computer science at some point or another is an abstract metaphor or analogy.

When you look up the definition and etymology of "input", it says to "put on" or "impose" or "feed data into the machine". We're not literally feeding the machine data, it doesn't eat the data and subsist on it.

You could go on and on and nitpick every single one of these, and I don't think the use of "want" (i.e. anthropomorphizing the networks to have intent) is all that bad.

clbrmbr• 3 years ago

I didn’t hear much anthropomorphism outside of the subtitle.

Anyways, this is Scott’s writing style. I recall an earlier ACX post on alignment that was real heavy on ascribing desires and goals to AI models.

error9348• 3 years ago

No one knows how it works. Researchers simulate a weird type of pseudo-neural-tissue, “reward” it a little every time it becomes a little more like the AI they want, and eventually it becomes the AI they want.

There is a distinction to be made in "knowing how it works" on architecture vs weights themselves.

asylteltine• 3 years ago

I’ve never understood the hidden layers argument. Ultimately these models are executing code. You can examine the code. Why can’t that be done?

afthonos2• 3 years ago

Think of it this way: looking at the neurons is like looking at the assembly and trying to understand how a video is played on YouTube. Yes, in theory it’s possible, but realistically no one would be able to do that. At least not in a reasonable amount of time.

astrange• 3 years ago

Video decoding works by reading and writing pixels to the same buffers you use for displaying them, so you could see a lot of how it works by watching the accesses there.

OkayPhysicist4• 3 years ago

Fundamentally, artificial neural networks boil down to some code that can be written, mathematically, as

σ(C*σ(B*σ(A*x)))... = y

repeated matrix-vector multiplication, separated by nonlinear functions, σ(). A,B,C etc here are your weights, x is your input vector, y is your output vector. Any nonlinear function will work, some behave better than others, but they must be present, otherwise you could simplify the problem by apply the associative property to the weights side of things, and you'd effectively have a single layer that can only simulate linear functions. A hidden layer is just any vector that is intermediate to that calculation, for example the result of σ(A*x) would be the first hidden layer.

So right off the bat, you obviously have a problem: "Examining the code" means examining the weights, which are just gigabyte/terabyte sized matrices. Opaque is putting it mildly. The only sane approach is to start from one of our known-meaningful vectors (the input or output), and work our way inwards from there, either seeing what elements of hidden layer vectors have the most significant value when a certain input is applied, or determining what hidden layer vector values produce values closest resembling a desired output.

Then you start running into the problems described in the article.

janalsncm9• 3 years ago

This is a really good explanation for a fully connected network. Most models will have more complex architectures so are even more complicated than this.

drdeca2• 3 years ago

If by “code” you mean “the source code people wrote”, then, that source code by itself does not determine the behavior. The behavior is determined by the numbers.

Saying that understanding that code implies understanding how the network works, is like saying “John wrote this emulator for GBA games, therefore he must understand how the game [some GBA game] works”.

Except, instead of the game being written in assembly, it was written in malbolge.

cwillu• 3 years ago

“This is great for AIs but bad for interpreters. We hoped we could figure out what our AIs were doing just by looking at them. But it turns out they’re simulating much bigger and more complicated AIs, and if we want to know what’s going on, we have to look at those. But those AIs only exist in simulated abstract hyperdimensional spaces. Sounds hard to dissect!”

“Still, last month Anthropic’s interpretability team announced that they successfully dissected of one of the simulated AIs in its abstract hyperdimensional space.

(finally, we’re back to the monosemanticity paper!)

First the researchers trained a very simple 512-neuron AI to predict text, like a tiny version of GPT or Anthropic’s competing model Claude.

Then, they trained a second AI called an autoencoder to predict the activations of the first AI. They told it to posit a certain number of features (the experiments varied between ~2,000 and ~100,000), corresponding to the neurons of the higher-dimensional AI it was simulating. Then they made it predict how those features mapped onto the real neurons of the real AI.

They found that even though the original AI’s neurons weren’t comprehensible, the new AI’s simulated neurons (aka “features”) were! They were monosemantic, ie they meant one specific thing.

Here’s feature #2663 (remember, the original AI only had 512 neurons, but they’re treating it as simulating a larger AI with up to ~100,000 neuron-features).”

quickthrower2• 3 years ago

With these models, the Pytorch code is an interpreter, and the 100Gb of weights is the code. So you have say a 10G (10 weights = 1 LOC) LOC codebase, all in assembly, no comments. G'luck.

Oh and it doesn't always use logic. So there are no ORs and ANDs and IFs, just +, *, exp, max, etc.

adamnemecek2• 3 years ago

Superposition makes sense when you understand all of ML as a convolution.

blovescoffee3• 3 years ago

You can just note the fact that two bits can represent four values.

adamnemecek2• 3 years ago

Can you explain machine learning through this lens?

kevdozer1• 3 years ago

such a wonderful article, really enjoyed reading it

nl• 3 years ago

Personally I find the original paper much better written and easier to understand: https://transformer-circuits.pub/2023/monosemantic-features/...

zone411• 3 years ago

If you have a background in ML, then yes, a paper is almost always better. I've recommended papers over sources like Towards Data Science etc. here. However, for laypeople, I doubt it would be as effective - they'd need to look up terms like MLP, ReLU, UMAP, Logit, or even what an activation function is, and they are the target audience of this post.

FeepingCreature4• 3 years ago

There's also a difference in goals. The point of a paper is to convince an expert that the claim is correct. With an article, you can just assume that the claim is correct (or give it any level of credence you want), because the point is to inform more than convince. So the article can skip or sketch all the showing-your-work and steps and proofs and just present the conclusions.

csours2• 3 years ago

I feel like we're a few more paradigm shifts away from self-driving cars, and this is one of them - being able to actually understand neural nets and modify them in a constructive way more directly - aka engineering.

Some more:

    cheaper sensors (happening now)
    better sensor integration (happening now, kind of)
    better tools for ml grokking and intermediate engineering (this article, kind of)
    better tools for layering ml (probably the same thing as above)
    a new model for insurance/responsibility/something like this (unsure)
    better communication with people inside and outside the car (barely on the radar)

Merrill2• 3 years ago

When LLMs are trained on text, are the words annotated to indicate the semantic meaning, or is the LLM training process expected to disambiguate the possibly hundreds of semantic meanings of an individual common word such as "run"?

OkayPhysicist4• 3 years ago

The task LLMs are trained on is "predict the next word", which elegantly is included for free in your training set of text. Typically no annotation is provided, since that would involve a ton of human labor doing the annotations.

Merrill2• 3 years ago

Could you ask a specialized AI to define each of the words in a block of text to automate the meaning extraction?

DonsDiscountGas• 3 years ago

Isn't this also(just?) a description of how high-dimensional embedding spaces work? Putting every kind of concept all in the same space is going to lead to some weird stuff. Different regions of the latent space will cover different concepts, with very uneven volumes, and local distances will generally be meaningful (red vs green) but long-distances won't (red vs. ennui).

I guess we could also look at it the other way; embedding spaces work this way because the underlying neurons work this way.

bjackman• 3 years ago

I had the same feeling, I have always seen it like:

1. Neurons encode a concept and activate when it shows up.

2. No, it's way more complicated and mysterious than that.

And now this seems to add:

3. Actually, it's only more complicated than that in a fairly straightforward mathematical sense. It's not that mysterious at all.

I suspect this means that either I'm not picking up on subtleties in the article, or Scott is representing it in a way that slightly oversimplified the situation!

On the other hand, the last quote in the article from the researchers does seem to be hitting the "it's not that mysterious" note. A simple matter of very hard engineering. So, I dunno. Cool!

ggm• 3 years ago

As long as you read it with the skeptics "yes, but it's not intelligence" its a good read.

It's when you read it with the "at last, I can understand how reasoning and inference with meaning is going to emerge from this" you have a problem.

It's a great read but what you bring to it, informs what you take from it.

lngnmn2• 3 years ago

Every training set will produce a different set of weights, even the same training set with produce different weights with different initialization, leave alone slightly different architectures.

So what exactly is the point, except "look at us, we are so clever"?

chrissnow2023• 3 years ago

COOL~ interpreting a big AI with a bigger is like interpreting 42 with Earth~

keithalewis• 3 years ago

[disclaimer: after talking to many people much smarter than me, I might, just barely, sort of understand this. Any mistakes below are my own]

Thanks for the heads up! AI is better at making up stories than humans already. Hodl my beer while I go buy some BSitCoins.

Nelkins• 3 years ago

This reminds of research done on category-specific semantic deficits, where there can be neurodegeneration that impacts highly specific or related knowledge (for example, brain trauma that affects a person's ability to understand living things like zebras and carrots, but not non-living things like helicopters or pliers).

https://academic.oup.com/brain/article/130/4/1127/278057

dmd• 3 years ago

The idea of a smaller NN outperforming what you might think it could do by simulating a larger one reminds me of something I read about Portia spiders (going on a deep dive into them after reading the excellent 'Children of Time' by Adrian Tchaikovsky). The idea is that they're able to do things with their handful - on the order of tens of thousands - of neurons that you'd think would require 4 or 5 orders of magnitude more, by basically time-sharing them; do some computation, store it somehow, then reuse the same neurons in a totally different way.

pas• 3 years ago

see https://www.nature.com/articles/nature12160 "mixed selectivity neurons"

brindlejim2• 3 years ago

This is egregiously false. ML models are math and code. They are not tissue; there are no neurons in an ML model. None. The analogy is false. And it is used here to make you think that ML models are more human than they are, or more superhuman. There is no "bigger AI waiting to get out". Shoggoth anyone? These are scare tactics from hyperventilating EAs (both Scott and the Anthropic team).

makeworld• 3 years ago

There are neurons.

https://en.wikipedia.org/wiki/Artificial_neuron

brindlejim2• 3 years ago

The repetition of a false analogy does not make it true. They are no more neurons than transistors are "little brain cells". This is just shorthand for outsiders who can't hope to understand. But on HN, we have technologists. The nodes of an ML model are made of math and code. There is no neuron there, and those nodes work very different than neurons in their action and structure. The analogy breaks down.

drc500free• 3 years ago

The original is quite long, but quite interesting.[1] Reading it makes me feel like I did reading A Brief History of Time as a middle schooler - concepts that are mainly just out of reach, with a few flashes that I actually understand.

One particularly interesting topic is the "theories of superposition" section, which gets into how LLMs categorize concepts. Are concepts all distinct or indistinct? Are they independent or do they cluster? It seems that the answer is all of the above.

This ties into linguistic theories of categorization[2] that I saw referenced in (of all places) a book about the partition of Judaeo-Christianity in the first centuries CE.

Some categories have hard lines - something is a "bird" or it is not. Some categories have soft lines - like someone being "tall." Some categories work on prototypes, making them have different intensities within the space - A sparrow, swallow, or robin is more "birdy" than a chicken, emu, or turkey. Apparently Wittgenstein was the first to really explore with Family Resemblances that a category might not have hard boundaries, according to people who study these things.[3] These sorts of "manifolds" seem to appear, where some concepts are not just distinct points that are or aren't.

It's exciting to see that LLMs may give us insights into how our brains store concepts. I've heard people criticize them as "just predicting the next most likely token," but I've found myself lost when speaking in the middle of a garden path sentence many times. I don't know how a sentence will end before I start saying it, and it's certainly plausible that LLMs actually do match they way we speak.

Probably the most exciting piece is seeing how close they seem to get to mimicking how we communicate and think, while being fully limited to language with no other modeling behind it - no concept of the physical world, no understanding of counting or math, just words. It's clear when you scratch the surface that LLM outputs are bullshit with no thought underneath them, but it's amazing how much is covered by linking concepts with no logic other than how you've heard them linked before.

[1] https://transformer-circuits.pub/2023/monosemantic-features/...

[2] https://www.sciencedirect.com/science/article/abs/pii/001002...

[3] https://en.wikipedia.org/wiki/Family_resemblance

Let's try to understand AI monosemanticity

Related Stories