From Memorization to Reasoning in the Spectrum of Loss Curvature

by andy12_ • 9 months ago

65 14 9 months ago

andy12_2• 9 months ago

Very concise summary of the procedure described in this paper:

1. Run the model once across a dataset to estimate loss curvature per MLP weight matrix via K-FAC (activation/gradient covariances).

2. Decompose each weight matrix into curvature-ordered components; low-curvature directions correspond most to verbatim memorization, higher curvature to shared/general mechanisms.

3. Edit by dropping the low-curvature subspace and keep only the top directions.

vessenes• 9 months ago

Thank you for this huge time saver.

Now, about the paper-that’s super interesting. I imagine the dream here is to distil down into a “reasoning” core. Or maybe reclaim space for more generalization. Lots of interesting use cases.

getnormality3• 9 months ago

Thank you!

I think you may have accidentally switched low and high in #2, no? The abstract speaks of high curvature as associated with memorization:

curvature for memorized training points is much sharper than non memorized

radarsat1• 9 months ago

This sounds more correct to me. I've read previously somewhere that better generalization is usually associated with wider, smoother minima, and this is why regularization is important, because it has a smoothing function on the loss landscape.

andy12_2• 9 months ago

Actually, no! Look at this in the paper

In extending from studying per-example to bulk memorization, we propose a novel inversion of the previous interpretation of loss curvature: while individual memorized points are associated with high curvature, the direction of curvature varies across examples, meaning that, averaged across multiple examples, memorization directions are actually flatter than generalizing directions, which maintain a consistent moderate curvature across points

vatsachak• 9 months ago

The decomposition they use "averages out the points of high curvature" therefore those components of the decomposition which correspond to "higher curvature" are those components which are used across multiple data points. Therefore they are the "general reasoning"

kingstnap• 9 months ago

A very similar idea is presented here in the first 5 minutes of this recent talk. But more from observing a kink in loss curves.

https://youtu.be/UyK3DgWY7yw?si=NN3f9Erik8o_Nfbs

NitpickLawyer• 9 months ago

Our work enhances the understanding of memorization in neural networks with practical applications towards removing it

Cool stuff. In a recent podcast Karpathy was also talking about this. He sees this as the next "target": models that don't memorise, because you can look it up in an oracle, but still keep the "reasoning" qualities.

esafak2• 9 months ago

How can you generalize without facts? They are the foundation on which generalization is built. Like programming without memorizing the keywords. Unless you make a distinction between facts that let you generalize, and facts that do not, like random ID numbers.

icandoit• 9 months ago

We want the LLM to learn the multiplication algorithm not an incomplete set of tables. The algorithm might be smaller and will be more complete.

Honestly, our technology has outpaced our epistemology. So we don't really know what a fact is or isn't. Are facts what we call our supervised learning experiences? You think the sun rises, no the earth spins. Your belief that the sun rises helps you predict sunset and sunrise. Your belief would be quaint to someone born and raised on a space station. Apollos chariot moves the sun across the sky doesn't it?

esafak2• 9 months ago

There is a related line of work that suggests spikes in the ESD are related to the generalization vs memorization too; e.g.,

From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks (https://openreview.net/pdf?id=DJHB8eBUnt)

From Memorization to Reasoning in the Spectrum of Loss Curvature

Related Stories