Attention Residuals

https://github.com/MoonshotAI/Attention-Residuals

by GaggiX • 4 months ago

241 34 4 months ago

jszymborski2• 4 months ago

This is reminds me of the input gates of an LSTM.

vibe42• 4 months ago

Interesting to compare with deepseek mHC (Jan 2026), which also changes how residual connections work: https://arxiv.org/abs/2512.24880

jjcm• 4 months ago

Two things stand out to me with this:

1. Drops compute required for training by ~20%. This approach wont just help the ever escalating model sizes larger companies are pushing for, it means things like autoresearch can iterate on new model architectures faster.

2. WAY lower bandwidth requirements for inference. Means with approaches like this it should run on consumer hardware far better. It apparently requires 1/6th the memory bandwidth of a traditional approach for better results.

This is a big improvement if it can be generalized. They're claiming it's a drop in replacement, so it seems like it can as well.

dvt2• 4 months ago

Drops compute required for training by ~20%.

This is not true. Authors claim that w.r.t. training, their method adds negigible overhead for AttnRes with no memory impact (but is way more complicated for Block AttnRes since we need to use pipelining for larger models, hence the O(Ld) & O(Nd) figures, with N ≪ L).

WAY lower bandwidth requirements for inference.

Also not true. Paper has nothing to do with inference, apart from the benchmarks. If you're looking at the graph about "compute advantage," it's about training compute. They do some interpolation to get to the 1.25x number, basically answering the question "if non-AttnRes architecture were trained, how much compute would it take to get to the same loss as AttnRes?" (The answer being ~20% more compute.) It's an interesting claim, but there's all kinds of weird and unexpected convergence that can happen, so take it with a grain of salt.

observationist2• 4 months ago

I think what they're getting at is that for a given unit of compute, this method achieves 125% performance.

If model A reaches performance level 100 using 100 units of compute using old methods, and you train model B using AttnRes, aiming at performance level 100, it costs you 80 units of compute.

It probably doesn't map precisely, but that's where people are diverging from the claim - it doesn't explicitly say anything about reduced inference or training time, but that's the implicit value of these sorts of things. Less compute to equivalent performance can be a huge win for platforms at scale as well as for local models.

com2kid• 4 months ago

2. WAY lower bandwidth requirements for inference. Means with approaches like this it should run on consumer hardware far better. It apparently requires 1/6th the memory bandwidth of a traditional approach for better results.

That should be the headline right there. Giant side 60 font headline.

Some people have PhDs in burying the lede!

talloaktrees• 4 months ago

except it's not true

westurner• 4 months ago

ScholarlyArticle: "Attention Residuals" (2026) https://arxiv.org/abs/2603.15031 :

Abstract: Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the memory and communication overhead of attending over all preceding layer outputs for large-scale model training, we introduce Block AttnRes, which partitions layers into blocks and attends over block-level representations, reducing the memory footprint while preserving most of the gains of full AttnRes. [...]

czbond• 4 months ago

Ah - now I understand how this has 2k+ (supposedly legitimate) Github stars in less than a week. Thank you - I was more skeptical

jryio• 4 months ago

This is the key piece

Full AttnRes is straightforward but requires O(Ld) memory at scale. Block AttnRes partitions layers into N blocks, accumulates within each block via standard residuals, and applies attention only over block-level representations. With ~8 blocks, it recovers most of Full AttnRes's gains while serving as a practical drop-in replacement with marginal overhead.

Murfalo• 4 months ago

Amazingly, the first author is a high school student! https://nathanchen.me/public/About%20me.html

brcmthrowaway• 4 months ago

We're about to get an onslaught of young Chinese geniuses (raised in China). It's pure statistics

Sadly, same can't be said about India (infrastructure/food security lags China).

jldugger3• 4 months ago

It's pure statistics

I'm not so sure about that: https://www.populationpyramid.net/china/2026/ suggests peak high school in china was years ago.

eru• 4 months ago

I don't think you can blame food security here.

Even if food security holds back 10% of Indians (which would still be a huge tragedy), that would still leave the other 90% for the 'onslaught'. 10% is just a made up number. But even with 50% you'd get an 'onslaught'.

So if we are seeing less than that, it's probably down to other factors.

srean• 4 months ago

Sadly, same can't be said about India

And quality of leadership.

They (barring a few exceptions) are happy to gloat in imagined past glories of vedic aeroplanes, inter-species head transplants apparent performed in Hindu golden age and loyalty based funding that produces institutions like Galgotia University.

casey2• 4 months ago

Sadly, not the US either. It entirely cultural. Starting around Victorian times wealthy people took great pride in children who didn't work, fast forward to the industrial revolution a symbol of poverty was the child being the main breadwinner. Combine both of those class signalling methods and you get children who are locked out of work, but this manifests as forced busywork, electives, sports & music

In China since this class doesn't exist and forced busywork for class signaling is largely driven underground due to government policies driven young people just work on whatever the government policy says, which right now is tech. Hence you see smart young people getting into tech. While in the US they do Finance (if they built up the right portfolio to be allowed in the elite jobs club), speedrunning (if they didn't). Ironically the current US look a lot like imperial China, while China looks like the US of the late 1800s, but fully vertically integrated

bawana• 4 months ago

Long, long ago, I remember the first toy I ever got that was made in chiina. It was a wooden cube puzzle. various interlocking differently shaped pieces that when assembled formed a cube. It was so different to all the other toys made in america by hasbro, mattel, tonka, etc. Back then I felt like I was holding a toy made by the ancient Greeks, a puzzle to teach geometry, analysis, pattern recognition. So abstract, so removed from daily life, it transported me into a different world. Like chess, it was an engaging abstraction. But unlike chess, it was not about conflict but rather interrelating pieces to make a greater whole.

So this is what really unsettles me. Not that China graduates more engineers every year than we have entirely employed in the US, but rather, that these individuals are not about delegating work, but actually doing it. Whereas the western credo is to get someone else to do the work (or in the words of PAtton, to get some one else to die for his country), I get the feeling that China will get robots and AI to do the work. I am reminded of the joke about Chinese factories having only 1 security guard and 1 dog. The guard is there to feed the dog.

youknownothing• 4 months ago

"the western credo is to get someone else to do the work"

This. One of the things that most shocked me when I moved to London was how bad English people were at hard skills, but also how easily giving orders and "projecting gravitas" came to them. Everyone wants to be a "leader", which sadly has become code for reaping benefits of other people's work.

Eufrat• 4 months ago

I don’t want to discount the hard work that went into this, but I think it is always important to recognize this is mostly not a representation of competence, but of means and access. I have never seen someone fortunate enough to get such opportunities that wasn’t from affluence. Even the high school he attends (Basis International Park Lane Harbour^) is in an affluent resort community in southern China.

I don’t say this out of spite, it’s awesome he’s been able to get such opportunities and has been able to utilize them, but I think it’s important to be realistic about what these narratives imply without additional context and the answer is, sure, you can be successful with grit, brains and determination, but…the odds are very much not in your favor if you are in the wrong income bracket.

^ This is not doxxing. His LinkedIn profile states it plainly.

caderosche• 4 months ago

Very cool!

imtringued• 4 months ago

I don't know why all the posts fail to summarize the results properly.

I had a similar idea at the back of my head but here is a layman explanation:

Standard attention threads the previous layers output to the next layers input. By adding residual connections to each layer, the layers learn an update rule.

There is an obvious limitation here. Only the first layer gets to see the original input and all subsequent layers only get to see the previous layer output.

With attention residuals, the idea is that you have a tiny attention operator that decides between using the original input and any of the previous layer outputs.

antirez• 4 months ago

Exactly. I was reading all the other comments and wondering why many looked like they were talking of something else.

ilreb• 4 months ago

[dupe]https://news.ycombinator.com/item?id=47401111

roger_• 4 months ago

Great idea and seems quite obvious in hindsight.

Is it guaranteed to have the same effect on vanishing gradients though? What if it put weight 1 on a layer that had a tiny gradient?

Attention Residuals

Related Stories