Modded-NanoGPT: NanoGPT (124M) quality in 3.25B tokens

by ocean_moist • 2 years ago

81 11 2 years ago

gavindean90• 2 years ago

Seems like this is a modded NanoGPT not the original.

munchler• 2 years ago

Yes. It’s literally called “Modded-NanoGPT”.

Scene_Cast2• 2 years ago

I wonder how much improvement is owed to which changes. I've also never heard of "Muon - Momentum Orthogonalized by Newton-schulz" being used.

EDIT: there's a bit more info on his twitter - https://x.com/kellerjordan0

It looks like he created this optimizer. Works on 2D matrices only.

whiplash451• 2 years ago

Cool work. No license?

molticrystal• 2 years ago

Just needs a Zero To Hero series episode offering line by line commentary to follow along on why each choice was made over alternatives.

m3kw9• 2 years ago

So it compresses info better.

pyinstallwoes• 2 years ago

That is literally intelligence.

parineum2• 2 years ago

It's not.

byyoung3• 2 years ago

do you have a baseline of the regular implementation with 3x learning rate?

Related Stories