kz919

"AdamW has been the default optimizer for transformer pretraining. For many years, our community searches for faster and more stable optimizers with only constraint positive outcomes. In this work, we propose a \textbf{single-line modification in Pytorch} to any momentum-based optimizer, which we rename Cautious Optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that this modification preserves Adam's Hamiltonian function and it does not break the convergence guarantee under the Lyapunov analysis. In addition, a whole new family of optimizers is revealed by our theoretical insight. Among them, we pick the simplest one for empirical experiments, showing speed-up on Llama and MAE pretraining up to 1.47×. Code is available at https://github.com/kyleliang919/C-Optim"

obviously because they can't beat it... There will be zero reason to use it when you have better transformer based models that can fit the existing infrastructure.

you are not wrong, but I don't think that is their point. It's just a muscle flex to shows you that their hardwares work. If it can train 176B or whatever, it should be an easy peasy for them to train 13B.

you just need to teach it to use a calculator https://arxiv.org/pdf/2302.04761.pdf

I wonder how consistent the chip output would be. Also compiler and software stack sound painful to build.