Scaling Laws, Carefully

https://lilianweng.github.io/posts/2026-06-24-scaling-laws/

by tehnub • 26 days ago

88 20 26 days ago

lilianweng.github.io

I really wish more people skeptical of AI capabilities would read about scaling laws -- Lilian is always so marvelous at giving a deep overview of the technical side but the whole point of this is: there are scaling laws, and they hold and continue to hold. This is such a huge basis for the predictions about AI capabilities for the past like 5 years.

FromTheFirstIn5• 26 days ago

And sitting right next to the data and compute factors in every cross entropy loss equation is the entropy of the language, which is just a fixed constant. There’s such a hard cap on cross entropy loss training and I never hear it come up!

aspenmartin4• 26 days ago

Right but that is context dependent; it drops with context length, depends on tokenizer, etc. It doesn't end up being super relevant, despite the fact that if you look at the loss for real models it's relatively large in absolute terms. But that doesn't really matter -- all of the interesting stuff happens once you start getting closer and closer to it. You've gotten past all of the easy tokens that dominate the entropy and now you get to the really challenging ones that we care about (like e.g. very difficult reasoning about a next step).

3170702• 22 days ago

Right, and what happens at that limit is most exciting! A model that has a cross entropy at that limit for a data stream of text, produces a stream of text that is both theoretically and practically indistinguishable from the original stream.

And so if the datastream has been produced by something intelligent, the resulting model is indistinguishable from that intelligence. That is the whole compression idea behind artificial intelligence.

The limit is not a bug, it's a feature!

krackers• 21 days ago

Isn't that one of the reasons why KL-divergence is used, at least in DPO/RL for LLM? Otherwise the model can effectively cheat and mode collapse. For pre-training against a 1-hot label the KL-divergence should be equivalent to cross-entropy anyway.

an0malous• 22 days ago

Why should the skeptics be reading it? The scaling laws show diminishing returns on more training data and larger models.

From the Kaplan scaling laws paper:

We have observed consistent scalings of language model log-likelihood loss with non-embedding parameter count N, dataset size D, and optimized training computation Cmin, as encapsulated in Equations (1.5) and (1.6). Conversely, we find very weak dependence on many architectural and optimization hyperparameters. Since scalings with N,D,Cmin are power-laws, there are diminishing returns with increasing scale.

So the skeptics are right to be skeptical of LLMs being all you need for continued advancement in this space. It seems like the believers are the ones who need to learn about the scaling laws.

aspenmartin4• 21 days ago

We have observed consistent scalings of language model log-likelihood loss with non-embedding parameter count N, dataset size D, and optimized training computation Cmin, as encapsulated in Equations (1.5) and (1.6). Conversely, we find very weak dependence on many architectural and optimization hyperparameters. Since scalings with N,D,Cmin are power-laws, there are diminishing returns with increasing scale.

This is why, because this completely misses the point. Every power law has diminishing marginal returns, that is by construction. The point is that they diminish predictably and smoothly across seven orders of magnitude of compute. There is no plateau anywhere in the observed range, and in fact larger models more sample efficient than expected.

You can't just cherry pick one sentence containing the phrase "diminishing returns" and ignore the actual paper. The whole reason the labs spent the GDP of a small country on GPUs is that the curve is boring and reliable. No one just was like "oops we missed that line in the Kaplan paper, shit".

Lots of arguments as to why things may slow down or hit fundamental limits requiring a paradigm shift. But at several points in the past 5 years many major players have assumed this to be the case and have been continually proven wrong. Not to mention the plethora of socioeconomic horrors coming out of this Pandora's box, which I believe to be a large colocated B300 compute cluster somewhere in the midwest.

A point that would have been consistent with Kaplan was them finding alpha ~ 0.05 on compute means halving the reducible loss is a million times more expensive. But this was invalidated by Chinchilla who found it to be ~0.36 (100x more compute, not 1million). And loss has a non-linear relationship with capabilities, but roughly halving pretraining loss is ~doubling the METR task horizon. We also haven't factored in something incredibly important which is: algorithmic / architectural / training pipeline gains. These are substantial too!

gdiamos2• 22 days ago

When I first saw scaling laws in that deep speech experiment notebook, I didn’t believe it could be real. I was worried for months that we made a mistake, or that it only worked for that one dataset.

I started to believe it after we (Joel Hestness in particular) reproduced it in so many experiments in “scaling is predictable empirically”.

The OpenAI work replicated it in a completely different environment, and at that point I was sure it was real.

Sometimes people ask me why I was so surprised by it. Prior work like Banko and Brill and the unreasonable effectiveness of data argued for more data. ML theory had similar models for toy problems, eg coin flips.

At the time I thought deep learning was supposed to be complex. Speech and language datasets seemed much more complex than toy problems. Optimization of deep transformers was complex.

The idea that it was possible for the whole thing to be governed by a 3 term equation seemed too simple. The implication was that it was simple to manufacture intelligence.

Ten years later, I still think it is still the most interesting observation I have seen. We are still learning what it looks like to live in a world where it is possible to manufacture intelligence.

nok22kon• 22 days ago

the scaling laws work within a "generation". but what about across them?

GPT-3 was 175B, models like Gemma4 with 31B vastly outperform it, so there is more to it

as Karpathy noted, the initial GPTs were trained on complete garbage (literally, the average document from the Common Crawl is random nonsense), yet they worked. now we can use present LLMs to curate the data for the next generation

hnuser123456• 22 days ago

I dunno if you've seen the subreddit, "Sub Simulator GPT2", but I found it around 2020-2021. It seemed to contain GPT2-style models trained/finetuned on several popular subreddits, talking to each other as stereotypical regulars of each sub would. Most of the replies were fairly coherent and somewhat related to the "thread topic", but of course even GPT3.5 would make all of them look beyond drunk only a few years later. I already had a vague understanding of neural networks and the advances in image processing at the time, but couldn't have predicted where we are now. I wonder what it'll look like in a few more years as we continue how to learn how to make this capability useful and reliable, and hopefully sometimes keep finding additional conscionable entertainment and educational applications.

gdiamos2• 22 days ago

Scaling laws assume the error metric and data distribution.

There is a lot of follow on work that explains what happens as you change them, e.g. Scaling Laws for Transfer - https://arxiv.org/pdf/2102.01293

I think it’s fortunate that transfer works in a similar way.

Common crawl (and Reddit, stack overflow, etc but not 4chan) was much easier to get access to at the time than using mechanical Turk.

There is certainly room for more work. There were many papers on scaling laws in NeurIPS this year.

ekelsen• 22 days ago

Jeff Dean has a paper in 2007 that has proto scaling law plots for ngram language models.

https://aclanthology.org/anthology-files/anthology-files/pdf...

beyonddream• 22 days ago

Nice find! The final paragraph of the Conclusion is amazingly prescient!

"Significantly, we found that translation quality as indicated by BLEU score continues to improve with increasing language model size, at even the largest sizes considered. This finding underscores the value of being able to train and apply very large language models, and suggests that further performance gains may be had by pursuing this direction further."

Scaling Laws, Carefully

Related Stories