Solar-10B

https://twitter.com/_philschmid/status/1734992933764411788

by tosh • 3 years ago

29 11 3 years ago

I know the hugging face leaderboard isn't wildly accurate.

But the top models right now are almost all under 70B. Most are 7B, and the top is 10B. If the benchmarks are even remotely accurate then this is rather wild.

Apparently multiple groups found different "secret sauces", names upstage and whatever UNA is?

appplication• 3 years ago

I mean this isn’t too surprising that smaller models do better. I imagine transformers are as prone to overfitting as any statistical data model. Also there is probably some selection bias: bigger models are more expensive and there are just less people training and iterating with them

spott• 3 years ago

There are orders of magnitude fewer people playing with large (>40B) parameter models than the small ones, which means even fewer people finetuning those models.

I can’t imagine this is anything but selection bias.

thot_experiment2• 3 years ago

I suspect we're seeing Goodhart's Law in action here combined with a multiplier based on hardware gating, similar to what's happened with Stable Diffusion finetunes. There are some fantastic 1.5 finetunes if you're seeking to gen a particular style/subject but these finetunes are horribly overfitted and fall flat when used outside of their specific niche (big tiddy anime girl most often). These models come into being because the barrier to entry is low and the model is focused on a narrow set of metrics. This is the reason we see smaller models "outperforming" larger ones.

I have tried a lot of these LLMs locally and I have rarely ever found the smaller models to outperform the bigger models when used casually/generically. The same is generally true for 1.5 vs SDXL the moment the prompt strays away from some variant of beautiful woman.

22c2• 3 years ago

I agree for the most part, but there is also the other side which is how useful it can be. If you fine tune your model on anime girls, it becomes a very useful model for generating anime girls and perhaps it's now worse at drawing stick figures or cave paintings, but a lot of people don't need the model to draw stick figures or cave paintings and if they did, they could get a model fine tuned to do that.

On the LLM side, there's a similar phenomenon where a huge amount of training could be on SEO spam tags, or /r/counting or producing nonsense or generating obfuscated C or reciting excerpts of Shakespeare word for word, etc.

If what you're measuring is generally more useful then you can end up with a better model using these methods, likely at the cost worse performance on things that you aren't measuring.

I would gladly take a 10b parameter coding assistant that doesn't know how to write in iambic pentameter or recite digits of pi, or translate words from Swahili to Turkish etc. but is much better at code completion.

thot_experiment2• 3 years ago

Totally valid points! However I think a lot of the real "magic" comes from the sort of cross-domain "thinking" that the larger models are capable of and I think that axis hard to benchmark. (though because it seems so hard to quantify, there's a high chance i'm making shit up)

DrSiemer• 3 years ago

Is there a reason all these language models rarely have any indication of the amount of GPU ram required to run them?

wsgeorge• 3 years ago

I would guess that a) they assume their users will have a lot of GPU ram, b) actual running costs will depend on what inference engine/framework you're using (for example, some GGUF quants are very cheap).

Reubend• 3 years ago

It's because you can quantize to different extents, at the expense of a small amount of accuracy. The original model probably needs much more vRAM than most people will actually end up needing in prod.

Solar-10B

Related Stories