Best 7B LLM on leaderboards made by an amateur following a medium tutorial

https://huggingface.co/CultriX/MistralTrix-v1

by Der_Einzige • 3 years ago

181 49 3 years ago

huggingface.co

brcmthrowaway2• 3 years ago

Very interesting this was managed with a 6 months out of date course.

sour-taste• 3 years ago

On a free collab instance...

brcmthrowaway2• 3 years ago

How much more low hanging fruit is there?

SushiHippie• 3 years ago

The GPU he used is not free, you need to buy compute units for these "premium GPUs".

https://colab.research.google.com/signup

cjbprime• 3 years ago

Is it possible that they fine-tuned on the leaderboard test set?

nabakin• 3 years ago

The only leaderboard this model is good on is the HuggingFace LLM Leaderboard which is known to be manipulated and victim to gross overfit. The Lmsys Arena Leaderboard is a better representation of the best models.

refulgentis2• 3 years ago

Thank you!!! _And_ it has the proprietary models...insanely more useful.

It's a bug, not a feature, that the stock leaderboard ends up with endless fine tunes, and as you point out and is demonstrated by the article, its more about something else than about quality

Der_Einzige• 3 years ago

Even that chatbot arena shows that many models freely available and open source are better than some versions of GPT-3.5 and are within a small stones throw of the latest GPT 3.5

regularfry• 3 years ago

It's interesting that despite that, phi-2 is still way out in front of the 3B set on HF. I was convinced something would have caught up by now.

barnabee2• 3 years ago

If only all people and companies were as honest as this guy about how much of their success they owe to luck and others!

belter• 3 years ago

If only VCs would be as smart...

barnabee2• 3 years ago

And I thought I was being unrealistic

wavemode• 3 years ago

"Leaderboards", meh.

This tweet is still very true: https://twitter.com/karpathy/status/1737544497016578453

not2b• 3 years ago

Goodhart's law would seem to apply:

https://en.wikipedia.org/wiki/Goodhart%27s_law

Nevertheless, scoring so well on this benchmark is an accomplishment, though I'm not in a position to evaluate how significant it is.

make32• 3 years ago

that's why the huggingface llm arena exists

int_19h3• 3 years ago

Nothing beats an actual human spending a couple hours with the model when it comes to meaningful evaluation.

jasonjmcghee• 3 years ago

Why does huggingface list this as a 9B model?

make32• 3 years ago

it's trained with a lora adaptor, so it's either an error or they also count the adaptor. they use a 16 param inner lora dimension however, so it's unlikely that that's the reason (too small)

an important point to keep in mind is that at inference, the lora adaptors are made to be merged into the base model so they don't affect inference speed. (you need to explicitly do it though, if you train your own adaptor)

latexr• 3 years ago

Are we sure the tutorial was medium? It might have been quite good, or at least above average. Ba dum tss.

“medium” should be capitalised in the title, as it refers to the blogging platform.

https://medium.com

ben_w• 3 years ago

Agreed, because I initially read it as "mediocre" rather than the brand.

idorube• 3 years ago

And here I thought he was helped by a crystal ball...

politelemon• 3 years ago

Good thread I saw on Reddit about this a few days ago.

https://www.reddit.com/r/LocalLLaMA/comments/18xbevs/open_ll...

Many top models are overfitting to top leaderboards rather than be actually useful.

SubiculumCode3• 3 years ago

It seems so easy for just one poisoned model (trained with test data) to infect a ton of finetune model mixtures...it could happen without intention.

Under this scenario, the ones that achieve the top performance are the closest relation to the poison model?

mbb70• 3 years ago

I'm as quick to jump on the Medium roastwagon as anyone else, but I will say Towards Data Science has a surprising number of quality tutorials running the full spectrum of data science tasks.

That and they have great SEO, you basically can't avoid them.

behnamoh• 3 years ago

I avoid them easily using Kagi :))

rck• 3 years ago

Everyone knows the hf leaderboard is actively being gamed (Goodhart's law strikes again), but the guy who wrote the Medium post is active in doing stuff with models, and the tutorial is (clearly) pretty good.

josu• 3 years ago

Well, "Amateur following tutorial games HF leaderboard to become the #1 model" seems just as impressive to me.

SubiculumCode3• 3 years ago

It seems that a lot of the leaders are the results of mixing finetunes, which really makes me think that there was a leak of test sets into the training data.

SubiculumCode3• 3 years ago

To reply to myself. I am not saying that this model did, or that even if it did, that it was done intentionally. ML is hard, and there are so many ways for data to leak.

What I AM surprised about is that it is not clear what CultriX did that was better than what a ton of others have done.

Any clues?

ramoz3• 3 years ago

Who is using 7Bs in a serious manner, instead of OpenAI, in a cost efficient way?

justinl33• 3 years ago

1. fine tune to specific tasks 2. Not subject to OpenAI’s censorshi 3. Can run on local instead of cloud compute (offline) 4. experimentation

ramoz3• 3 years ago

Not sure what you’re responding to

dingdingdang• 3 years ago

Using Mistral ft optimized 1218 7b q5km - very useful for basic queries/creative input. Often just as useful as ChatGPT and feels far more "real" to have it fully local, don't want to depend fully on one proprietary service for something as fundamentally useful as this!

ukuina2• 3 years ago

I have spent more time than I would like to admit trying to get 3B-34B models working for "serious" usecases like RAG, code comprehension/generation and text summarization, across dozens of "leading" free LLMs, and none of them (not even Mixtral MoE or its derivatives) come close to gpt-3.5-turbo for consistency and depth, and there is no equal for gpt-4(-turbo).

LLM Leaderboards and benchmarks do not show the complete picture for YOUR specific usecase.

dcreater2• 3 years ago

If you can post results that would be extremely extremely helpful. This is what I've been looking for -real world performance - rather than gamed leaderboards

ramoz3• 3 years ago

And this is my drawn out experience as well

reexpressionist• 3 years ago

If the end goal is document classification and/or semantic search, the Reexpress Fast I model (3.2 billion parameters) is a good choice. The key is that it produces reliable uncertainty estimates (for classification), so you know if you need a larger (or alternative) model. (In fact, an argument can be made that since the other models don't produce such uncertainty estimates, they are not ideal for serious use cases without adding an additional mechanism, such as ensembling with the Reexpress model.)

Havoc• 3 years ago

Nobody really trusts the leaderboards anymore. Overfitted and in some cases just outright cheating by training against common test sets

https://twitter.com/karpathy/status/1737544497016578453

Best 7B LLM on leaderboards made by an amateur following a medium tutorial

Related Stories