Mistral-7B-OpenOrca. First 7B model to beat all other models <30B

https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca

by guybedo • 3 years ago

21 12 3 years ago

easygenes3• 3 years ago

This new release is a finetune of Mistral-7B base model. They claim it achieves 98% of the eval performance of Llama2-70B.

easygenes3• 3 years ago

Having tested this model a lot today, I can say that it indeed in a lot of ways feels much like a 70B model, much to my surprise.

One thing that really struck me: I asked it details about my local suburbs (minor cities in a larger metro). I have tried this on all different Llamas, and only the 70B models don't just hallucinate everything or tell me they don't know anything. In contrast, this model knows quite a bit about my local area. Not as good as Llama2-70B, but not incredibly far off.

brucethemoose23• 3 years ago

Though it does seem to he very coherent, I am still very suspicous of test data contamination in Mistral.

XWin was the SOTA llama finetune last I checked, so we will see how it fares with Mistral...

easygenes3• 3 years ago

XWin is good for AlpacaEval, but was optimized specifically for that. Evaluated more broadly, there are other better choices.

squigz3• 3 years ago

Could you elaborate what you mean by "test data contamination" and why that's a specific concern for these models vs others?

brucethemoose23• 3 years ago

LLMs are evaluated by asking them a set of standard test questions and seeing how they respond.

But they are not supposed to train on these test questions. If they do train on those test Q/A pairs, intentionally or not, the answers will be conspicuously good. A purposely seperate test dataset like that is standard in basically all ML research.

But Mistral has given basically zero information about the Mistral 7B training dataset. And they have less reputation to lose from "cheating" than Meta does.

brianjking• 3 years ago

Are there any quantized versions yet? https://huggingface.co/s3nh/Open-Orca-Mistral-7B-OpenOrca-GG... doesn't seem to have any weights uploaded.

Secondly, is this instruction tuned?

SushiHippie• 3 years ago

GGUF: https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF GPTQ: https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GPTQ AWQ: https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-AWQ

Thebloke is really really fast, he quantized and uploaded them already yesterday.

Mistral-7B-OpenOrca. First 7B model to beat all other models <30B

Related Stories