EagleX 1.7T: Soaring past LLaMA 7B 2T in both English and Multi-lang evals

https://substack.recursal.ai/p/eaglex-17t-soaring-past-llama-7b

by lhofer • 2 years ago

36 9 2 years ago

substack.recursal.ai

LorenDB• 2 years ago

To be clear, this is a 7B model. It's just trained on 1.7 trillion tokens. At first I was confused why they were making such a big deal of of a massive 1.7T model outperforming a 7B model.

By the way, GPT-4 has been predicted to be using a 1.7T model, although OpenAI has not confirmed this to my knowledge.

mmoskal• 2 years ago

The most interesting bit is that this is RWKV model, meaning constant size state (no quadratic attention). AFAIK the biggest open weights non-transformer model.

nhggfu• 2 years ago

sounds dreamy. anyone know how I can install this on my m1 mac?

YetAnotherNick• 2 years ago

All evals are 0 shot

My bet is that this is the reason they are scoring high in "their" benchmarks. For model which are just trained on completely unlabelled data like llama, 0 shot won't work well.

e.g. For llama Hellaswag accuracy is 57.13% in their benchmark compared to 78.59% in [1].

[1] https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

veselin• 2 years ago

I think this is simply the default of lm-evaluation-harness. They said they ran every single benchmark they could out of the box.

GaggiX• 2 years ago

Let’s look what went really badly: Math.
We dug through the dataset we used for training, and realized we missed out the entire math dataset (along with a few others) due to an error. Oops.

This is kinda hilarious.

turnsout• 2 years ago

Makes me excited for the next model though!

squigz• 2 years ago

All data shown here is made available in the Google Sheet over here:

Over where?

viraptor• 2 years ago

[June 2024] v6 MoE model

How does that work for RWKV architecture? Wouldn't you have to feed the same data through all the experts regardless if they're currently active or not to keep the rolling state consistent? Or am I misunderstanding that architecture?

EagleX 1.7T: Soaring past LLaMA 7B 2T in both English and Multi-lang evals

Related Stories