The tide is shifting: 1.3B outperforms 7B Llama 2

https://arxiv.org/abs/2309.05463

by __vec__ • 3 years ago

63 15 3 years ago

arxiv.org

__vec__• 3 years ago

Textbooks Are All You Need II: phi-1.5 technical report

"Perhaps achieving ChatGPT’s level of capability at the one billion parameters scale is actually achievable?"

refulgentis• 3 years ago

On a long enough timeline, Id say yes.

Right now they're comparing to the 7B Llama 2, which is a shadow of 65B llama2, which is a couple steps off ChatGPT, which is a shadow of GPT4.

I'm still comfy saying yes because there's no reason to doubt it will follow the same logic as "eventually $phone will have the same FLOPS as $desktop"

brucethemoose22• 3 years ago

we are seeing improvement on that front thanks to the absence of web data

Bingo. This (not the parameter count) is the amazing thing to me.

Garbage in garbage out, and there is a ton of garbage in the Falcon/Llama (and OpenAI?) datasets. It feels like such a waste of compute and parameter space.

tayo422• 3 years ago

it might not exactly be a waste, when you are inputting your training data you need some context on what the source is and then add a weight to how valid it is. That is how we learn, if i talk to a phd on some topic, I would pay close attention to what they say and give it more significance, maybe even write it down, compared to a youtube video on that topic, compared to a guy on a street talking about that same topic, ill probably nod my head and walk away and try to forget as soon as i can. but there probably still is something valid to learn from that last conversation, how a certain type of person structures sentences, responds, the words used etc...

do these llms get trained with something like a credibility weight on the training data? that was it seems they did in this paper, just manually curated that

mpixel• 3 years ago

Do people want that though? I don't want phd level responses for my queries. I want it to be better than what I could come up in a minute or by searching half an hour. Rather than some highly advanced highly detailed response I could probably not understand if the topic is not something I'm sufficient in to begin with.

Think common use cases. A lot of users are students, do I want it to write an essay like a linguist? Or solve my homework using the better but more advanced techniques and style?

famouswaffles• 3 years ago

Garbage in garbage out, and there is a ton of garbage in the Falcon/Llama (and OpenAI?) datasets. It feels like such a waste of compute and parameter space.

This is true and most researchers understand this on some level (https://arxiv.org/abs/2305.07759) but understanding it doesn't really make the problem any easier. How do you curate a general purpose model's dataset without throwing the baby out with the bathwater ?

EDIT: This paper seems to take a good stab at the question. https://arxiv.org/abs/2309.04564

brucethemoose22• 3 years ago

How do you curate a general purpose model's dataset without throwing the baby out with the bathwater

Time, money, and human work. The pruning process needs focused input from experts other than ML researchers and data scientists.

waste_monk• 3 years ago

How do you curate a general purpose model's dataset without throwing the baby out with the bathwater ?

Maybe we could train an AI to do it ;)

nbardy• 3 years ago

The title is clickbait and not the title of the paper.

YetAnotherNick• 3 years ago

Llama generally acheives much higher accuracy with very small amount of fine-tuning(on similar quality of dataset like this paper) on lot of tasks. So the model understanding is present in llama to get higher accuracy. e.g Hellaswag, ARC and MMLU for 7b model is 0.8, 0.57 and 0.52 respectively[0], while phi-1 is 0.48, 0.45 and 0.38.

I don't think finetuning phi-1 on good quality synthetic data will increase its accuracy as it is only trained on that.

[0] https://huggingface.co/pankajmathur/orca_mini_v3_7b

matteoraso• 3 years ago

The model can be downloaded here: https://huggingface.co/microsoft/phi-1_5

The tide is shifting: 1.3B outperforms 7B Llama 2

Related Stories