Llama 4 Smells Bad

https://fastml.com/llama-4-smells-bad/

by alexmolas • 1 year ago

41 26 1 year ago

fastml.com

croisillon• 1 year ago

did Meta open a time wormhole to release Llama 4 on May 5th?

GaggiX• 1 year ago

GPT-4o-mini, which is, according to itself, a model with 1.3B or 1.5B or 1.7B parameters

I have no idea how the author can remotely trust GPT-4o-mini in this case. The number of parameters is almost certainly way off.

simonw5• 1 year ago

This article's credibility suffers a little from the way it talks about GPT-4o mini:

"just in front of GPT-4o-mini, which is, according to itself, a model with 1.3B or 1.5B or 1.7B parameters, depending on when you ask."

Then later:

"On the Artificial Analysis benchmark Scout achieved the same score as GPT 4o mini. A 109B model vs a 1.5B model (allegedly). This is ABYSMAL."

Asking models how many parameters they have doesn't make sense.

There is absolutely no way GPT-4o mini is 1.5B. I can run a 3B model on my iPhone, but it's a fraction of the utility of GPT-4o mini.

fancyfredbot2• 1 year ago

It's strange because there's no need to make this assumption about GPT-4o in order to demonstrate their point.

gliptic2• 1 year ago

It's strange that someone from FastML can be confused about this, unless it's supposed to be a bad joke.

lhl2• 1 year ago

This is just someone's personal blog/opinion. I wouldn't read too much into it... "The site is run by Zygmunt Zajc (pronounced “Ziontz”). ... An economist by education"

PunchTornado• 1 year ago

we are in for a lot of pain if seemingly intelligent people make mistakes like this. grabbing the number of params from what gpt gives you. how can you do that?

jychang3• 1 year ago

Correct, in that models know nothing about themselves other than what they are told. Deepseek R1 will tell you that it's created by OpenAI.

GPT-4o mini is supposed to be ~8b params from estimates.

simonw5• 1 year ago

The best source I've seen for the 8B number is this TechCrunch article: https://techcrunch.com/2024/07/18/openai-unveils-gpt-4o-mini...

OpenAI would not disclose exactly how large GPT-4o mini is, but said it’s roughly in the same tier as other small AI models, such as Llama 3 8b, Claude Haiku and Gemini 1.5 Flash.

As far as I can tell all of the 8B rumors were seeded by that loosely sourced comparison to Llama 3 8B.

I know for a fact that Gemini 1.5 Flash is NOT an 8B model, because a separate model called "Gemini 1.5 Flash 8B" came out after that article was published - the first Gemini model with a documented parameter count. Flash 8B is priced at half the cost of regular Flash.

There's also this paper that mentions 8B but provides no source for that at all, which makes me suspect their initial source may have been that TechCrunch rumor: https://arxiv.org/pdf/2412.19260

gwd• 1 year ago

The comment at the top says it's a draft. It's not unreasonable to ask for random values from a GPT for "filler" for the draft (or even just make them up), just to stay in the flow, and then track down the real numbers later.

NitpickLawyer• 1 year ago

I flagged it for these reasons as well. It's just a bad article. Shows very poor understanding of the basics of LLM workings, and the field in general.

Lingers on the "cheated" benchmark (lmsys) but never mentions all the other 3rd party benchmarks performed after the inference fixes, which are in line with what Meta originally published. To be clear, submitting a different fine-tuned model to one arena and releasing the untuned model without clearly mentioning this, is bad. But conflating the "human prefference" bench with the others and not mentioning the models capabilities on other benchmarks is also bad writing.

The MoE paragraphs are bad, and the writer never explains why the copy 17B vs VRAM size is bad, they just leave it there unexplained.

Poor form, I was expecting better from someone working in this field.

bradley13• 1 year ago

This seems to be a general problem at the moment. The most usable models are not the newest. The newer models (obviously, I haven't tried them all) may do better on benchmarks, but actual usability is worse.

To create useful LLMs required some genuine breakthroughs. It seems to me that we have reached the limits of what we can achieve with current architectures. Progress will require some new insights and breakthroughs.

fancyfredbot2• 1 year ago

If you game the benchmark then you always get found out by your users. Yet the practice remains common in hardware. Outright lies are uncommon but misleading and cherry picked numbers are pretty much standard practice.

The fact that misleading benchmarks don't even drive profit at Meta didn't seem to stop them doing the same thing, but perhaps this isn't very surprising. I imagine internal incentives are very similar.

Unlike the hardware companies though, gaming the benchmark in LLMs seems to involve making the actual performance worse, so perhaps there is more hope that the practice will fade away in this market.

danielhanchen• 1 year ago

There were actually multiple bugs which impacted long context benchmarks and general inference - I helped fix some of them.

1. RMS norm eps was 1e-6, but should be 1e-5 - see https://github.com/huggingface/transformers/pull/37418

2. Llama 4 Scout changed RoPE settings after release - conversion script for llama.cpp had to be fixed. See https://github.com/ggml-org/llama.cpp/pull/12889

3. vLLM and the Llama 4 team found QK Norm was normalizing across entire Q & K which was wrong - accuracy increased by 2%. See https://github.com/vllm-project/vllm/pull/16311

If you see https://x.com/WolframRvnwlf/status/1909735579564331016 - the GGUFs I uploaded for Scout actually did better than inference providers by +~5% on MMLU Pro. https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-... has more details

jychang3• 1 year ago

Do you think there are more bugs in Llama 4 at this time? Or have the bugs been patched, and the current version of llama.cpp + whatever the latest GGUF version is would be representative of the true performance of Llama 4?

I see you've uploaded new Maverick GGUF/safetensors files yesterday, along with a lot of other models like Deepseek R1, was there an issue with the older model files?

simonw5• 1 year ago

If they hadn't rushed out a release on a Saturday their launch partners might have had more time to iron out those bugs!

pixelesque• 1 year ago

This is a draft. Come back later for the final version.

There are quite a few issues with the content from a factual point-of-view (several sibling comments mention things): could have done with a lot more proof-reading and research I think.

elaus• 1 year ago

I don't understand the rationale for publishing an article in such an early draft stage, even with the small disclaimer at the top. It would make sense if only a bit of polish were missing, but when there are factual errors (that are not marked as such), it seems much better to delay publication until the content is correct.

NanoYohaneTSU• 1 year ago

Reminder that 1 year ago, AI tech bronies were saying that AI is only going to improve from here. It didn't. It stagnated because it's reached the peak of LLMs, as predicted.

And it still can't create images correctly, as in actual image creation, not woven pixels with tons of artifacts.

simonw5• 1 year ago

If you think LLMs haven't improved in the last 12 months you haven't been paying attention.

Image creation has been mostly a separate conversation, although GPT-4o images dramatically improved the state of the art for consistency in editing existing images just a few weeks ago.

simonw5• 1 year ago

The initial Llama 4 release is disappointing: the models are too big for most people to run, and not high quality enough be worth running if you can afford the hardware.

I'm still optimistic for Llama 4.1 and 4.2.

Llama 3 got exciting at the 3.2 and 3.3 stages: smaller models that were distilled from the big ones and ran on a laptop (or even a phone).

3.2 3B and 3.3 70B were really interesting models.

I'm hopeful that we will get a Llama 4 ~25B, since that seems to be a sweet spot for laptop models right now - Gemma 3 27B and Mistral Small 3.1 (24B) are both fantastic.

bambax• 1 year ago

Anyway, on Saturday (!) May the 5th, Cinco de Mayo, Meta released Llama 4

Wat. We're still in April. Cinco de Abril.

lhl2• 1 year ago

While Llama 4 had a pretty bad launch (the LM Arena gaming in particular is terrible), having run my own evals on it (using the April 5 v0.8.3 vLLM release - https://blog.vllm.ai/2025/04/05/llama4.html , so before the QKNorm fix https://github.com/vllm-project/vllm/pull/16311) - it seemed pretty decent to me.

For English, on a combination of MixEval, LiveBench, IFEval, and EvalPlus Maverick FP8 (17B/400B) was about on par with DeepSeek V3 FP8 (37B/671B) and Scout (17B/109B) was punching in the ballpark of Gemma 3 27B, but not too far off Llama 3.3 70B and Mistral Large 2411 (123B).

Llama 4 claimed to be trained on 10X more multilingual tokens than Llama 3 and testing on Japanese (including with some new, currently unreleased evals) the models did perform better than Llama 3 (although I'd characterize their overall Japanese performance as "middle of the pack": https://shisa.ai/posts/llama4-japanese-performance/

I think a big part of the negative reaction is that in terms of memory footprint, Llama 4 looks more built for Meta (large scale inference provider) than home users, although with the move to APUs and more efficient CPU offloading, there's still something to be said for strong capabilities at 17B of inference.

I think people are quick to forget that Llama 3, while not so disastrous, was much improved with 3.1. Also the competitive landscape is pretty different now. And I think the visual capabilities are being a bit slept upon, but I think that's also the case of releasing before the inference code was baked...

anonymousiam• 1 year ago

[trying to confuse an android]

Spock: Logic is a little tweeting bird chirping in a meadow. Logic is a wreath of pretty flowers which smell bad. Are you sure your circuits are registering correctly? Your ears are green.

https://www.imdb.com/title/tt0708432/quotes/?item=qt0406609

Llama 4 Smells Bad

Related Stories