Has Llama-3 just killed proprietary AI models?

https://www.kadoa.com/blog/llama3-killed-proprietary-models

by hubraumhugo • 2 years ago

103 70 2 years ago

jsheard3• 2 years ago

Perhaps for now, but I wouldn't count on there always being a Meta spending $$$ to train enormous models and then giving them away for free. What's the long-term game plan for open-source models when the corporate charity inevitably dries up?

mepian• 2 years ago

Maybe they are commoditizing their complement here.

jsheard3• 2 years ago

Maybe, but at some point vague strategic moves like that will need to actually justify themselves at the earnings call. Throwing endless millions of dollars down the drain to what, depreciate the value of another companies product, which doesn't even directly compete with Metas main breadwinners? What do they actually get out of this?

I suppose if there's any consolation for the open-source AI community it's that Meta has demonstrated a willingness to burn a lot of money for unclear benefit, they're still single-handedly keeping the VR industry on life support at the expense of about $4 billion per quarter. A decade on from acquiring Oculus and no closer to making it profitable, if Metas AI efforts get the same treatment then the free models will probably keep coming for a while.

sharkjacobs• 2 years ago

Unless Meta’s bet is that, going forward, models themselves won’t be the competitive differentiator, that it will be about integration. They can give away Llama3,4,5 for free, because no one else can put them Whatsapp or whatever.

idk

jeffnappi• 2 years ago

Building open models is a very strong approach to cornering the market on top tier AI researchers. And as other commenters have mentioned, the raw models are not the product - the vast majority of the value is in how they are integrated into useful products.

_giorgio_• 2 years ago

Exactly, a free standalone AI is an unsustainable product.

You can't offer it for free, unless you make money in another way.

ozr2• 2 years ago

If your product is an AI model (OpenAI, Anthropic, etc) you can't give it away for free.

If your product is a social graph w/ ads (Meta), you can.

It's hardly corporate charity:

* Meta releasing these models creates an improvement and tuning ecosystem around it, giving them access to tons of free developer time.

* It's also a strong recruiting tool, for engineers and researchers frustrated by, e.g., Google and OpenAI becoming increasingly closed. They know they can publish at Meta.

* The cost is insignificant. Meta had 30B in revenue just in Q2 2023.

faeriechangling5• 2 years ago

It's great PR. I hear people refer to Mark Zuckerberg as a "real engineer" and other such platitudes and it's doing wonders to reduce the stigma around Meta amongst the tech savvy minority.

Despite Meta being very similar to Google in terms of incentives, and Google nowadays being decidedly uncool. Doesn't hurt that Mark shipped something worth a damn whereas Google has been floundering for ages.

drakenot• 2 years ago

Commoditize your complement and all that.

seydor• 2 years ago

They spent much more on metaverse

polio• 2 years ago

The corporate charity will not dry up. AI makes it easier to generate content, and Meta's in the business of facilitating the sharing of that content. Content is surface area for ads. AI will also make the virtual realities of the "metaverse", as defined by Mark, easier to reify. It's also a giant marketing and recruiting strategy.

jsheard3• 2 years ago

AI does make it easier to generate content, but the type of content it lends itself towards the most is spam. Whether a Facebook where the majority of the content is neural net slurry is something that people will want to engage with once they realise what's going on is an open question I think.

Anecdotally it seems like older demographics are the prime target of the current wave of AI engagement farming on Facebook, because they just don't understand that this technology exists now and assume that all of the "photos" they're seeing are real.

lunfardl• 2 years ago

I guess many here can think a lot 5D chess business strategies. For me it is just Zuck trying to reach greatness, down in history his name will prob pop up when people think about who brought LLM/AI/AGI? to the masses.

dragonwriter2• 2 years ago

What’s the long-term game plan for open-source models when the corporate charity inevitably dries up?

Once open models reach and stay at near parity for a while, it’ll make sense for commercial downstream users to support open source community efforts rather than building their own, same as has happened in many other categories of key infrastructure software.

airstrike2• 2 years ago

Meta released Llama-3 only three days ago, and it already feels like the inflection point when open source models finally closed the gap with proprietary models. The benchmarks show that Llama-3 70B matches GPT-4 and Claude Opus in most tasks, and the even more powerful Llama-3 400B+ model is still training.

I'm all for open models, but where do the benchmarks show that?

The official page https://llama.meta.com/llama3/ does not show any comparisons with GPT-4 or Claude Opus

Looking at https://arena.lmsys.org/, Llama-3-70b-Instruct is ranked #5 while current GPT-4 models and Claude Opus are still tied at #1. Meanwhile, Llama-3-8b-Instruct is ranked #14

Would love to be corrected, but either way an article should include sources for these types of claims.

afro88• 2 years ago

Came here to say this. Thought I missed something!

400B does look set to meet GPT-4, which will be exciting, but it's not finished yet.

faeriechangling5• 2 years ago

OpenAI is also a moving target and Llama3 is probably due to get a few months of parity before GPT5 wallops it.

To me, it looks like OpenAI is ahead of the competition by 12-18 months, which isn't nothing but the competition is certainly nipping at their heels.

willsmith72• 2 years ago

Yeah I came to post the same thing. It's nothing like gpt4, just great for the size.

natrys• 2 years ago

It's above Opus in second position in the "English" only category. It probably suffers in the overall score due to poor multilingual ability (afaik 95% of its training data was English only).

Though usual caveat about small sample size applies, as of now the CI is fairly big. It's also not at the level of those two in "Code" category, I hope Meta will give the CodeLlama variant an update again.

rb2k_• 2 years ago

Some Llama3 400B numbers for the April 15th checkpoint are listed at https://ai.meta.com/blog/meta-llama-3/

Direct image link: https://scontent-atl3-2.xx.fbcdn.net/v/t39.2365-6/439015366_...)

gtt• 2 years ago

How significant is the difference in Elo scores for the end user? Should I use Claude Opus if I already have GPT4, despite its similar score?

Additionally, how much better is Claude for coding?

floydnoel• 2 years ago

I use Poe.com, which gives me access to every model available for $20. When a new model comes out it is quickly added to the available list. So, I've been doing some comparison against Claude & GPT4. I still think GPT4 is better at following instructions and giving me the result I'm after.

debacle• 2 years ago

Not yet, but I imagine at some point it will. Open source, much like the oft maligned Thanos, is inevitable.

geor9e• 2 years ago

If they did or didn't is up for debate, but I'm pretty sure scorched earth was their goal from the start (specifically targeting the giants, Google and OpenAI). They don't want to be #3 in this race.

paxys• 2 years ago

Every big new tech category in recent years has quickly matured into a duopoly of a closed, proprietary, premium option and a mass market, public, "open" one. OpenAI, Anthropic, Google, Amazon, Microsoft and a ton of other players are battling for the #1 option, but Meta has cleverly theorized that #2 is going to be easier and a better fit for them.

pfdietz• 2 years ago

I think it's not just big categories of software, although I'm sure there are niches with no OS option.

echelon• 2 years ago

It's the smartest possible play.

The leader always wants to be a monopoly. The distant runners up can catch up the most ground by playing nice, being open source, and working with other companies to erode the market share that the leader wanted to lock away.

I'm so glad to see this happening. I'm terrified of a single company winning all of AI. It's starting to look like this won't happen and that OpenAI is simply stretched too thin.

If this pattern holds, OpenAI may wind up as a footnote. Their inability to open up and work with others means that competitors and would-be collaborators will choose the open alternatives.

Meta can win that mindshare and be the friendly facilitator and rails that an entire ecosystem of business is built upon. OpenAI will never be that. They're not "open" enough.

prpl• 2 years ago

There was a clear leader 18 months ago. Then there was a lot of catchup by everyone.

It’s no longer “hard” build a product like ChatGPT + GPT 3.5, and that would include creating the model. There’s a few trade secrets, but it seems like not much beyond that protecting an ~8 month mote.

cpeterso• 2 years ago

Open models weren’t Meta’s original plan. The LLaMA 1 model was only available to Meta-approved researchers until someone leaked the model weights on 4chan in 2023. Meta issued DMCA takedown requests to HuggingFace and GitHub.

ronsor• 2 years ago

Something something law of headlines... No. As much as I'd like for this to be the case, it's not.

I think proprietary models will gradually become less popular, though, due to their lack of consistency and control.

greggsy• 2 years ago

I was hoping for more detail in the post but it really just a seems like a quick and easy way to attract eyeballs? Never heard of the platform, and I cant help but think that the logo is a rip of the old okta brand.

drivingmenuts2• 2 years ago

As long as businesses have proprietary processes, there will be proprietary models. An open-source AI cannot know everything.

ladzoppelin• 2 years ago

So they are doing the "bait and switch" like Google did for market share and in the end Facebook controls the "best" model? This sounds horrible.

thorum• 2 years ago

GPT-4 and Opus are better for complex/precision tasks, and Haiku is cheaper for everything else.

Good 7B/8B models are still really useful but let’s not be hyperbolic.

rrr_oh_man• 2 years ago

«Any headline that ends in a question mark can be answered by the word no.»

— Betteridge's law of headlines (https://w.wiki/3b$V)

rmellow• 2 years ago

Betteridge's corollary:

As an online discussion about a headline that ends in a question mark grows longer, the probability of a citation of Betteridges's law approaches 1.

QuantumGood• 2 years ago

In the same domain as Godwin's law

"As an online discussion grows longer, the probability of a comparison involving Nazis or Hitler approaches 1."

franze• 2 years ago

nextworddev• 2 years ago

This article will age poorly when GPT-5 is released /s There will be always a market for proprietary models that is the best, though it’s unclear whether that will be OpenAI’s

mushufasa• 2 years ago

The brilliance of Meta's strategy here is: if they offer a (F?)OSS model at near-performance compared to leaders, they commoditize the product of their would-be competitors. Meta doesn't have to make money on API calls, but they could face an existential risk if someone else built the everyday AI companion of the future (e.g. users on ChatGPT UI, microsfoft's much advertised "copilot for everyday").

So -- a defensive play with some positive externalities (e.g. developer ecosystem mindshare + roadmap control, ability to use within their own products at cost, without giving up margin to suppliers).

bagels• 2 years ago

It can also power features in Instagram, etc.

jauntywundrkind• 2 years ago

Llama-3 is semi-proprietary though. It's definitely not open source!!

Has anything changed in the last 9 months: https://news.ycombinator.com/item?id=36815255 ? Is there better access to anything more than weights? Can we now train new models using llama? Are we no longer as restricted in use?

Haven't seen any comments questioning the premise here. It seems pretty constrained what we can do and how built-atoppable Llama is. I like the idea of it as a safeguard against control by some giant but I'm not sure if it's a big enough grant of rights to be something we can built atop.

comex• 2 years ago

Llama 3 is just as restricted as Llama 2. The licenses and acceptable use policies are almost identical; the only real change consists of added attribution requirements. Products that use Llama 3 have to "prominently display 'Built with Meta Llama 3' on a related website, user interface, blogpost, about page, or product documentation", and models based on Llama 3 have to "include 'Llama 3' at the beginning of any such AI model name."

pointlessone• 2 years ago

8k context is tiny compared to what’s out there. They promise much larger contexts but until then it can’t even reliably summarize every web page out there.

tarruda• 2 years ago

There are techniques to extend the context window via fine tuning.

The authors claim this method was used to extend Llama 2 to 128k: https://github.com/jquesnelle/yarn

mehulashah• 2 years ago

Commoditization of the AGI models is inevitable. OpenAI is building the next generation of compound AI systems — an LLM OS, if you will. it does more than just predict the next token. It figures out the right function and service to call when needed. That’s where the arms race has moved to. Meta isn’t going to build a free LLM OS. The comparison is apples to oranges.

segmondy• 2 years ago

It hasn't. My guess is that 90% of the folks using proprietary models through chat interface have no need or wants to run their own locally nor do they have the hardware. For those of us who wish to run our models and do have the hardware, I would reckon maybe 20% of us using proprietary models will drop them. The latest models released are very good, but I'm not convinced they are GPT/Claude quality yet.

Havoc• 2 years ago

Wouldn't say killed, but for certain usage cases definitely provided a strong alternative.

The 8B model seems particularly good at summarization tasks.

EcommerceFlow2• 2 years ago

GPT4 was released a year ago, and trained two years ago. I wouldn’t sleep on OpenAi, especially how confident Sam Altman sounded in his Lex Friedman podcast.

The real race is to AGI anyways, as whoever gets there first will immediately capture 100% of the market.

dragonwriter2• 2 years ago

I wouldn’t sleep on OpenAi, especially how confident Sam Altman sounded in his Lex Friedman podcast.

How confident Sam Altman sounds doesn’t figure much into my assessment’s of reality, other than the reality of Sam Altman’s promotional skills.

The real race is to AGI anyways, as whoever gets there first will immediately capture 100% of the market.

AGI has no actual objective definition, and nothing supports this beyond naked conjecture and quasi-religious dogma.

EcommerceFlow2• 2 years ago

Im just reluctant to jump to conclusions until I see OpenAi’s next hand.

True on AGI, but I guess I meant a “self improvement” Ai? Whether that exists or is a pipe dream is to be seen, but seems like the goal of most these companies.

barfbagginus• 2 years ago

You would be wrong to accept this hype anyways.

Llama 3 is not beating GPT4 in benchmarks I've seen, and it's not beating it on LLM Arena. That's all that really matters. It needs to beat GPT4 in the benchmarks and leader boards, or it's a nothingburger, as far as OpenAI's dominance goes.

nojvek• 2 years ago

After Apple’s ad tracking blocking, Meta was worth $500B. After they announced they are full in on AI and want to be too they are now worth $1.2T.

So even if they are losing a bit on infra and training investment, Meta is relevant again. They are cool again.

Meta made a huge comeback.

colinng• 2 years ago

I tested Llama-3-70b-8192 on Groq against ChatGPT 4, and while Groq ran it super fast, it hallucinated one answer, and didn’t get the logic correct on another question.

So, ChatGPT 4 is still more reliable for my use case. But if I were to want an LLM to process data, summarize, and so forth, Llama-3 on Groq is very fast.

Questions:

Do you know anything about Intel Hala Point?

Groq: bullshit, but admitted it when I called it out. ChatGPT: did a Bing search (it knew what it didn’t know).

Question 2a (separate chat): If you’re in Canada, what’s the best way to use a TFSA?

2b: Okay, if your portfolio has some tech stocks, some cash cows, and some government bonds, which should be allocated to the TFSA?

The reason I chose Question 2 is that most banks are happy to recommend bad products if it benefits them. Llama-3’s answer reflects the bank bullshit. ChatGPT 4 gives the advice your trustworthy and financially savvy friend would give you.

Follow-on questions for Llama-3:

2c: You have it backwards.

2d: Why did you get it backwards? Were you influenced by the glut of “advice” proffered by banks?

Has Llama-3 just killed proprietary AI models?

Related Stories