Not all 'open source' AI models are open: here's a ranking

https://www.nature.com/articles/d41586-024-02012-5

by weinzierl • 2 years ago

127 18 2 years ago

mirzap• 2 years ago

Which models claim that they are "open source" but are not? I think none of them. I think people confuse "open weight" with "open source." That confusion comes from the "internet AI experts," not the labs that produced them.

l5870uoo9y• 2 years ago

The confusion also stems from companies deliberately using the term for marketing purposes. Strong separation between the open source, academia and business environments doesn’t exist as much as it used to. Can’t imagine a Richard Stallman or Linus Thorvalds engaging in blurry half-open source, half business projects.

ninjin2• 2 years ago

Facebook has been touting their work as both "open source" and "open science", despite being in direct violation of the terms [1, 2]. Others have done similar things, but not on such a big scale and with models with such big reach. In the community, the terms have now become sufficiently muddied that it is hard to have a conversation about what an open model even is.

[1] https://arxiv.org/abs/2307.09288

[2] https://opensource.org/blog/metas-llama-2-license-is-not-ope...

Thankfully, OLMo [3] reverted their decision to go with a license that contained usage restrictions and hopefully this sets the standard going forward so that the corporate actors no longer can appropriate the term and will find a suitable term on their own such as "available". But it pains me to see that they had to write "truly open" to distinguish themselves from the "poseurs".

[3] https://allenai.org/olmo

dartos• 2 years ago

When llama got leaked meta went full EEE and now discussion around open source and LLMs will take years to straighten out

imjonse2• 2 years ago

Mistral even used the term 'truly open' in their announcement:

https://mistral.ai/news/mixtral-8x22b/

Also from the Llama 3 announcement: 'Today, we’re introducing Meta Llama 3, the next generation of our state-of-the-art open source large language model.'

imjonse2• 2 years ago

Section 1.3 of the paper has lots of examples.

https://dl.acm.org/doi/pdf/10.1145/3630106.3659005#page=6

lolinder• 2 years ago

Aside from what others have mentioned (tons of them do use the term open source), "open weight" is deliberately meant to parallel "open source" but in practice it doesn't. Most "open weight" models have usage restrictions that would make a code license most definitely not open.

The right term for most of these models would be "weights available".

tazu• 2 years ago

For some snark, I'd include "OpenAI" in the table with X on all the metrics...

ninjin2• 2 years ago

The authors did exactly that in the paper which the articles attempts to summarise:

https://dl.acm.org/doi/pdf/10.1145/3630106.3659005#page=6

evrydayhustling3• 2 years ago

Interesting, I wonder why it was omitted in Springer's top tier journal Nature? Oh yeah: https://openai.com/index/axel-springer-partnership/

Edit: Looks like the science publishing company is "operated independently", despite family ties - https://media.springernature.com/full/springer-cms/rest/v1/c...

vhiremath4• 2 years ago

When they say “LLM data”, does that usually include the tokenizer as well? Beginner question from someone at the end of Karpathy’s Zero to Hero course.

monkeydust• 2 years ago

I think France (Macron) help push through the 'open' exemption, whatever that practically turns out to be, for the final EU AI Act...obviously not lost on people that is where Mistral lives.

AndrewKemendo• 2 years ago

Standard open source rules apply. Unless you can:

Modify the source code

Compile it yourself*

And be able to do it on your own compute*

Then it’s not open

* Which would include the data and the “compiler” which is generally the non-FOSS cUDNN or CUDA drivers btw

*it’s almost like we had a mini-computer, microprocessor revolution 40 years ago for precisely the same reason

Not all 'open source' AI models are open: here's a ranking

Related Stories