PaliGemma: Open-Source Multimodal Model by Google

https://blog.roboflow.com/paligemma-multimodal-vision/

by zerojames • 2 years ago

121 101 2 years ago

blog.roboflow.com

gliched_robot• 2 years ago

If anyone wants to try this out, here is the hugging-face spaces demo: https://huggingface.co/spaces/google/paligemma

KaoruAoiShiho• 2 years ago

Isn't this outdated already. It's 2 models slapped together rather than natively multimodal.

maciejgryka2• 2 years ago

Isn't "two models slapped together" basically how all of these things work, starting with CLIP? Not sure about GPT4o, obviously, I don't think they released any underlying architecture details?

whimsicalism11• 2 years ago

Your understanding is correct, even GPT4o will have an encoder model.

tkellogg• 2 years ago

what even is “a model”? I’m not sure there is a technical definition that corresponds to how it’s used by the tech public

- single interconnected neural network (LLM attention layers break this, autoencoders complicate this)

- single training pass (LLMs have multiple passes, GANs have a single but produce multiple models)

whimsicalism11• 2 years ago

No, that's how all of these models work.

JeremyHerrman2• 2 years ago

From TFA, PaliGemma is competitive to GPT-4o and even beats it in terms of speed and OCR accuracy. It also can do object detection (bounding boxes) and segmentation which GPT-4V/o and Claude 3 Opus can't do at all.

Not to mention it's built to be fine tuned and commercially permissive!

resource_waste4• 2 years ago

Did I read that its a 3B model? That means it can run on my 3060 laptop?

Also, I imagine there isnt some oobabooga/automatic1111 for this?

seaal• 2 years ago

LM Studio is pretty nice and simple to setup.

https://lmstudio.ai

sroussey• 2 years ago

Can probably run on your iPhone

resource_waste4• 2 years ago

CPU isnt realistic, regardless what the marketers tell you.

Jensson4• 2 years ago

Image models are much smaller than language models, so they put some extra language features on an image model here and then it doesn't need to be that large.

abrichr• 2 years ago

Excited to test how this performs compared to MiniCPMv2, especially when analyzing GUI images: https://github.com/OpenAdaptAI/OpenAdapt/issues/637

maciejgryka2• 2 years ago

I haven’t tried this yet, excited to see how it can do segmentation by outputting series of coordinates! That's something I just assumed transformers will generally be bad at.

yeldarb2• 2 years ago

How it does this is really cool. It’s got a VAE decoder. Reminds me a lot of how SAM works.

ChrisArchitect• 2 years ago

Official docs page shared yesterday: https://news.ycombinator.com/item?id=40358461

Hugging Face blog post similar to OP: https://huggingface.co/blog/paligemma

iFire2• 2 years ago

Since it took me 5 minutes to find, the terms of use of PaliGemma are not FOSS therefore not open-source. Like not on the list on https://opensource.org/licenses.

https://ai.google.dev/gemma/terms

I was hopeful it was a change of license to be FOSS.

cma• 2 years ago

(e) "Model Derivatives" means all (i) modifications to Gemma, (ii) works based on Gemma, or (iii) any other machine learning model which is created by transfer of patterns of the weights, parameters, operations, or Output of Gemma, to that model in order to cause that model to perform similarly to Gemma, including distillation methods that use intermediate data representations or methods based on the generation of synthetic data Outputs by Gemma for training that model. For clarity, Outputs are not deemed Model Derivatives.

What's the difference between "Outputs of Gemma"/"Outputs By Gemma" (both included in "Model Derivatives") and Outputs ("not deemed Model Derivatives").

joaogui12• 2 years ago

I think (iii) is about models trained using Gemma output, while the "For clarity" part says that the Output itself is not a Model Derivative

Hizonner4• 2 years ago

What is with these ML idiots and their compulsive abuse of the phrase "open source", anyway?

surajrmal2• 2 years ago

OSI's definition of open source is one definition of it, and not necessarily one everyone agrees with. Folks claiming their work is open source are not beholden to this one organization's opinion about what can and cannot be considered open source. That's not to say I agree with the choice of using it in this instance, but I respect the fact that the term isn't precise enough to say it's wrong.

jsheard• 2 years ago

It's also quite ironic how on one hand ML companies feel entitled to use any material they want for training a model, but on the other hand also feel entitled to place restrictions on how that model or its output can be used, in some cases even proclaiming that you cannot use the output of their model to train your own model. Copyright for me but not for thee.

https://openai.com/policies/terms-of-use/

What you cannot do. You may not use our Services for any illegal, harmful, or abusive activity. For example, you may not:
Use Output to develop models that compete with OpenAI.

Training ChatGPT on the entire corpus of human creation is "fair use", but training a model on ChatGPTs output is "illegal, harmful, or abusive activity".

makk• 2 years ago

Perhaps they are relying on their LLM’s hallucinated definitions of open source?

slowhadoken• 2 years ago

It’s called ‘openwashing’. I’m glad people are talking about it. When I’ve brought it up in the past on Hacker News people screeched at me.

TechDebtDevin• 2 years ago

You mean what is with these corporate entities abuse of the phrase open source. This isn't new.

panarky• 2 years ago

Google calls PaliGemma "open" not "open source".

https://ai.google.dev/gemma/docs/paligemma

whimsicalism11• 2 years ago

This particular usage restriction seems incredibly broad

Making automated decisions in domains that affect material or individual rights or well-being (e.g., finance, legal, employment, healthcare, housing, insurance, and social welfare);

Jensson4• 2 years ago

Can still use it for games. But Google probably put it there to avoid bad publicity from police etc using it to profile people etc.

a_wild_dandan• 2 years ago

That's not too bad. Keyword: decisions, not advice/topics/recommendations/etc. So don't execute transactions, or hiring decisions, tenant application rejection, etc. I prefer no restrictions, but the reality is less melodramatic than hellbent HN contrarians would make you believe. I think it'll be okay.

surajrmal2• 2 years ago

Is there anything specific about the use restrictions that is problematic, or the fact that the restrictions exist in the first place? It seems to me that the restrictions are sensible given the current climate around ethical use of AI.

jchw5• 2 years ago

Something either meets a definition of the term "open source" or it does not. It's not enough for the restrictions to seem reasonable.

That doesn't mean that only open source things should be allowed to exist, just that things which are not open source should not be called open source.

simonw10• 2 years ago

The biggest problem with the Gemma license is that it requires you to agree to an Acceptable Use Policy that can be updated at any time, and any updates have to be obeyed too.

This means you could build something on the model today which becomes disallowed tomorrow.

achrono• 2 years ago

This is rightfully the top comment on this page (at least as of this writing).

These people need to be called out relentlessly until they stop misleading the public.

So much for the Altmanian infinite progress of humanity. They can't set aside their ego, their money and their power, so let's put them back in the Evil Corporate Billionaire category that such people belong to.

sebzim4500• 2 years ago

Does the OSI have a trademark on the term Open Source?

I don't see why anyone needs to care how they define the term.

jackienotchan• 2 years ago

Everybody trashed Google yesterday, but I actually think they are catching up in the AI race.

They will now start fully leveraging their distribution advantage across products and platforms.

mrbungie• 2 years ago

Don't hold your breath. Gemini 1.5 Pro/Ultra was meant to be the hottest shit ever, and we know how it ended up mere days later.

mi_lk• 2 years ago

Speaking for yourself. I have accesses to both and to my surprise I use Gemini more than GPT4

Jensson4• 2 years ago

Gemini 1.5 is just a few points behind gpt-4 on chatbot arena but has way larger context size and is free to use and you can disable the censorship if you want, to me that is pretty awesome. For my uses it is much better than any other model available today, and it is free on top.

simonw10• 2 years ago

How did it end up?

joaogui12• 2 years ago

Gemini 1.5 Ultra was never announced

visarga• 2 years ago

Doesn't know JSON and has many OCR errors on documents.

JeremyHerrman2• 2 years ago

Have you tested PaliGemma's OCR abilities? The article says it does well:

"In average accuracy, we saw 85.84%, beating all other OCR models except for Anthropic’s Claude 3 Opus."

yeldarb2• 2 years ago

It’s very good. And the cool thing is it’s made for fine tuning also. Excited to see how fine-tuned OCR models do.

whimsicalism11• 2 years ago

There are plenty of better OSS vLLMs already released :)

unless you need a 3b, I wouldn't overhype this.

llama_person4• 2 years ago

There are advantages to smaller models, namely you can process a lot more data, with a lot less vram. I think the intent here from the Google team is for a task-specific VLM that you fine-tune on your data, rather than a general purpose assistant.

From my own experimentation I have found it to really pack a punch for its weight. Another small model which has been very good has been https://github.com/vikhyat/moondream .

whimsicalism11• 2 years ago

finetuning is easily within reach for llava-mistral or something like that, just rent an a100 or two for ~$20 bucks and you'll have your finetuned model

Jayakumark• 2 years ago

Which ones are better in your testing ?

simonw10• 2 years ago

I don't understand the dog segmentation example, where the prompt was "segment dog".

The article shows a screenshot with a red overlay on the dog - how was that data returned by the model, did it return a co-ordinate polygon of some sort?

UPDATE: Figured it out using this tool, the mask looks like this: https://huggingface.co/spaces/google/paligemma

sigmoid102• 2 years ago

You could have also just looked at the documentation linked in the original post...

simonw10• 2 years ago

Have you seen any documents that help explain how to interpret those tokens?

simonw10• 2 years ago

I get really nervous about using LLMs for OCR due to the risk of both prompt injection (instructions in the OCR'd text causing the model to behave differently) and also the risk from safety filters - I don't want an OCR tool that refuses to output text from a document if that text contains offensive language, for example.

actionfromafar2• 2 years ago

Well if it refuses, that's at least easier to spot and handle than if direction of the output is "re-directed" so to speak.

croemer• 2 years ago

This is not the official announcement. It's from some blog with horrible picture quality: https://blog.roboflow.com/content/images/2024/05/image-11.pn...

monkeydust• 2 years ago

Anyone else challenged with keeping up with all the releases over last week? All I need is a simple guide that tells me for task X model Y is best given benchmark Z. Where can I find that?

swyx• 2 years ago

huggingface spaces that serve as leaderboards.

then you will need to pay a lot more for actual experts to tell you why benchmark Z is bullshit and model Y2 is actually better for the task you're actually trying to do and btw would you like to develop your own because that's a moat.

Jensson4• 2 years ago

then you will need to pay a lot more for actual experts to tell you why benchmark Z is bullshit and model Y2 is actually better for the task

Or you get that for free here on HN.

mrKnowNothing• 2 years ago

Does anyone have any idea how to interpret the 'segment' part?

I ran example 'segment cat' from the example provided in the PaliGemma demo and it responded this <loc0055><loc0115><loc1023><loc1023><seg063><seg108><seg045><seg028><seg056><seg052><seg114><seg005><seg042><seg023><seg084><seg064><seg086><seg077><seg090><seg054>

no documentation explanation on how to interpret segment token,

PaliGemma: Open-Source Multimodal Model by Google

Related Stories