Ollama's new engine for multimodal models

https://ollama.com/blog/multimodal-models

by LorenDB • 1 year ago

353 76 1 year ago

ics• 1 year ago

I'll have to try this later but appreciate that the article gets straight to the point with practical examples and then the details.

newusertoday• 1 year ago

why does ollama engine has to change to support new models? every time a new model comes ollama has to be upgraded.

nkwaml• 1 year ago

Because of things like this: https://github.com/ggml-org/llama.cpp/issues/12637

Where "supporting" a model doesn't mean what you think it means for cpp

Between that and the long saga with vision models having only partial support, with a CLI tool, and no llama-server support (they only fixed all that very recently) the fact of the matter is that ollama is moving faster and implementing what people want before lama.cpp now

And it will finally shut down all the people who kept copy pasting the same criticism of ollama "it's just a llama.cpp wrapper why are you not using cpp instead"

Maxious• 1 year ago

There's also some interpersonal conflict in llama.cpp that's hampering other bug fixes https://github.com/ikawrakow/ik_llama.cpp/pull/400

w8nC• 1 year ago

Now it’s just a wrapper around hosted APIs.

Went with my own wrapper around llama.cpp and stable-diffusion.cpp with optional prompting hosted if I don’t like the result so much, but it makes a good start for hosted to improve on.

Also obfuscates any requests sent to hosted, cause why feed them insight to my use case when I just want to double check algorithmic choices of local AI? The ground truth relationship func names and variable names imply is my little secret

simonw2• 1 year ago

The timing on this is a little surprising given llama.cpp just finally got a (hopefully) stable vision feature merged into main: https://simonwillison.net/2025/May/10/llama-cpp-vision/

Presumably Ollama had been working on this for quite a while already - it sounds like they've broken their initial dependency on llama.cpp. Being in charge of their own destiny makes a lot of sense.

lolinder• 1 year ago

Do you know what exactly the difference is with either of these projects adding multimodal support? Both have supported LLaVA for a long time. Did that require special casing that is no longer required?

I'd hoped to see this mentioned in TFA, but it kind of acts like multimodal is totally new to Ollama, which it isn't.

simonw2• 1 year ago

There's a pretty clear explanation of the llama.cpp history here: https://github.com/ggml-org/llama.cpp/tree/master/tools/mtmd...

I don't fully understand Ollama's timeline and strategy yet.

refulgentis7• 1 year ago

It's a turducken of crap from everyone but ngxson and Hugging Face and llama.cpp in this situation.

llama.cpp did have multimodal, I've been maintaining an integration for many moons now. (Feb 2024? Original LLaVa through Gemma 3)

However, this was not for mere mortals. It was not documented and had gotten unwieldy, to say the least.

ngxson (HF employee) did a ton of work to get gemma3 support in, and had to do it in a separate binary. They dove in and landed a refactored backbone that is presumably more maintainable and on track to be in what I think of as the real Ollama, llama.cpp's server binary.

As you well note, Ollama is Ollamaing - I joked, once, that the median llama.cpp contribution from Ollama is a driveby GitHub comment asking when a feature will land in llama-server, so it can be copy-pasted into Ollama.

It's really sort of depressing to me because I'm just one dude, it really wasn't that hard to support it (it's one of a gajillion things I have to do, I'd estimate 2 SWE-weeks at 10 YOE, 1.5 SWE-days for every model release), and it's hard to get attention for detailed work in this space with how much everyone exaggerates and rushes to PR.

EDIT: Coming back after reading the blog post, and I'm 10x as frustrated. "Support thinking / reasoning; Tool calling with streaming responses" --- this is table stakes stuff that was possible eons ago.

I don't see any sign of them doing anything specific in any of the code they link, the whole thing reads like someone carefully worked with an LLM to present a maximalist technical-sounding version of the llama.cpp stuff and frame it as if they worked with these companies and built their own thing. (note the very careful wording on this, e.g. in the footer the companies are thanked for releasing the models)

I think it's great that they have a nice UX that helps people run llama.cpp locally without compiling, but it's hard for me to think of a project I've been more by turned off by in my 37 years on this rock.

tommica3• 1 year ago

Sidetangent: why is ollama frowned upon by some people? I've never really got any other explanation than "you should run llama.CPP yourself"

nicman23• 1 year ago

cpp was just faster and with more features that is all

cwillu2• 1 year ago

cpp is the thing doing all the heavy lifting, ollama is just a library wrapper.

It'd be like if handbrake tried to pretend that they implemented all the video processing work, when it's dependent on libffmpeg for all of that.

lhl• 1 year ago

Here's some discussion here: https://www.reddit.com/r/LocalLLaMA/comments/1jzocoo/finally...

Ollama appears to not properly credit llama.cpp: https://github.com/ollama/ollama/issues/3185 - this is a long-standing issue that hasn't been addressed.

This seems to have leaked into other projects where even when llama.cpp is being used directly, it's being credited to Ollama: https://github.com/ggml-org/llama.cpp/pull/12896

Ollama doesn't contributed to upstream (that's fine, they're not obligated to), but it's a bit weird that one of the devs claimed to have and uh, not really: https://www.reddit.com/r/LocalLLaMA/comments/1k4m3az/here_is... - that being said they seem to maintain their own fork so anyone could cherry pick stuff it they wanted to: https://github.com/ollama/ollama/commits/main/llama/llama.cp...

tommica3• 1 year ago

Thanks for the good explanation!

gavmor• 1 year ago

Here's a recent thread on Ollama hate from r/localLLaMa: https://www.reddit.com/r/LocalLLaMA/comments/1kg20mu/so_why_...

kergonath2• 1 year ago

r/localLLaMa is very useful, but also very susceptible to groupthink and more or less astroturfed hype trains and mood swings. This drama needs to be taken in context, there is a lot of emotion and not too much reason.

speedgoose• 1 year ago

To me, Ollama is a bit the Docker of LLMs. The user experience is inspired and the model file syntax is also inspired by the Dockerfile syntax. [0]

In the early days of Docker, we had the debate of Docker vs LXC. At the time, Docker was mostly a wrapper over LXC and people were dismissing the great user experience improvements of Docker.

I agree however that the lack of acknowledgement to llama.cpp for a long time has been problematic. They acknowledge the project now.

[0] https://github.com/ollama/ollama/blob/main/docs/modelfile.md

octocop• 1 year ago

For me it's because ollama is just a front-end for llama.cpp, but the ollama folks rarely acknowledge that.

buyucu2• 1 year ago

I abandoned Ollama because Ollama does not support Vulkan: https://news.ycombinator.com/item?id=42886680

You have to support Vulkan if you care about consumer hardware. Ollama devs clearly don't.

diggan4• 1 year ago

Besides the "culture"/licensing/FOSS issue already mentioned, I just wanted to be able to reuse model weights across various applications, but Ollama decided to ship their own way of storing things on disk + with their own registry. I'm guessing it's because they want to eventually be able to monetize this somehow, maybe "private" weights hosted on their registry or something. I don't get why they thought splitting up files into "blobs" made sense for LLM weights, seems they wanted to reduce duplication (ala Docker) but instead it just makes things more complicated for no gains.

End result for users like me though, is to have to duplicate +30GB large files just because I wanted to use the weights in Ollama and the rest of the ecosystem. So instead I use everything else that largely just works the same way, and not Ollama.

tommica3• 1 year ago

That is an interesting perspective, did not know about that at all!

bearjaws• 1 year ago

Anyone who has been around for 10 years can smell the Embrace, Extend, Extinguish model 100 miles away.

They are plainly going to capture the market, and switch to some "enterprise license" that lets them charge $, on the backs of other peoples work.

wirybeige• 1 year ago

They refuse to work with the community. There's also the open question of how they are going to monetize, given that they are a VC-backed company.

Why shouldn't I go with llama.cpp, lmstudio, or ramalama (containers/RH); I will at least know what I am getting with each one.

Ramalama actually contributes quite a bit back to llama.cpp/whipser.cpp (more projects probably), while delivering a solution that works better for me.

https://github.com/ollama/ollama/pull/9650 https://github.com/ollama/ollama/pull/5059

jimjimwii• 1 year ago

For me it's the R1 fiasco and their dishonesty. How anyone can continue to trust a project that brazenly mislead their users to such an extent just to cash in on the hype is beyond me.

oezi2• 1 year ago

I wish multimodal would imply text, image and audio (+potentially video). If a model supports only image generation or image analysis, vision model seems the more appropriate term.

We should aim to distinguish multimodal modals such as Qwen2.5-Omni from Qwen2.5-VL.

In this sense: Ollama's new engine adds vision support.

prettyblocks2• 1 year ago

I'm very interested in working with video inputs, is it possible to do that with Qwen2.5-Omni and Ollama?

oezi2• 1 year ago

I have only tested Qwen2.5-Omni for audio and it was hit and miss for my use case of tagging audio.

machinelearning• 1 year ago

What's a use case are you interested in re: video?

tough2• 1 year ago

https://huggingface.co/blog/smolvlm

Koshima• 1 year ago

The timing makes sense if you consider the broader trend in the LLM space. We're moving from just text to more integrated, multimodal experiences, and having a tightly controlled engine like this could be a game changer for developers building apps that require real-time, context-rich understanding.

bradly3• 1 year ago

The strength with Ollama for me was the ease of being able to run a simple Docker command and be up and running locally without any tinkering, but with image and video Docker is no longer an option as Docker does not use the GPU. I'm curious how Ollama plans to support their Docker integration going forward or if it is a less important part of the project that I'm giving it credit for.

IanCal• 1 year ago

You can use a GPU with docker - at least on some platforms. There's more setup though, nvidia have some details to help https://docs.nvidia.com/datacenter/cloud-native/container-to...

bradly3• 1 year ago

Thank you. I should have specified on MacOS. I ran into this recently trying to setup stable-diffusion-webui/InvokeAI/Foocus and finding it much more complicated to get working for me on my personal laptop than the llms.

mark_l_watson• 1 year ago

I have mostly used Ollama to run local models for close to a year, love it, but I have barely touched Llava, etc. multi modal support because all my personal use cases are text based.

Question: what are cool and useful multi modal projects have people here built using local models?

I am looking for personal project ideas.

JKCalhoun• 1 year ago

Does Ollama support the "user context" that higher level LLMs like ChatGPT have?

I'm not clear what they are called (or how implemented) — but perhaps 1) the initial prompt/context (that, for example, Grok has got in trouble with recently) and 2) the kind of saved context that allows ChatGPT to know things about your prompt-history so it can better answer future queries.

(My use of ollama has been pretty bare-bones and I have not seen anything covering these higher level features in -help.)

lxgr2• 1 year ago

My understanding is that ollama is more of an "LLM backend", i.e. it provides a server process on your machine that answers requests relatively statelessly.

I believe it keeps the model loaded across sessions, and possibly keeps the KV cache warm for ongoing sessions (but I doubt it, based on the API shape; I don't see a "session" parameter), but that's about it. Nothing seems to be written to disk.

Features like ChatGPT's "memories" or cross-chat context require a persistence layer that's probably best suited for a "frontend". Ollama's API does support passing in requests with history, for example: https://github.com/ollama/ollama/blob/main/docs/api.md#chat-...

codybontecou2• 1 year ago

Is there more to memory than just an entry into the context/messages array passed to the LLM?

yossi_peti• 1 year ago

Their example "understanding and translating vertical Chinese spring couplets to English" has a lot of mistakes in it. I'm guessing the person writing the blog post to show off that example doesn't actually know Chinese.

What is actually written: Top: 家和国盛 Left: 和谐生活人人舒畅迎新春 Right: 平安社会家家欢乐辞旧岁

What Ollama saw: Top: 盛和家国 (correct characters but wrong order) Left: It reads "新春" (new spring) as 舒畅 (comfortable) Right: 家家欢欢乐乐辞旧岁 (duplicates characters and omits the first four)

mchiang• 1 year ago

I'm one of the maintainers who ran that example. I am Chinese.

The English translation, I thought was pretty spot on. We don't hide the mistakes of the models or fake the demos.

Overtime, of course I hope the models to improve much more

clpm4j• 1 year ago

The whole '*llama' naming convention in the LLM world is more confusing to me than it probably should be. So many llamas running around out here.

mcbuilder• 1 year ago

Unfortunately the speed of AI/ML is so crazy fast. I don't know a better way to keep track other than paying attention all the time. The field also loves memey names. A few years ago everyone was naming models after Sesame Street characters, there were the YOLO family of models. Conference papers are not immune, in fact they are greatest "offenders".

andy_xor_andrew• 1 year ago

They are talking a lot about this new engine - I'd love to see details on how it's actually implemented. Given llama.cpp is a herculean feat, if you are going to claim to have some replacement for it, an example of how you did it would be good!

Based on this part:

We set out to support a new engine that makes multimodal models first-class citizens, and getting Ollama’s partners to contribute more directly the community - the GGML tensor library.

And from clicking through a github link they had:

https://github.com/ollama/ollama/blob/main/model/models/gemm...

My takeaway is, the GGML library (the thing that is the backbone for llama.cpp) must expose some FFI (foreign function interface) that can be invoked from Go, so in the ollama Go code, they can write their own implementations of model behavior (like Gemma 3) that just calls into the GGML magic. I think I have that right? I would have expected a detail like that to be front and center in the blog post.

Hugsun• 1 year ago

Ollama are known for their lack of transparency, poor attribution and anti-user decisions.

I was surprised to see the amount of attribution in this post. They've been catching quite a bit of flack for this so they might be adjusting.

ac29• 1 year ago

I am amused that one of the handful examples they chose to use is wrong:

"The best way to get to Stanford University from the Ferry Building in San Francisco depends on your preferences and budget. Here are a few options:

1. *By Car*: Take US-101 South to CA-85 South, then continue on CA-101 South."

CA 85 is significantly farther down 101 than Palo Alto.

Ollama's new engine for multimodal models

Related Stories