Ollama now supports AMD graphics cards

by tosh • 2 years ago

633 215 2 years ago

sofixa• 2 years ago

This is great news. The more projects do this, the less of a moat CUDA is, and the less of a competitive advantage Nvidia has.

anonymous-panda• 2 years ago

What does performance look like?

sevagh• 2 years ago

Spoiler alert: not good enough to break CUDA's moat

Zambyte9• 2 years ago

I get 35tps on Mistral:7b-Instruct-Q6_K with my 6650 XT.

ixaxaar• 2 years ago

Hey I did, and sorry for the self promo,

Please check out https://github.com/geniusrise - tool for running llms and other stuff, behaves like docker compose, works with whatever is supported by underlying engines:

Huggingface - MPS, cuda VLLM - cuda, ROCm llama.cpp, whisper.cpp - cuda, mps, rocm

Also coming up integration with spark (TorchDistributor), kafka and airflow.

Symmetry2• 2 years ago

Given the price of top line NVidia cards, if they can be had at all, there's got to be a lot of effort going on behind the scenes to improve AMD support in various places.

jeff-davis• 2 years ago

What are the barriers to doing so?

reilly3000• 2 years ago

I’m curious as to how they pulled this off. OpenCL isn’t that common in the wild relative to Cuda. Hopefully it can become robust and widespread soon enough. I personally succumbed to the pressure and spent a relative fortune on a 4090 but wish I had some choice in the matter.

Apofis• 2 years ago

I'm surprised they didn't speak about the implementation at all. Anyone got more intel?

refulgentis5• 2 years ago

They're open source and based on llama.cpp so nothings secret.

My money, looking at nothing, would be on one of the two Vulkan backends added in Jan/Feb.

I continue to be flummoxed by a mostly-programmer-forum treating ollama like a magical new commercial entity breaking new ground.

It's a CLI wrapper around llama.cpp so you don't have to figure out how to compile it

j33zusjuice• 2 years ago

Ahhhh, I see what you did there.

harwoodr• 2 years ago

ROCm: https://github.com/ollama/ollama/commit/6c5ccb11f993ccc88c47...

programmarchy2• 2 years ago

Apple killed off OpenCL for their platforms when they created Metal which was disappointing. Sounds like ROCm will keep it alive but the fragmentation sucks. Gotta support CUDA, OpenCL, and Metal now to be cross-platform.

jart9• 2 years ago

What is OpenCL? AMD GPUs support CUDA. It's called HIP. You just need a bunch of #define statements like this:

    #ifndef __HIP__
    #include <cuda_fp16.h>
    #include <cuda_runtime.h>
    #else
    #include <hip/hip_fp16.h>
    #include <hip/hip_runtime.h>
    #define cudaSuccess hipSuccess
    #define cudaStream_t hipStream_t
    #define cudaGetLastError hipGetLastError
    #endif

Then your CUDA code works on AMD.

moffkalast• 2 years ago

OpenCL is as dead as OpenGL and the inference implementations that exist are very unperformant. The only real options are CUDA, ROCm, Vulkan and CPU. And Vulkan is a proper pain too, takes forver to build compute shaders and has to do so for each model. It only makes sense on Intel Arc since there's nothing else there.

zozbot234• 2 years ago

SYCL is a fairly direct successor to the OpenCL model and is not quite dead, Intel seems to be betting on it more than others.

mpreda• 2 years ago

ROCm includes OpenCL. And it's a very performant OpenCL implementation.

taminka• 2 years ago

why though? except for apple, most vendors still actively support it and newer versions of OpenCL are released…

karmakaze2• 2 years ago

It would serve Nvidia right if their insistence on only running CUDA workloads on their hardware results in adoption of ROCm/OpenCL.

KeplerBoy• 2 years ago

OpenCL is fine on Nvidia Hardware. Of course it's a second class citizen next to CUDA, but then again everything is a second class citizen on AMD hardware.

aseipp• 2 years ago

You can use OpenCL just fine on Nvidia, but CUDA is just a superior compute programming model overall (both in features and design.) Pretty much every vendor offers something superior to OpenCL (HIP, OneAPI, etc), because it simply isn't very nice to use.

shmerl• 2 years ago

May be Vulkan compute? But yeah, interesting how.

deadalus• 2 years ago

I wish AMD did well in the Stable Diffusion front because AMD is never greedy on VRAM. The 4060Ti 16GB(minimum required for Stable Diffusion in 2024) starts at $450.

AMD with ROCm is decent on Linux but pretty bad on Windows.

choilive• 2 years ago

They bump up VRAM because they can't compete on raw compute.

risho• 2 years ago

it doesn't matter how much compute you have if you don't have enough vram to run the model.

wongarsu• 2 years ago

Or rather Nvidia is purposefully restricting VRAM to avoid gaming cards canibalizing their supremely profitable professional/server cards. AMD has no relevant server cards, so they have no reason to hold back on VRAM in consumer cards

api2• 2 years ago

They lag on software a lot more than the lag on silicon.

Adverblessly• 2 years ago

I run A1111, ComfyUI and kohya-ss on an AMD (6900XT which has 16GB, the minimum required for Stable Diffusion in 2024 ;)), though on Linux. Is it a Windows specific Issue for you?

Edit to add: Though apparently I still don't run ollama on AMD since it seems to disagree with my setup.

wastewastewaste• 2 years ago

You don't need 16gb, literally majority of people don't have that and use 8gb and up. especially with forge

simon83• 2 years ago

Does anyone know how the AMD consumer GPU support on Linux has been implemented? Must use something else than ROCm I assume? Because ROCm only supports the 7900 XTX on Linux[1], while on Windows[2] support is from RX 6600 and upwards.

[1] https://rocblas.readthedocs.io/en/rocm-6.0.0/about/compatibi... [2]: https://rocblas.readthedocs.io/en/rocm-6.0.0/about/compatibi...

Symmetry2• 2 years ago

The newest release, 6.0.2, supports a number of other cards[1] and in general people are able to get a lot more cards to work than are officially supported. My 7900 XT worked on 6.0.0 for instance.

[1]https://rocm.docs.amd.com/projects/install-on-linux/en/lates...

bravetraveler• 2 years ago

I wouldn't read too much into support. It's more in terms of business/warranty/promises than what can actually do things

I've had a 6900XT since launch and this is the first I'm hearing "unsupported", having played with ROCM plenty over the years with Fedora Linux.

I think, at most, it's taken a couple key environment variables

HarHarVeryFunny• 2 years ago

How hard would it be for AMD just to document the levels of support of different cards the way NVIDIA does with their "compute capability" numbers ?!

I'm not sure what is worse from AMD - the ML software support they provide for their cards, or the utterly crap documentation.

How about one page documenting AMD's software stack compared to NVIDIA, one page documenting what ML frameworks support AMD cards, and another documenting "compute capability" type numbers to define the capabilities of different cards.

KronisLV• 2 years ago

Feels like all of this local LLM stuff is definitely pushing people in the direction of getting new hardware, since nothing like RX 570/580 or other older cards sees support.

On one hand, the hardware nowadays is better and more powerful, but on the other, the initial version of CUDA came out in 2007 and ROCm in 2016. You'd think that compute on GPUs wouldn't require the latest cards.

hugozap3• 2 years ago

I'm a happy user of Mistral on my Mac Air M1.

isoprophlex2• 2 years ago

How many gbs of RAM do you have in your M1 machine?

jonplackett• 2 years ago

Is it easy to set this up?

mysteria• 2 years ago

The Ollama backend llama.cpp definitely supports those older cards with the OpenCL and Vulkan backends, though performance is worse than ROCm or CUDA. In their Vulkan thread for instance I see people getting it working with Polaris and even Hawaii cards.

https://github.com/ggerganov/llama.cpp/pull/2059

Personally I just run it on CPU and several tokens/s is good enough for my purposes.

bradley13• 2 years ago

No new hardware needed. I was shocked that Mixtral runs well on my laptop, which has a so-so mobile GPU. Mixtral isn't hugely fast, but definitely good enough!

superkuh• 2 years ago

llama.cpp added first class support for the RX 580 by implementing the vulkan backend. There are some issues on older kernel amdgpu code where a llm process VRAM is never reloaded if it gets kicked out to GTT (in 5.x kernels) but overall it's much faster than the clBLAST opencl implementation.

jmorgan2• 2 years ago

The compatibility matrix is quite complex for both AMD and NVIDIA graphics cards, and completely agree: there is a lot of work to do, but the hope is to gracefully fall back to older cards.. they still speed up inference quite a bit when they do work!

XCSme• 2 years ago

My 1080ti runs ok even this 47B model: https://ollama.com/library/dolphin-mixtral

Zambyte9• 2 years ago

It's pretty funny to see this blog post, when I have been running Ollama on my AMD RX 6650 for weeks :D

They have shipped ROCm containers since 0.1.27 (21 days ago). This blog post seems to be published along with the latest release, 0.1.29. I wonder what they actually changed in this release with regards to AMD support.

Also: see this issue[0] that I made where I worked through running Ollama on an AMD card that they don't "officially" support yet. It's just a matter of setting an environment variable.

[0] https://github.com/ollama/ollama/issues/2870

Edit: I did notice one change, now the starcoder2[1] model works now. Before that would crash[2].

[1] https://ollama.com/library/starcoder2

[2] https://github.com/ollama/ollama/issues/2953

throwaway5959• 2 years ago

I mean, it was 21 days ago. What’s the difference?

yjftsjthsd-h• 2 years ago

2 versions, apparently

mchiang• 2 years ago

While the PRs went in slightly earlier, much of the time was spent on testing the integrations, and working with AMD directly to resolve issues.

There were issues that we resolved prior to cutting the release, and many reported by the community as well.

Zambyte9• 2 years ago

Thank you for clarifying and thanks for the great work you do!

cjbprime3• 2 years ago

Maybe they wanted to wait for bug reports for 21 days before publishing a popular blog post about it..?

layoric• 2 years ago

What kind of performance do you see with an RX 6650?

SushiHippie• 2 years ago

Can anyone (maybe ollama contributors) explain to me the relationship between llama.cpp and ollama?

I always thought that ollama basically just was a wrapper (i.e. not much changes to inference code, and only built on top) around llama.cpp, but this makes it seem like it is more than that?

Zambyte9• 2 years ago

llama.cpp also supports AMD cards: https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#hi...

renewiltord3• 2 years ago

Wow, that's a huge feature. Thank you, guys. By the way, does anyone have a preferred case where they can put 4 AMD 7900XTX? There's a lot of motherboards and CPUs that support 128 lanes. It's the physical arrangement that I have trouble with.

nottorp2• 2 years ago

Used crypto mining parts not available any more?

duskwuff• 2 years ago

Crypto mining didn't require significant bandwidth to the card. Mining-oriented motherboards typically only provisioned a single lane of PCIe to each card, and often used anemic host CPUs (like Celeron embedded parts).

segmondy2• 2 years ago

You don't need 128 lanes. 8x PCIe3 is more than enough, so for 4 cards that's 32. Most CPUs have about 40lanes. If you are not doing much that would be more than sufficient. Buy a PCIe riser. Go to amazon and search for it, a 16x to 16x PCIe riser. They go for about $25-$30 often about 20-30cm. If you want really long one, you can get one from China a 60cm for about the same price, you just have to wait for 3 weeks. That's what I did. Stuffing all those in a case is often difficult, so you have to go open rig. Either have the cables running out your computer and figuring out a way to support the cards while keeping them cool or just buy a small $20-$30 open rig frame.

renewiltord3• 2 years ago

Ah ha, that's the part I was curious about. I was wondering if I could keep everything cool open rig. I'm waiting for the stuff to arrive: risers, board, CPU, GPUs. And I've been putting it off because I wasn't sure how about the case. All right then, open rig frame. Thank you!

Dylan16807• 2 years ago

Most CPUs have about 20 lanes (plus a link to the chipset).

On the one hand, they will be gen 4 or 5, so they're the equivalent of 40-80 gen 3 lanes.

On the other hand, you can only split them up if you have a motherboard that supports bifurcation. If you buy the wrong model, you're stuck dedicating the equivalent of 64 gen 3 lanes to a single card.

Edit: Actually, looking into it further, current Intel desktop processors will only run their lanes as 16(+4) or 8+8(+4). You can kind of make 4 cards work by using chipset-fed slots, but that sucks. You could also get a PCIe switch but those are very expensive. AMD will do 4+4+4+4(+4) on the right boards.

kodarna• 2 years ago

I wonder why they aren't supporting RX 6750 XT and lower yet, are there architectural differences between these and RX 6800+?

Zambyte9• 2 years ago

They don't support it, but it works if you set an environment variable.

https://github.com/ollama/ollama/issues/2870#issuecomment-19...

slavik81• 2 years ago

Those are Navi 22/23/24 GPUs while the RX 6800+ GPUs are Navi 21. They have different ISAs... however, the ISAs are identical in all but name.

LLVM has recently introduced a unified ISA for all RDNA 2 GPUs (gfx10.3-generic), so the need for the environment variable workaround mentioned in the other comment should eventually disappear.

latchkey• 2 years ago

If anyone wants to run some benchmarks on MI300x, ping me.

freedomben• 2 years ago

I'm thrilled to see support for RX 6800/6800 XT / 6900 XT. I bought one of those for an outrageous amount during the post-covid shortage in hopes that I could use it for ML stuff, and thus far it hasn't been very successful, which is a shame because it's a beast of a card!

Many thanks to ollama project and llama.cpp!

mey2• 2 years ago

Sad to see that the cut off is just after 6700 XT which is what is in my desktop. They indicate more devices are coming, hopefully that includes some of the more modern all in one chips with RDNA 2/3 from AMD as well.

mey2• 2 years ago

It appears that the cut off lines up with HIP SDK support from AMD, https://rocm.docs.amd.com/projects/install-on-windows/en/lat...

throawayonthe• 2 years ago

I’ve already been using ollama with my 6700xt just fine, you just have to set some env variable to make rocm work “unoficially”

The linked page says they will support more soon, so i’m guessing this will just be integrated

bavell• 2 years ago

I've been using my 6750XT for more than a year now on all sorts of AI projects. Takes a little research and a few env vars but no need to wait for "official" support most of the time.

rcarmo• 2 years ago

Curious to see if this will work on APUs. Have a 7840HS to test, will give it a go ASAP.

airocker6• 2 years ago

I heard "Nvidia for LLM today is similar to how Sun Microsystems was for the web"

api2• 2 years ago

... for a very brief period of time until Linux servers and other options caught up.

airocker6• 2 years ago

OS was possibly much more complicated to write at that time than CUDA is to write today. And competition is too strong. It might be more briefer than even Sun.

rahimnathwani3• 2 years ago

Here is the commit that added ROCm support to llama.cpp back in August:

https://github.com/ggerganov/llama.cpp/commit/6bbc598a632560...

65a2• 2 years ago

Yep, and it deserves the credit! He who writes the cuda kernel (or translates it) controls the spice.

I had wrapped this and had it working in Ollama months ago as well: https://github.com/ollama/ollama/pull/814. I don't use Ollama anymore, but I really like the way they handle device memory allocation dynamically, I think they were the first to do this well.

rahimnathwani3• 2 years ago

I'm curious about both:

- what's special about the memory allocation, and how might it help me?

- what are you now using instead of ollama?

tarruda• 2 years ago

Does this work with integrated Radeon graphics? If so it might be worth getting one of those Ryzen mini PCs to use as a local LLM server.

stebalien• 2 years ago

Not yet: https://github.com/ollama/ollama/issues/2637

eclectic296• 2 years ago

I'm not sure why Ollama garners so much attention. It has limited value - used for only experimenting with models + cannot support more than 1 model at a time. It's not meant for production deployments. Granted that it makes the experimentation process super easy but for something that relies on llama.cpp completely and whose main value proposition is easy model management I'm not sure it deserves the brouhaha people are giving it.

Edit: what do you do after the initial experimentation? you need to deploy these models eventually to production. I'm not even talking about giving credit to llama.cpp, just mentioning that this product is gaining disproportionate attention and kudos compared to the value it delivers. Not denying that it's a great product.

reustle• 2 years ago

Even for just running a model locally, Ollama provided a much simpler "one click install" earlier than most tools. That in itself is worth the support.

Aka456• 2 years ago

Koboldcpp is also very, very good, plug and play, very complet web UI, nice little api with sse text streaming, vulkan accelerated, have an AMD fork...

crooked-v• 2 years ago

Granted that it makes the experimentation process super easy

That's the answer to your question. It may have less space than a Zune, but the average person doesn't care about technically superior alternatives that are much harder to use.

thejohnconway• 2 years ago

*Nomad

And lame.

davidhariri• 2 years ago

As it turns out, making it faster and better to manage things tends to get people’s attention. I think it’s well deserved.

vikramkr• 2 years ago

It's nice for personal use which is what I think it was built for, has some nice frontend options too. The tooling around it is nice, and there are projects building in rag etc. I don't think people are intending to deploy days services through these tools

andrewstuart• 2 years ago

Sounds like you are dismissing Ollama as a "toy".

Refer:

https://paulgraham.com/startupideas.html

nerdix2• 2 years ago

The answer to your question is:

ollama run mixtral

That's it. You're running a local LLM. I have no clue how to run llama.cpp

I got Stable Diffusion running and I wish there was something like ollama for it. It was painful.

viraptor• 2 years ago

On a mac, https://drawthings.ai is the ollama of Stable Diffusion.

ghurtado• 2 years ago

For me, ComfyUI made the process of installing and playing with SD about as simple as a Windows installer.

jameshart7• 2 years ago

The README is pretty clear, albeit it talks about a lot of optional steps you don’t need, but it’s essentially gonna be something like:

   git clone https://github.com/ggerganov/llama.cpp.git
   cd llama.cpp
   make
   wget https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/resolve/main/mixtral-8x7b-v0.1.Q4_K_M.gguf?download=true
   ./main -m ./mixtral-8x7b-v0.1.Q4_K_M.gguf -n 128

icelain• 2 years ago

Check out EasyDiffusion.

cbhl• 2 years ago

In my opinion, pre-built binaries and an easy-to-use front-end are things that should exist and are valid as a separate project unto themselves (see, e.g., HandBrake vs ffmpeg).

Using the name of the authors or the project you're building on can also read like an endorsement, which is not _necessarily_ desirable for the original authors (it can lead to ollama bugs being reported against llama.cpp instead of to the ollama devs and other forms of support request toil). Consider the third clause of BSD 3-Clause for an example used in other projects (although llama.cpp is licensed under MIT).

airocker6• 2 years ago

More than one is easy: put it behind a load balancer. Put one ollama in one container or one port.

Zambyte9• 2 years ago

That is still one model per instance of Ollama, right?

eclectic296• 2 years ago

FWIW Ollama has no concurrency support even though llama.cpp's server component (the thing that Ollama actually uses) supports it. Besides, you can't have more than 1 model running. Unloading and loading models is not free. Again, there's a lot more and really much of the real optimization work is not in Ollama; it's in llama.cpp which is completely ignored in this equation.

Karrot_Kream4• 2 years ago

I mean, it takes something difficult like an LLM and makes it easy to run. It's bound to get attention. If you've tried to get other models like BERT based models to run you'll realize just how big the usability gains are running ollama than anything else in the space.

If the question you're asking is why so many folks are focused on experimentation instead of productionizing these models, then I see where you're coming from. There's the question of how much LLMs are actually being used in prod scenarios right now as opposed to just excited people chucking things at them; that maybe LLMs are more just fun playthings than tools for production. But in my experience as HN has gotten bigger, the number of posters talking about productionizing anything has really gone down. I suspect the userbase has become more broadly "interested in software" rather than "ships production facing code" and the enthusiasm in these comments reflects those interests.

FWIW we use some LLMs in production and we do not use ollama at all. Our prod story is very different than what folks are talking about here and I'd love to have a thread that focuses more on language model prod deployments.

jart9• 2 years ago

Well you would be one of the few hundred people on the planet doing that. With local LLMs we're just trying to create a way for everyone else to use AI that doesn't require sharing all their data with them. First thing everyone asks for of course is how to turn the open source local llms into their own online service.

evilduck2• 2 years ago

I am 100% uninterested in your production deployment of rent seeking behavior for tools and models I can run myself. Ollama empowers me to do more of that easier. That’s why it’s popular.

jameshart7• 2 years ago

OP’s point is more that Ollama isn’t what’s doing the empowering. Llama.cpp is.

brucethemoose2• 2 years ago

Interestingly, Ollama is not popular at all in the "localllama" community (which also extends to related discords and repos).

And I think thats because of capabilities... Ollama is somewhat restrictive compared to other frontends. I have a littany of reasons I personally wouldn't run it over exui or koboldcpp, both for performance and output quality.

This is a necessity of being stable and one-click though.

elwebmaster• 2 years ago

You are not making any sense. I am running ollama and Open WebUI (which takes care of auth) in production.

ecnahc515• 2 years ago

Ollama is the Docker of LLMs. Ollama made it _very_ easy to run LLMs locally. This is surprisingly not as easy as it seems, and incredibly useful.

kergonath6• 2 years ago

It's not meant for production deployments.

I am probably not the demographics you expect. I don’t do “production” in that sense, but I have ollama running quite often when I am working, as I use it for RAG and as a fancy knowledge extraction engine. It is incredibly useful:

- I can test a lot of models by just pulling them (very useful as progress is very fast),

- using their command line is trivial,

- the fact that it keeps running in the background means that it starts once every few days and stays out of the way,

- it integrates nicely with langchain (and a host of other libraries), which means that it is easy to set up some sophisticated process and abstract away the LLM itself.

what do you do after the initial experimentation?

I just keep using it. And for now, I keep tweaking my scripts but I expect them to stabilise at some point, because I use these models to do some real work, and this work is not monkeying about with LLMs.

I'm not even talking about giving credit to llama.cpp, just mentioning that this product is gaining disproportionate attention and kudos compared to the value it delivers.

For me, there is nothing that comes close in terms of integration and convenience. The value it delivers is great, because it enables me to do some useful work without wasting time worrying about lower-level architecture details. Again, I am probably not in the demographics you have in mind (I am not a CS person and my programming is usually limited to HPC), but ollama is very useful to me. Its reputation is completely deserved, as far as I am concerned.

rkwz• 2 years ago

I use it for RAG and as a fancy knowledge extraction engine

Curious, can you share more details about your usecase?

Abishek_Muthian• 2 years ago

For me all the projects which enable running & fine-tuning LLMs locally like llama.cpp, ollama, open-webui, unsolth etc. play a very important part in democratizing AI.

what do you do after the initial experimentation? you need to deploy these models eventually to production

I built GaitAnalyzer[1], to analyze my gait laptop; I had deployed it briefly in production when I had enough credits to foot the AWS GPU bills. Ollama made it very simple to deploy the application, Anyone who has used docker before can now run GaitAnalyzer in their computer.

[1] https://github.com/abishekmuthian/gaitanalyzer

observationist• 2 years ago

There's a thing somewhat conspicuous in its absence - why isn't llama.cpp more directly credited and thanked for providing the base technology powering this tool?

All the other cool "run local" software seems to have the appropriate level of credit. You can find llama.cpp references in the code, being set up in a kind of "as is" fashion such that it might be OK as far as MIT licensing goes, but it seems kind of petty to have no shout out or thank you anywhere in the repository or blog or ollama website.

GPT4All - https://gpt4all.io/index.html

LM Studio - https://lmstudio.ai/

Both of these projects credit and attribute appropriately, Ollama seems to bend over backwards so they don't have to?

jart9• 2 years ago

ollama has made a lot of nice contributions of their own. It's a good look to give a hat tip to the great work llama.cpp is also doing, but they're strictly speaking not required to do that in their advertising any more than llama.cpp is required to give credit to Google Brain, and I think that's because llama.cpp has pulled off tricks in the execution that Brain never could have accomplished, just as ollama has had great success focusing on things that wouldn't make sense for llama.cpp. Besides everyone who wants to know what's up can read the source code, research papers, etc. then make their own judgements about who's who. It's all in the open.

refulgentis5• 2 years ago

Like what? What contributions has ollama made to llama.cpp? It's not a big deal or problem. It just hasnt.

And they could have very, very, easily, there's a server, sitting right there.

They chose not to.

That's fine. But it's a choice.

The rest I chalk it up to inexperience and being busy.

galaxyLogic2• 2 years ago

Newton only gave credits to "Gods". He said he stands on shoulders of Gods or something like that. But he never mentioned which Gods in particular, did he?

jjice• 2 years ago

Just downloaded this and gave it a go. I have no experience with running any local models, but this just worked out of the box on my 7600 on Ubuntu 22. This is fantastic.

haolez2• 2 years ago

Is there an equivalent to ollama or gpt4all for Android? I'd like to host my model somewhere and talk to it via app.

verdverm8• 2 years ago

At that point, it sounds like your after an API endpoint for any model. Lots of solutions out there, depends more on your hosting

haolez2• 2 years ago

No, I want the opposite: I want a generic chatbot app that connects to any endpoint.

treprinum• 2 years ago

Any info on the performance? How does it compare to 4080/4090?

OtomotO• 2 years ago

Hm, fooocus manages to run, but for Ollama I get:

time=2024-03-16T00:11:07.993+01:00 level=WARN source=amd_linux.go:50 msg="ollama >recommends running the https://www.amd.com/en/support/linux-drivers: amdgpu version file >missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or >directory" time=2024-03-16T00:11:07.993+01:00 level=INFO source=amd_linux.go:85 msg="detected amdgpu >versions [gfx1031]" time=2024-03-16T00:11:07.996+01:00 level=WARN source=amd_linux.go:339 msg="amdgpu >detected, but no compatible rocm library found. Either install rocm v6, or follow manual >install instructions at https://github.com/ollama/ollama/blob/main/docs/linux.md#man..." time=2024-03-16T00:11:07.996+01:00 level=WARN source=amd_linux.go:96 msg="unable to verify >rocm library, will use cpu: no suitable rocm found, falling back to CPU" time=2024-03-16T00:11:07.996+01:00 level=INFO source=routes.go:1105 msg="no GPU detected"

Need to check how to install rocm on arch again... have done it once, a few moons back, but alas...

jmorgan2• 2 years ago

Ah, this is probably from missing ROCm libraries. The dynamic libraries are available as one of the release assets (warning: it's about 4GB expanded) https://github.com/ollama/ollama/releases/tag/v0.1.29 – dropping them in the same directory as the `ollama` binary should work.

h4x0rr• 2 years ago

No support for my rx 5700xt :(

userbinator• 2 years ago

Thanks, Ollama.

(Sorry, could not resist.)

ekianjo• 2 years ago

Does it also support AMD APUs?

VadimPR• 2 years ago

Ollama runs really, really slow on my MBP for Mistral - as in just a few tokens a second and it takes a long while before it starts giving a result. Anyone else run into this?

I've seen that it seems to be related to the amount of system memory available when ollama is started (??) however LM Studio does not have such issues.

skenderbeu• 2 years ago

llama.cpp now supports AMD graphics. Ollama just wraps around it

avocardio• 2 years ago

@geohot

Ollama now supports AMD graphics cards

Related Stories