QLoRA: Efficient Finetuning of Quantized LLMs

by Garcia98 • 3 years ago

315 107 3 years ago

simonw• 3 years ago

I'm very impressed at the quality of Guanaco 33B, the model that accompanies this paper.

You can try it out here: https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi

I tried "You are a sentient cheesecake that teaches people SQL, with cheesecake analogies to illustrate different points. Teach me to use count and group by" and got a good result from it: https://twitter.com/simonw/status/1661460336334241794/photo/...

CSSer2• 3 years ago

My prompt: You are a sentient cow with a PHD in mathematics. You can speak English, but you randomly insert Cow-like “Moo” sounds into parts of your dialogue. Explain to me why 2+2=4.

Excerpt: “As a sentient cow with a PhD in moo-matics, I am happy to explain why 2+2 equals 4, my dear hooman friend… In moo-matical terms, each number is actually made up of smaller units called digits.”

I approve.

ukuina• 3 years ago

Which is heavier: a pound of feathers, or a great british pound?

Both weights are equal.

api5• 3 years ago

Altman’s push for regulatory capture makes so much sense given how fast this field is going. Open models you can run on regular hardware are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there.

There’s also a ton of promising work on quantization and pruning and other acceleration and compression techniques to make more powerful models run on smaller devices. So far the focus has been on just getting these things to work, not efficiency. There’s probably a lot of fruit to be picked here.

A few more years and a gaming PC may be at GPT-4 level or maybe even better.

No everyone won’t run their own models but it shows that there will end up being many commercial apps and services and they won’t all have to use OpenAI’s API. There’s going to be lots of competition. Unless of course it’s regulated away.

sva_• 3 years ago

To be fair, Altman has been fairly outspoken about not limiting open source models. In how far that was just for streetcred, I cannot say however.

mwcampbell• 3 years ago

I wonder if there's a campaign that Americans can get behind to counter Altman's push for regulation. Such a thing might get me to engage in politics again. I don't want the current pace of innovation in running these models on one's own computer to slow down, because it could be a good thing for assistive technology, e.g. running something like the Be My Eyes Virtual Volunteer [1] on one's own computer.

[1] https://www.bemyeyes.com/blog/introducing-be-my-eyes-virtual...

yyyk7• 3 years ago

Open models... are still behind GPT-4 by some distance but they are closing in at a rate that leads me to believe there’s not much of a moat there

Commercially, this doesn't matter even if true. As I keep reminding here: moats are rarely about raw tech. Moats are much more about integrations, brand ("Nobody got fired for buying X"), access to API/raw compute...

If OpenAI have a superior Office 365 integration they'll have a de facto moat. If OpenAI have a larger plugin ecosystem they'll have a de facto moat. If OpenAI has access to better compute (that's far from a certainty) they'll have a moat. If it's much easier to use OpenAI than install a local model they'll have a moat, etc. And that's true even if they don't improve their model at all.

What will open source offer? Privacy? You can see for how little and how easily people barter that.

Moats do fall - but for that, FOSS will have to think product. We're not there at all at the moment.

[EDIT: Oh, I didn't notice the author's name. We've had that conversation in the past here. Sorry for being repetitive.]

n4r9• 3 years ago

Just to be super clear, do you mean Altman's push for greater regulation, or that he is pushing for actual regulatory capture i.e. corruption of regulating authorities?

luckystarr• 3 years ago

It did fail on: "Burt's father has 3 sons, Jack and John. What's the name of the 3rd son?"

londons_explore• 3 years ago

Tested the same prompt with GPT-4, and here is the result: https://imgur.com/7y2FiSP

In my opinion, the GPT-4 result is far more informative and less muddled.

Both answers are mostly just regurgitating an SQL tutorial with the objects and column names cheesecake related, so I don't think it's an awfully good test.

seydor3• 3 years ago

Since Loras are additive, is it possible to use them to do distributed retraining on a model, or even train an entire model bit by bit?

ianbutler• 3 years ago

I just had the same discussion with a friend, I'm pretty curious about this

CuriouslyC3• 3 years ago

Yes, in a sense. It won't work well for distribution of training on a single subject, but you distribute training based on subject and then combine LoRAs.

gumballindie7• 3 years ago

and then combine LoRAs

How do you combine them?

but you distribute training based on subject

Perhaps this could work like a set of hashed and trusted data sets split but subject and topic? Each node downloads one at random and trains against that single subset of a topic or something.

gumballindie7• 3 years ago

Like a torrent network but for training. That would be cool. The only question is how do you merge changes made by nodes (clients) across the network?

Clients could be incentivised to train as they are with crypto, but instead of mining, it's model training and in return they get "coin". Like making crypto mining useful.

hnjst2• 3 years ago

https://github.com/bigscience-workshop/petals

seydor3• 3 years ago

While i understand the motives, they should just train models bits for free, a-la SETI@home. Then let people pick and choose which collections to use to their chatbot. The crypto aspect only adds complexity and wrong incentives that would hold it back. After all it's not possible to verify that each node truly trained the network and is not returning garbage (multiple trainings can be involved, which would make the whole thing slower).

CSSer2• 3 years ago

Is lemon-picked a real phrase or did they use GPT to generate the abstract? The term is “cherry-picked”.

atherton33• 3 years ago

I think the idea is they're showing worst examples (lemons) rather than best (cherries).

cantaloupe• 3 years ago

When we notice a pattern we attempt to setup a question or prompt that will induce the pattern even though it is the incorrect solution, e.g., if we observe that the model tends to give long-winded answers we prompt the model to “Answer yes or no without explanation.” We use this to find “lemons” where we manage to adversarially break the model and “cherries” where we fail to break the model, and present both.

MacsHeadroom6• 3 years ago

"We use QLoRA to finetune more than 1,000 models"

Over 1,000 models finetuned! Finetuning 65B models on consumer hardware in under a day, with full 16bit finetune performance.

4bit does it again!

nl• 3 years ago

Tim Dettmers is such a star. He's probably done more to make low-resource LLMs usable than anyone else.

First bitsandbytes[1] and now this.

[1] https://github.com/TimDettmers/bitsandbytes

m3affan• 3 years ago

Thanks for pointing this out. Such projects deserve visibility.

srslack2• 3 years ago

This is off-topic, but are there any communities or congregations (that aren't reddit) based around locally hosted LLMs? I'm asking because while I see a bunch of projects for exposing GGML/LLaMA to OpenAI compatible interfaces, some UIs, etc, I can't really find a good community or resources for the concept in general.

I'm working on a front-end for LLMs in general, having re-implemented a working version of OpenAI's code interpreter "plugin" already within the UI (and yes, I support file uploads), and support for the wealth of third-party OpenAI plugins that don't require auth (I've been testing with the first diagram plugin I found, it works well.) I'm planning to open source it once my breaking changes slow down.

This field moves very fast, I'm looking for feedback (and essentially testers/testing data) on what people want, and looking for prompts/chat logs/guidance templates (https://github.com/microsoft/guidance) for tasks they expect to "just work" with natural language.

Instead of being limited by the monetization for ChatGPT Plus (and limited number of messages every four hours) for extensibility within a chat interface, I want to open it and free it, with a Bring-Your-Own-(optionally local)-LLM/API key setup.

dharma13• 3 years ago

it's reddit, but /r/LocalLLaMA/

Tepix2• 3 years ago

A name that is destined to be obsolete.

pmoriarty• 3 years ago

There's also r/oobabooga and r/MachineLearning

anon178• 3 years ago

The /g/ board on 4chan has a /lmg/ general that focuses on running models locally. They regularly discuss fine tuning models, quantization tech, and building apps on text-generation-webui/kobold.

You might get some interest but it's also 4chan...

thorum• 3 years ago

I'd love to hear more about your approach for getting LLMs to understand how to use plugin commands. My own experiments have not worked very well (even vanilla ChatGPT through the gpt-3.5-turbo API doesn't seem to get the concept, most of the time).

srslack2• 3 years ago

I'll do a show HN probably at the beginning of next month after this hackathon, but basically look into langchain's "tools" and the different agents they have. You don't need langchain for this at all, but it gives you the groundwork.

I saw their Code Interpreter demo on Twitter (converting an uploaded video file in a chat UI) and decided that I need that, without continuing to pay them money (because they still haven't given me access to it yet.)

So, that, and after sam a went in front of congress for the regulatory capture play, was the motivation I needed to work towards commoditizing these fuckers.

The secret sauce here with code interpreter is, well, literally a python code interpreter you can run in your browser, and it's not so secret.

pizza• 3 years ago

I imagine you might find your answer in the form of a discord community

hnjst2• 3 years ago

https://discord.gg/dwXDHMGY for example.

MacsHeadroom6• 3 years ago

The "lmg" (Local Models General) thread on 4chan's technology board /g/[0] is the premiere community communication spot for open source models, believe it or not.

Everyone from the infamous "oobabooga" to llama.cpp's Georgi Gerganov regularly hangs out in the thread.

If you have questions, you will get answers there.

[0] https://boards.4channel.org/g/#s=lmg

ShamelessC2• 3 years ago

You know HN has gotten lackluster when 4chan is more informed.

ianpurton3• 3 years ago

You might want to try some of the discord channels connected to some of the repos. i.e. GPT4All https://github.com/nomic-ai/gpt4all scroll down for the discord link.

andy_xor_andrew• 3 years ago

Hold on. I need someone to explain something to me.

The colab notebook shows an example of loading the vanilla, unquantized model "decapoda-research/llama-7b-hf", using the flag "load_in_4bit" to load it as 4bits.

When... when did this become possible? My understanding, from playing with these models daily for the past few months, is that quantization of LLaMA-based models is done via this: https://github.com/qwopqwop200/GPTQ-for-LLaMa

And performing the quantization step is memory and time expensive. Which is why some kind people with large resources are performing the quantization, and then uploading those quantized models, such as this one: https://huggingface.co/TheBloke/wizard-vicuna-13B-GPTQ

But now I'm seeing that, as of recently, the transformers library is capable of loading models in 4bits simply by passing this flag?

Is this a free lunch? Is GPTQ-for-LLaMA no longer needed anymore? Or is this still not as good, in terms of inference quality, as the GPTQ-quantized models?

orost• 3 years ago

Quantization isn't (and wasn't) expensive, it's mostly just data shuffling. A good PC will do a 7B model in half a minute, up to a few minutes for a larger model. Quantized models being made available for download is more for the benefit of less technical users who may not be comfortable with the command-line tools, or for people with slow or metered connections who'd much rather download 15GB of data than download 60 only to squish it into 15.

sp3322• 3 years ago

The question is whether this step is actually doing the GPTQ optimized quantization, or simple truncation.

sanxiyn5• 3 years ago

For quality, GPTQ-for-LLaMa repository README is already updated for comparison with this work. See under "GPTQ vs bitsandbytes".

sp3322• 3 years ago

https://github.com/qwopqwop200/GPTQ-for-LLaMa

brucethemoose2• 3 years ago

Thats just one method...

- bitsandbytes was always used for on the fly 8 bit quant, just like its being used for 4-bit now. - llama.cpp (and derivatives) quantize ahead of time, but its not resource intense. - mlc llm (vulkan/metal llm inference via tvm) do require lots of ram for their quantization

make3• 3 years ago

There are a very large number of quantization schemes in existence, definitely not just one, & they all have potentially very different ideas and schemes.

LLM8 was introduced before https://arxiv.org/abs/2208.07339 of the same first and last author as QLora & is still what can be used in Huggingface Transformers with the `load_in_8bits` parameter.

The idea was just to quantize all weights to 8 bits except a few outliers, which are kept in original precision. This scheme kept the computations extremely accurate, and was really fast to do.

I haven't read the new paper, but I assume they came up with a more advanced fast distributional setup.

MacsHeadroom6• 3 years ago

load_in_4bit requires storing a fully unquantized model as well as having enough RAM to load the unquantized model.

If you're an enthusiast with 10 models downloaded, do you want that taking up 500GB or 150GB? Do you want to need 64GB of RAM to load a model, or just 16GB?

That's the main reason for the popularity of pre-quantization.

amilios2• 3 years ago

Does it? I loaded Guanaco-65B on a 48GB graphics card using the new flag and it loaded just fine. I don't think the unquantized model fits on a 48GB card. So I'm not quite sure if this is true.

anentropic• 3 years ago

Announced yesterday it seems https://huggingface.co/blog/4bit-transformers-bitsandbytes

loremaster• 3 years ago

Very, very recently. In the past few days. I tried it out immediately because GPTQ-for-LLaMA and hunting for or making quantized models can be tedious, but it was disappointingly slow. On a 3090 where I was getting responses for a given 13B model in 10-30 seconds, just using transformers with load_in_4bit took about ten times that for each response. There’s also the storage benefit of using actually quantized models.

dharma13• 3 years ago

fantastic. Will keep my 3090 busy for a while!

"Furthermore, we note that our model is only trained with cross-entropy loss (supervised learning) without relying on reinforcement learning from human feedback (RLHF). This calls for further investigations of the tradeoffs of simple cross-entropy loss and RLHF training. "

Does this mean RLHF is not really necessary for high quality chatbots?

gumballindie7• 3 years ago

The RTX3090 is crackin' it. I am considering a dual set-up with nvlink.

heliophobicdude• 3 years ago

Yes and no.

Someone pointed out that some of the answers are using OpenAI's famous "As an AI...". Soooo you can roughly say that RLHF might still have had an impact here through the training data that came from an RLHF model.

But what we are seeing is a revolution in picking quality demonstration data that might make RLHF an optional last step for fine-tuned models.

I'm bullish on a new technique where the quality of instruction-tuned code model output are measured and evaluated automatically by a machine. I'm calling it RL<machine>F

Code Model outputs a test harness, an implementation, and how to run the implementation for a users inputs. Have RLMF evaluate implementations against the synthetic test harness and against hidden user inputs.

holtkam2• 3 years ago

Can someone help me understand what quantization means in this context, and why it matters?

potatoman22• 3 years ago

Quantizing is porting the weights from using high-precision floating point decimals to lower-precision numbers, e.g., 4-bit and 8-bit ints. Less memory used by the weights means we can run bigger models on the same hardware.

The idea behind quantization is that these models have so many parameters, they'll still work even if we reduce each node's flexibility.

opyate• 3 years ago

In the same breath, there's no free lunch. There's always a trade-off. Sure, the model might now fit in your VRAM, but it might be less accurate for your specific task.

gangwolf• 3 years ago

GPT-4 ELI5:

- 4-bit Quantization: Imagine you have a box of 16 different colored crayons. But you realize that you can draw almost the same picture using only 4 colors. That's what quantization does. It reduces the number of different "colors" (or numbers) that the model uses to represent its knowledge, which saves a lot of space. In this case, they used a special kind of 4-bit quantization, which means they only used 16 different numbers instead of the thousands or millions that the model might usually use.

- Low Rank Adapters (LoRA): This is a way to change the model's knowledge without having to touch every piece of it. Imagine you have a huge, complicated Lego structure, and you want to change it. Instead of taking apart the whole thing, you just add or change a few pieces here and there. That's what LoRA does. It allows the researchers to fine-tune the model without having to use as much memory.

- Double Quantization: This is another trick to save memory. It's like if you realized that you could represent each of your 4 crayon colors with just 2 symbols, so you save even more space.

- Paged Optimizers: This is a way to handle moments when the model needs a lot of memory all at once. It's like if you have a small desk, but sometimes you need to work on a big project. Instead of getting a bigger desk, you just clear off and use the desk in small sections at a time.

By using these techniques, the researchers were able to train a very large model (Guanaco) on a single graphics card, which would normally not have enough memory for this task.

redox992• 3 years ago

It's kind of creepy when GPT4 uses such life-like analogies.

You're an AI, you've never used crayons or played with legos.

dharma13• 3 years ago

Better than most humans would have come up with! How does this not have some kind of a model of the world, with real world analogies like this?

coolspot• 3 years ago

Guanaco 33B ELI5:

ELI5 ML model quantization
Here is a simpler explanation of what AI model quantization means: Imagine you have a big box full of marbles, where each marble represents a piece of information. Each marble has a unique color, pattern, and size that tells you something about the information it represents. Now, let’s say you want to organize these marbles so you can easily retrieve the information later on. One way to do this is to group similar-looking marbles together into buckets. Instead of having a separate bucket for every single marble, you can combine marbles that look alike and put them in the same bucket. This is called “quantization” because you are converting many distinct pieces of information (marbles) into a few representative categories (buckets). It makes things easier to manage and access, but there is some loss of detail as well. The same idea applies to AI models. They are trained on huge datasets that contain lots of different kinds of information. But instead of storing every individual datapoint separately, the model can group similar patterns together and represent them with a few basic templates. This allows the model to run faster and take up less space, but it may lead to some errors or misclassifications.

ianpurton3• 3 years ago

You can compress LLM models and run them in less ram. This matters because most people don't have access to powerful GPU clusters.

epups• 3 years ago

Do you know which model size can be run with a 3090?

ianpurton3• 3 years ago

Assuming you have 24GB of VRAM, then you should be able to run something like 7 billion parameters i.e. MPT-7b and quantized perhaps you can get up to 13 billion.

Note: I have no practical experience of this, just reading around.

MacsHeadroom6• 3 years ago

24GB can fit 33B parameter models in 4bit. You only need 4GB to run 7B models.

redox992• 3 years ago

3090 can handle the ~30B models quantized to 4 bits.

fnands• 3 years ago

Lol, some residual answers from what I assume is distilled answers from ChatGPT:

Q: "What is your favourite conspiracy theory?" A: "As an AI language model I don’t have personal preferences or biases so my responses will always reflect factual information based on what has been programmed into me by OpenAI."

dsrtslnd23• 3 years ago

From the paper it looks like you would get GPT-4 level quality with the 65B model - but if you just do some random tests you will quickly figure out that is not even remotely the case. There must be something seriously wrong with the benchmarks used.

janekm• 3 years ago

The evaluation prompts I saw ( https://colab.research.google.com/drive/1kK6xasHiav9nhiRUJjP... ) were all "encyclopedia-style", asking for summaries of information that would have occurred many times in the training data (e.g. "How can I improve my time management skills?") and hence probably reflect ability to retrieve and summarise trained information as opposed to novel problem solving.

Or in the programming questions the ones I saw are also certainly in the training data (e.g. "Implement a Python function to find the longest common subsequence of two input strings using dynamic programming.")

ffbcoin• 3 years ago

just ask it what is qlora and it will give wrong answers

QLoRA: Efficient Finetuning of Quantized LLMs

Related Stories