“Imprecise” language models are smaller, speedier, and nearly as accurate

https://spectrum.ieee.org/1-bit-llm

by jnord • 2 years ago

302 161 2 years ago

spectrum.ieee.org

hiddencost• 2 years ago

Nope nope nope.

Any time we find more efficiency, we can trade it for more quality by doing more compute. We'll always use as much compute as we can afford, until we stop getting quality gains that are worth the added cost.

XorNot• 2 years ago

Isn't this article just about an optimization though, sans the title?

I don't much care for all the "oh but the energy usage" claims in most tech things: it's all electricity, and it's all fungible. It usually seems to roll out as a proxy for "I don't like this thing".

Like even with cryptocurrency, there were a lot of people mistaking the issue of scalability - namely that "as a store of value" crypto would consume incredible amounts of other resources (and a lot of people got stuck trying to figure out how somehow "a hash" could be reclaimed for useful resources) to do less then alternatives, with "the energy usage itself is the problem".

Finding optimizations for LLMs is good because it means we can build cheaper LLMs, which means we can build larger LLMs then we otherwise could for some given constraint, which means we can miniaturize (or in this case specialize) more capable hardware. The thing which really matter is, can the energy usage be meaningfully limited to a sensible scaling factor given the capability that makes them useful?

Because environmentally, I can install solar panels to do zero-carbon training (and if LLMs are as valuable as they're currently being priced, this is a no-brainer - if people aren't lying about solar being "cheaper then fossil fuels").

dartos4• 2 years ago

It’s good that we can build cheaper LLMs, but the problem companies guzzling energy for LLM training won’t use less energy, they’ll just have better models

alisonkisk• 2 years ago

No, because the article is about sacrificing quality by using less energy. But the quality is already bad for non-toy purposes, so there is plenty of demand for higher-energy higher-quality systems.

eru• 2 years ago

Like even with cryptocurrency, there were a lot of people mistaking the issue of scalability - namely that "as a store of value" crypto would consume incredible amounts of other resources (and a lot of people got stuck trying to figure out how somehow "a hash" could be reclaimed for useful resources) to do less then alternatives, with "the energy usage itself is the problem".

To be fair, technology-wise they mostly solved this problem via proof-of-stake.

From an individual point of view you still expense enormous resources as a miner / validator in a proof-of-stake system. It's just that now the resources come in the form of lost opportunity costs for your staked tokens (eg staked Ethereum).

But from aggregated perspective of society, staked Ethereum is essentially free.

That has some parallels to how acquiring regular money, like USD, is something individuals spend a lot of effort on. But for the whole of society, printing USD is essentially free.

Because environmentally, I can install solar panels to do zero-carbon training (and if LLMs are as valuable as they're currently being priced, this is a no-brainer - if people aren't lying about solar being "cheaper then fossil fuels").

There's still opportunity costs for that energy. Unless you have truly stranded electricity that couldn't be used for anything else.

Finding optimizations for LLMs is good because it means we can build cheaper LLMs, which means we can build larger LLMs then we otherwise could for some given constraint, which means we can miniaturize (or in this case specialize) more capable hardware. The thing which really matter is, can the energy usage be meaningfully limited to a sensible scaling factor given the capability that makes them useful?

I agree with that paragraph. It's all about trade-offs. If we can shift the efficiency frontier, that's good. Then people can decide whether they want cheaper models at the same performance, or pay the same energy-price for better models, or a combination thereof. Or pay more energy for even better model

shikon7• 2 years ago

Yes, it seems the more efficient use we have for compute, the more valuable compute is to us, and the more compute we will be able to afford.

kazinator• 2 years ago

Cost is basically time here. "How much compute we can afford" is really "how long we are willing to wait for the result".

trashtester2• 2 years ago

Then again, time * size_of_cluster = cost.

If you train on a cluster that costs >$1M/day to operate, the wait time is likely to be a smaller concern than the financial cost, unless you're REALLY in a hurry to beat some competitor.

jandrese• 2 years ago

I'm starting to feel like the Thinking Machines CM-1 was 40 years ahead of its time.

chuckadams• 2 years ago

It had the most wonderful blinkenlights too! :)

deepfriedchokes• 2 years ago

One of my favorites, CM-1 t-shirt:

https://www.tamikothiel.com/cm/cm-tshirt.html

aussieguy1234• 2 years ago

When it needs to do more precise numerical calculations, perhaps, like a human, it could just use a calculator?

kromem3• 2 years ago

That's not really a concern.

If you have a trillion parameter 8-bit fp network or a trillion parameter 1.5-bit ternary network, based on the scaling in Microsoft's paper the latter will actually perform better.

A lot of the current thinking is that the nodes themselves act as superpositions for a virtualized network in a multidimensional vector space, so precision is fairly arbitrary for the base nodes and it may be that constraining the individual node values actually allows for a less fuzzy virtualized network by the end of the training.

You could still have a very precise 'calculator' feature in the virtual space no matter the underlying parameter precision, and because each parameter is being informed by overlapping virtual features, may even have less unexpected errors and issues with lower precision nodes.

xwolfi• 2 years ago

Yup, your response makes me think they should just use a calculator, like everyone.

CuriouslyC9• 2 years ago

People are trying to make these monolithic god models right now because everyone's chasing OpenAI's example with ChatGPT, but that approach is going to run out of steam for a lot of reasons. We're eventually going to end up with an agent model that does just what you say, recognizes specialized problem types such as math and calls the appropriate tool, such as a calculator or symbolic computation engine.

zozbot234• 2 years ago

Many models support function calling/RAG already, these are very similar features from a structural POV. But of course it's harder to train a model for such tasks, compared to just fitting some existing training set.

szundi3• 2 years ago

ChatGPT routinely does this already

meroes• 2 years ago

Hmm. And agent model for prompts we will ask K-12 math questions to? What are ultimately math questions often take lots of context to trudge through. It takes a solid core of natural language as well. I think there is always the need for a god model, because we want to speak in natural language as that’s easiest for us.

I’m not a theoretician nor scientist, but train models on math and logical reasoning. Some very simple word problems contains tons of context, like family trees, implicit information, etc. and the process of step by step reasoning, not just the final answer, requires even more natural language processing.

darkerside2• 2 years ago

It should just be able to program. A programming language is the ultimate verbal-to-math translation layer.

Tangent but... programming languages aren't designed for computers. Computers are perfectly happy with assembly or even binary. Programming languages are designed for humans, not just so we can see what others have done, but so that we can understand what we ourselves have done. We give the variable a name because we can't remember 0x0010101; but the computer remembers them both just fine.

boringg2• 2 years ago

Easier just to deploy more renewable energy

adtac• 2 years ago

Like often, both will probably happen

boringg2• 2 years ago

Right - it obviously isn't a binary option you can pursue both in unison.

I would want to continue the statement that we are still early innings on renewable energy -- and let's keep deploying it rapidly to manage increased compute demand.

alex_duf• 2 years ago

or is it cheaper to optimize a model to crush the competition on price?

brcmthrowaway• 2 years ago

Are there any 1-bit LLMs available on GitHub?

hughesjj• 2 years ago

Heck they have some zero bit llm's

https://github.com/kelseyhightower/nocode

lhl• 2 years ago

BitNet Implementations:

* https://huggingface.co/1bitLLM/bitnet_b1_58-large

* https://github.com/Oxen-AI/BitNet-1.58-Instruct

* https://github.com/nkotak/1.58BitNet

See some followups that has some training advice: https://github.com/microsoft/unilm/tree/master/bitnet

Der_Einzige• 2 years ago

Quantization is never free, and you can rest assured that even the "good" quants of the best models are highly crippled compared to their unquantized versions.

The "nearly as accurate" is only on their contrived benchmarks. I've never met a quantized model that actually behaved "98%" as good as the unquantized model, and I do LLM work daily and have since well before the ChatGPT era.

rockskon5• 2 years ago

I've never met an LLM that consistently behaved well at all. Quantized or unquantized.

Honestly it feels like the bulk of the industry is acting out one big LARP where they're always right around the corner from developing AGI and doing something big and amazing and....it just never materializes. Obnoxious AI agents, worthless AI-generated websites crowding out useful resources, unreliable AI search results.

The AI industry has done very well for itself selling hype. Now it needs actual good products.

hn_throwaway_993• 2 years ago

This makes 0 sense to me. I get tons of real world value and productivity benefits from AI, most specifically ChatGPT and cursor.sh.

I don't disagree that there is a ton of hype, a lot of it unwarranted, and I cringe when I see tons of companies trying to "throw AI against the wall and see if it sticks" (I personally nominate LinkedIn's AI blurbs on their feed as "most useless and annoying use of AI"). But still, I'm blown away with how much value I get from AI. It makes me a bit sad that so many of us have become so jaded that they see it as "one big LARP".

devbent• 2 years ago

Now it needs actual good products.

I asked chatgpt for recommended food pairings for a soup I made today. It did a great job.

Chatgpt also helped me debug a home automation issue I'd been having for over a year with some smart lights.

I find uses for chatgpt every day.

throw46365• 2 years ago

I had that feeling the other day, reading quotes from that OpenAI executive:

https://archive.is/LpDuJ

She said the executives told the board they "didn't think he was the right person to lead the company to AGI…”

The way I read it, it sort of sounded like you could substitute “AGI” with “Shangri-La”.

It’s always going to be just down the road, but they are sort of emotionally convinced that is where they are headed.

umanwizard3• 2 years ago

I’m blown away when people say stuff like this. GPT-4 makes me substantially more productive almost every day. It’s like we live in a different reality.

lucubratory• 2 years ago

Honestly it feels like the bulk of the industry is acting out one big LARP where they're always right around the corner from developing AGI and doing something big and amazing and....it just never materializes.

It's been less than two years since ChatGPT released.

segmondy3• 2 years ago

This is false. I have tested Q8s over f16 and the raw weights and pretty much seen no difference. I strictly run Q8s, I only keep the raw weights if I plan to do some fine tunes.

staticman2• 2 years ago

This matches the general consensus at the LocalLLama subreddit that Q8 is basically indistinguishable from the full model.

light_hue_12• 2 years ago

The performance loss from BiLLM is disastrous. It's basically useless. No one would ever want to use the resulting models. They hide their main results in the appendix: Table 8. page 15. https://arxiv.org/pdf/2402.04291

I won't go over the entire table in detail, but PIQA, BoolQ, HellaSwag, and WinoGrande should in the mid-to-high 70s for LLaMa2-7B. They drop that to 58, 62, 32, and 51. There are 700M parameter models that perform much better.

What they should have reported is effective number of parameters. Does LLaMa2-7B with their quantization method outperform a model that has amount of computation but uses that compute with say.. 16-bit quantization? If the answer is no, and it seems like it very clearly is, then the method is wholly worthless. Just use a smaller model to begin with.

The BitNet paper is better. But for some reason they only consider very small models. It's the obvious question and in their FAQ they don't provide a solid answer to it. Despite having all of the compute resources of MS. They could have easily run this experiment in the past year; I'm suspicious.

imtringued2• 2 years ago

BiLLM is about post training quantization and BitNet trains models from scratch. You do realize that one of those is going to give significantly worse results and the other is going to be significantly more expensive, well into the millions of dollars?

light_hue_12• 2 years ago

There's no mathematical reason why doing quantization after would be worse than training from scratch. That's nonsense.

There's also no practical reason. Training quantized networks is often harder! This is why people quantize after the fact or do distillation.

Nor is there any reason why we won't find some projection of weights onto the BitNet manifold.

If it was published by academics I'd believe the cost argument.

This was published by MS. They can run this experiment trivially. I have friends at MS with access to enough compute to do it in days.

Either the authors ran it and saw it doesn't work or they're playing with us. Not a good look. The reviewers shouldn't have accepted the paper in this state without an explanation of why the authors can't do this.

This is the question that determines if this work matters or is useless. Publishing before knowing that isn't responsible on anyone's part.

WiSaGaN• 2 years ago

This seems saying that the 1bit model is a bit better than GPTQ Q2. However, I find there are few situations where you would want to use GPTQ Q2 in the first places. You would want to run the F16 version if you want quality, and if you want to have a sweet spot, you usually find something like Q5_K_M of the biggest model you can run.

qeternity3• 2 years ago

Nobody is running llama.cpp in production…

luke-stanley2• 2 years ago

What do you mean? What makes you think that?

szundi3• 2 years ago

Probably more often than not

mlsu• 2 years ago

I don't believe that quantization comes for free. Someone made the observation that llama3 models quantize "worse" than llama2 models do -- they suffer in quality from quantization far more.

My intuition is that a model which is undertrained suffers less from quantization, because the training process has not utilized each weight to its full potential. One of the key findings with llama, and why it punches above its weight for its size, is that they trained it for longer on a much larger dataset then was "optimal" according to the literature up to that point.

Putting two and two together, it seems that:

small model, lots of data, long training > large model + quantization

That basically, quantization is a lossy shortcut to the tail of training long. Amount and quality of data is, as always, the most important part about all of this.

kir-gadjello• 2 years ago

While llama3-8b might be slightly more brittle under quantization, llama3-70b really surprised myself and others[1] in how well it performs even in the 2..3 bits per parameter regime. It requires one of the most advanced quantization methods (IQ2_XS specifically) to work, but the reward is a SoTA LLM that fits on one 4090 GPU with 8K context (KV-cache uncompressed btw) and allows for advanced usecases such as powering the agent engine I'm working on: https://github.com/kir-gadjello/picoagent-rnd

For me it completely replaced strong models such as Mixtral-8x7B and DeepSeek-Coder-Instruct-33B.

1. https://www.reddit.com/r/LocalLLaMA/comments/1cst400/result_...

d13• 2 years ago

How does it compare against unequalised Llama 3 8B at 16fp? I’ve been using that locally and it’s almost replaced GPT4 for me. Runs in about 14GB of VRAM.

renewiltord• 2 years ago

Holy wow. Thank you for this. Very cool. I’ve been using 8b for things it might be worth using 70b for.

endofreach2• 2 years ago

surprised myself and others[1] in how well it performs even in the 2..3 bits per parameter regime

I am too dumb for all of this ML stuff. Can you explain what exactly that means & why it's surprising?

acchow• 2 years ago

Quantization isn’t a shortcut. It’s a test.

More training leads to better compression. Trimming precision off the integers reveals that there was more compression that could be had.

raverbashing• 2 years ago

Quantization might need a 'relaxing' or 'normalizing' step before to improve results, it would be an interesting research area

Quantize each block or vector as a unit, bound to some metric instead of simply quantizing the whole thing all at once.

It's an interesting topic https://old.reddit.com/r/LocalLLaMA/comments/1ba55rj/overvie...

segmondy3• 2 years ago

depends on how far. Q8's are pretty much on par with f16/f32 weights. Q5/Q6's if they drop off barely drop below 2%. folks are running Q2, Q3, etc due to being GPU poor, of course it's going to be terrible. When you want to run a 70B model that's originally 140gb and all you have is 16gb of vram.

Xcelerate• 2 years ago

Has anyone attempted to compress network size further by extracting the symmetry invariants of the network—e.g., those that correspond to the permutation invariance of node shufflings that leave the DAG unchanged?

I did a rough calculation, and as the precision of the scalar weights decreases, the information content of the specific network permutation becomes a much higher percentage of its overall size.

Depending on the particular neural network architecture, there may be other symmetries beside the symmetric group that also represent compressible redundancies.

renonce5• 2 years ago

For a 4096x4096 matrix its symmetry group has size 4096!. The Stirling’s approximation gives ln(4096!)=4096ln(4096)-4096 which is about 10~11 bits per 4096 numbers. This is less than 0.003 bit per parameter saved.

roschdal• 2 years ago

"Nearly accurate" as in "incorrect"

CGamesPlay• 2 years ago

Don't worry, the LLMs were only "fairly accurate" (as in "incorrect") to begin with.

akira2501• 2 years ago

The lesson being... We spent billions for incremental gains masquerading as tectonic shifts. Whoops.

13alvone3• 2 years ago

I've been thinking about how far we've come with large language models (LLMs) and the challenge of making them almost perfect. It feels a lot like trying to get a spaceship to travel at the speed of light.

We’ve made impressive progress, getting these models to be quite accurate. But pushing from 90% to 99.9999999% accuracy? That takes an insane amount of data and computing power. It's like needing exponentially more energy as you get closer to light speed.

And just like we can’t actually reach the speed of light, there might be a practical limit to how accurate LLMs can get. Language is incredibly complex and full of ambiguities. The closer we aim for perfection, the harder it becomes. Each tiny improvement requires significantly more resources, and the gains become marginal.

To get LLMs to near-perfect accuracy, we'd need an infinite amount of data and computing power, which isn't feasible. So while LLMs are amazing and have come a long way, getting them to be nearly perfect is probably impossible—like reaching the speed of light.

Regardless I hope to appreciate the progress we've made but also be realistic about the challenges ahead. What do you think? Is this a fair analogy?

jjayj2• 2 years ago

I think I agree with your analogy, but would say 99% rather than 99.99999%.

Beyond that, I'm not entirely sure what a "perfect" LLM would even be defined as.

13alvone3• 2 years ago

That makes sense, and just harder and harder to get more accurate. Same as humans I supposed :)

torginus2• 2 years ago

But it still might be worth it. A 90% accurate model will only successfully complete a task consisting of 10 subtasks 0.9^10 = 35%of the time, while a 99% will do so 90% of the time making the former useless, but the latter quite useful.

CuriouslyC9• 2 years ago

Yes, but a 90% accurate model that's 10x faster than a 99% can be run 3x to achieve higher accuracy while still outperforming the 99% model, for most things. In order for the math to be in the big model's favor there would need to be problems that it could solve >90% of the time where the smaller model was <50%.

wruza• 2 years ago

There’s also a rumor that models these days employ a large “safety” parachute behind their engines all the time. Some of these get so big that models become dumber right before your eyes.

kreyenborgi• 2 years ago

"Data" isn't an inexhaustible resource, and also isn't fungible in the way energy is. Of the thousands of languages in the world, a fair chunk don't even have writing systems, and some have very few speakers left. Many are lost forever. Now ask the best llm trained on "all the data" to translate some fragment of some isolate language not in its training set and not very related to existing languages. You can't improve on that task by adding more sentences in English or by combining with learning on other modalities.

OmegaPoint• 2 years ago

"Data" isn't an inexhaustible resource

Synthetic data are the answers. For example see Tiny Stories dataset (https://arxiv.org/abs/2305.07759).

Now ask the best LLM trained on "all the data" to translate some fragment of some isolate language not in its training set and not very related to existing languages.

If you give them the dictionary and grammar book as in-context instructions, it can do pretty well.

“Gemini v1.5 learns to translate from English to Kalamang purely in context, following a full linguistic manual at inference time. Kalamang is a language spoken by fewer than 200 speakers in western New Guinea. Gemini has never seen this language during training and is only provided with 500 pages of linguistic documentation, a dictionary, and ~400 parallel sentences in context. It basically acquires a sophisticated new skill in the neural activations, instead of gradient finetuning.”

CuriouslyC9• 2 years ago

Yes and no. We don't need an insane amount of data to make these models accurate, if you have a small set of data that includes the benchmark questions they'll be "quite accurate" under examination.

The problem is not the amount of data, it's the quality of the data, full stop. Beyond that, there's something called the "No Free Lunch Theorem" that says that a fixed parameter model can't be good at everything, so trying to make a model smarter at one thing is going to make it dumber at another thing.

We'd be much better off training smaller models for specific domains and training an agent that can use tools deepmind style.

ben_w4• 2 years ago

The problem is not the amount of data, it's the quality of the data, full stop. Beyond that, there's something called the "No Free Lunch Theorem" that says that a fixed parameter model can't be good at everything, so trying to make a model smarter at one thing is going to make it dumber at another thing.

My understanding is NFL only applies if the target function is chosen from a uniform distribution of all possible functions — i.e. the "everything" that NFL says you can't predict is more like "given this sequence from a PRNG (but we're not telling you which PRNG), infer the seed and the function" and less like "learn all the things a human could learn if only they had the time".

drchickensalad• 2 years ago

Yeah LLM's are just a nontrivial stepping stone. Humans don't need to consume the entire set of worlds knowledge, repeated from thousands of different mouths coming from different angles, to be able to learn to output human like thought processes.

At some point we'll discover a mew algorithm/architecture that can actually continuously learn from its environment with limited information and still produce amazing results like us.

kjhcvkek77• 2 years ago

Well let's not forget that the large amount of information they ingest also leads to a superhuman level of knowledge though I guess for certain kinds of agents that is not really needed anyway.

2OEH8eoCRo0• 2 years ago

I've said this before but (as a noob) I don't think cramming all human knowledge into a model is the correct approach. It should be trained enough to understand language so that it can then go search the web or query a database for answers.

neolefty3• 2 years ago

The more certain the domain, the more that is possible. If you have a document database that you trust, great. For example a support desk's knowledge base. And especially if you have an escape valve: "Did this solve your problem? If not, let's escalate this to a human."

But if you are searching the Internet, you'll find multiple answers — probably contradictory — and the next step is to ask the model to judge among them. Now you want all the intelligence you can muster. Unless you really trust the search engine, in which case yeah a small model seems great.

darkerside2• 2 years ago

If people aren't speed of light perfect (and they're not), why could a computer be? What does perfection even mean?

Workaccount2• 2 years ago

I think you are probably right, but if humans are at 99.9% (which seems very unlikely) I don't think it will be long before you can trust a model more than a human expert.

Really though I think this line of thinking is better to revisit in 5 or so years. LLM's are still very new, and seemingly everyday new optimizations and strategies are being found. Let's at least hit a plateau before assessing limitations.

lambdaxyzw• 2 years ago

I don't think it will be long before you can trust a model more than a human expert.

You will never be able to trust a LLM more than a human expert. Because human expert will use the best available tools (for example, LLMs), will understand "what the client wants" and will put the data in the right context. At best human expert and LLM will be indistinguishable, but I really doubt it. And I think it will take a long time.

At least it's my opinion, we'll see what happens.

usgroup2• 2 years ago

Is there some work on how small a model with some specific epsilon perplexity could theoretically be? Given a fixed architecture and a fixed dataset, I presume there is a minimal number of parameters required for optimal representation.

renonce5• 2 years ago

If you are referring to what is theoretically possible with arbitrary computation in the model, it's called Kolmogorov complexity and it's not computable.

usgroup2• 2 years ago

With a fixed architecture, and a fixed dataset, as mentioned. So, a specific kind of neural network, and a fixed dataset.

chx• 2 years ago

Nearly as accurate? I guess zero is close to zero.

https://hachyderm.io/@inthehands/112006855076082650

You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.
Alas, that does not remotely resemble how people are pitching this technology.

Last5Digits• 2 years ago

Are you going to spam this same link in every single thread about LLMs on HN? People have provided good arguments refuting whatever you're trying to say here, but you just keep posting the same thing while not engaging with anyone.

bambax• 2 years ago

I'm surprised the comments here are so negative / cynical. Of course "quantization doesn't come for free": it's a trade off.

High precision models are very expensive to run: they require expensive hardware and lots of energy.

Isn't it possible, or even likely, that low-precision models would be good enough for many tasks?

az09mugen• 2 years ago

I totally agree with your statement. I consider LLMs imprecise in any case, it's not a perfect/exact science, just statistics. I only use LLMs for tasks where an error margin can be allowed.

In that perspective I'm totally fine with using a 5 MB LLM like this one : https://neuml.hashnode.dev/train-a-language-model-from-scrat...

kromem3• 2 years ago

Post-training quantization doesn't come for free, but the pretraining on constrained precision weights actually counterintuitively results in a performance increase per parameter as the number of parameters grows in the ternary BitNet paper.

Even if there was zero efficiency gain in ternary weights, large models should probably be trained on a network of precision limited weights from here on out given the research so far.

I suspect it relates to each weight relating to multiple 'features' in the network. The greater the precision, the more room it gives for competing features to compromise on node values that aren't best for either feature instead of reorganizing the feature mapping to avoid conflicts.

trashtester2• 2 years ago

The number of bits used per weight during training, could be included in the regularization, perhaps?

For instance, one could extend dropout regularization to several levels, where each weight could have random chances to include the most significant 2-16 bits part of the time (and still 0 part of the time), and where the impact on the gradient of having fewer bits could be used to tune the ideal number of bits for each weight.

Then one could add L1 regularization for the total number of bits used to squeeze the total down down to whatever size one aims for.

k__• 2 years ago

lol, just like with humans.

demosthanos• 2 years ago

I'm surprised the comments here are so negative / cynical.

And by "the comments" I'm assuming you mean the top comment [0] (and maybe its replies)? The rest don't really come off as negative at all, and you quoted directly from that top comment.

FWIW, I don't find that comment either negative or cynical. It starts out with that sentence you quoted, but it goes on to make a very interesting point about quantization most likely working best for models which are undertrained—models which store less information than their architecture size would suggest. That's a very valid point that I found insightful and interesting, not cynical.

[0] https://news.ycombinator.com/item?id=40531638

jononor2• 2 years ago

Several years of ML research for CNNs indicate at least that one can do very well with 8 bit integers. Such quantization is basically standard now for any deployment outside of GPU (where 8bit isn't any faster anyway due to the hardware).

renonce5• 2 years ago

I study LLM quantization and I have surveyed GPTQ and QuIP# and lots of quantization algorithms (specifically PTQ, post-training quantization) to develop my own, and my experience has led me to become extremely skeptical of many of the papers.

I've seen lots of headlines like "1-bit quantization" (including this one and https://arxiv.org/abs/2310.16795). What I've found in this space is that the headlines can often be intentionally misleading about what is actually achieved. If you read closer the abstract of this paper, it claims 8.41 perplexity on LLaMA2-70B at 1 bit, which is a HUGE decrease from 3.120 perplexity in FP16 and they will never mention that in the headline. Even LLaMA2-7B at INT8 achieves 5.677 perplexity with half of storage place (better with LESS space and LESS training). Some claim 1.58-bit quantization (each weight is either -1, 0 or 1) but in practice require very small group sizes, which means one or two extra FP16 numbers for every 64 weights and that means another 0.5 bit, so it's actually 2-bit quantization. And every quantization algorithm can claim they make language models smaller, speedier, and use less energy, so there's nothing special about these.

Here are the key metrics that I suggest checking when comparing quantization schemes:

* Perplexity. Note that it also depends on the dataset (either WikiText2 or C4, WikiText2 numbers are usually lower than C4) and context size (1024, 2048 or 4096, higher context sizes usually means less perplexity). Dataset and context size must match to make a meaningful comparison.

* Quantization bits. Many algorithms claiming 2-bit or 1-bit quantization has lots of extra parameters elsewhere, such as grouping. Download the quantized version of the model, check its file size, multiply by 8 and divide by the number of parameters. That gets you the ACTUAL quantization bits.

* Performance. Weights may need to be dequantized during inference which could introduce overhead. Some libraries have custom matmul kernel for dequantization that achieves performance close to FP16, others can be slower. Check its generation speed and inference speed.

Newer architectures such as Ampere contains INT8 cores which may make quantized version even faster than FP16, I haven't tried out yet.

There is also a lot of misleading comparisons in this space. Some methods like GPTQ only provide vector-matrix multiplication kernels, which means only a single token can be generated and batched inference (which is needed for generating initial KV cache, or for serving multiple users) can be much slower. If an algorithm claims a 3x speedup for something, check if they refer to single stream latency or multiple stream throughput. Some of that speedup comes from running a model on 2 cards instead of 5 cards without specifying if the cards have NVLink configured (you shouldn't run inference on multiple cards without NVLink, or you should expecet huge slowdown simply because of using 5 cards).

* Base model. Pick a STRONG base model like Llama-2-7b or Llama-3-8b etc. Not an undertrained model like SwitchTransformer etc which may have lots of redundant parameters in itself.

My personal favourite remains QuIP# (https://github.com/Cornell-RelaxML/quip-sharp). It lacks in the "performance" part as its matrix multiplication performance isn't on par yet but there is room for improvement, and it wins every other metric. And sad news: it's very likely we won't have practical 1-bit LLMs, never ever. We are reaching the end game between 2.5~4 bits. By "practical" I mean it should beat 3-bit LLMs with 3x less parameters or 2-bit LLMs with half as many parameters. There is a Shannon limit to quantization whatever methods you use.

kromem3• 2 years ago

Completely agree on PTQ, but curious on your thoughts for QAT, specifically BitNet 1.58 - in that paper it looks like parameter to parameter the constrained precision weights had improved perplexity vs floating point weights, particularly as the model size increased.

While I'd love to see it scaled up to at least ~50B models, it looks like limited weight precision might actually offer improved network optimization over unconstrained weights for pretraining.

Do you think that work is misrepresenting the gains, or that QAT is a different beast where quantization isn't as much a tradeoff as a potential net gain across the board?

renonce5• 2 years ago

Can't speak for QAT as I haven't yet dived into that area. I've quickly skimmed the BitNet and BitNet 1.58 paper. I think achieving comparable performance with a Llama model with the same number of parameters is impressive but unfortunately it seems they didn't release the training code so I can only tell from their paper. Fortunately they did talk about training details in the BitNet paper (not in BitNet 1.58 so I assume they remain the same):

Mixed precision training. While the weights and the activations are quantized to low precision, the gradients and the optimizer states are stored in high precision to ensure training stability and accuracy. Following the previous work [LSL+21], we maintain a latent weight in a high-precision format for the learnable parameters to accumulate the parameter updates. The latent weights are binarized on the fly during the forward pass and never used for the inference process.

In this case there are two areas to optimize for: training efficiency and inference efficiency.

If I understand correctly, it stores the weights, gradients and second-moment estimates in FP32 like every other mixed-precision training (the Gopher paper has details on why storing them in FP32 is important), and quantized weights are used in forward pass. What I'm not sure is whether latent weights are used in backward pass, and my instinction is that the "Straight-through estimator" requires high-precision latent weights so they may still be needed. Training FLOPS can be roughly estimated as 6 FLOP per parameter per token, where 2 is forward pass, 2 is gradient computation and 2 is gradient accumulation (see https://medium.com/@dzmitrybahdanau/the-flops-calculus-of-la...). If only forward pass is quantized, this means only 1/3 of all FLOPS are optimized (and even then it has to be accumulated in FP32). So I'm skeptical of the gains in training efficiency here, and I can't find the numbers (how much energy or how much time is used for training, compared to regular FP16 mixed precision training? The papers boast inference energy savings which makes me even more skeptical of training energy savings)

For quantization efficiency, while QAT can certainly avoid the quantization step, PTQ methods are very cheap (usually <24 hours on RTX 4090 for Llama-2-70b) so I consider the cost of the quantization step negligible. There is not much difference in inference efficiency gains as PTQ and QAT can quantize to the same format. For final accuracy, unfortunately there is a lack of comparison between QAT and PTQ of fp16 models, and PTQ has the advantage of not requiring access to the original dataset, so I think it's very hard to make a fair comparison here but it's also likely the only area where QAT has actual gains compared to best PTQ methods.

regularfry2• 2 years ago

Just on your very last point, I think you've nailed why a 1-bit quant of a bigger LM can't beat 3-bit quants of an LM a third the size, if what you mean is that more extreme compression of a LM is more likely to introduce harmful artefacts, so you need a better quantisation method to produce 1-bit than you do at 3-bit to end up with a model with the same information content retained.

What I don't think that tells us anything about is directly trained 1-bit LMs versus 3-bit LMs, because in that case there's no compression step to introduce quantisation artefacts. There might be an analogous training data size argument but it's not clear to me that there needs to be: a 3X parameter 1-bit LLM and a 1X parameter 3-bit LLM ought to be equivalent in terms of their information capacity.

Decabytes2• 2 years ago

I feel like we are getting closer and closer to a finding the Goldilocks Llm, that with some smarter training and the right set of parameters, will get us close to got 3.5 turbo performance but at a size, cost, and time effort that is significantly lower and that is runnable locally.

Combine that with what seems like every chip adding a neural engine and it feels like we are in the early days of high performance graphics again. Right now we are in the unreal engine voodoo era, where graphics cards/neural engines are expensive/rare. But give it a few generations and soon we can assume that even standard computers will have pretty decent NPUs and developers will be able to rely on that for models

moffkalast2• 2 years ago

3.5-turbo performance wasn't very good though, and according to API statistic analysis it's a Nx7B model so it's already rather small. Ultimately Llama-3-8B is already better in all measurable metrics except multilingual translation, but that's not saying much.

djeastm• 2 years ago

3.5 turbo performance

Is this the level of performance people are relying upon? While I've always been impressed with the technology itself, it's only starting with GPT 4 that I think it approaches adequate performance.

Decabytes2• 2 years ago

For the work that I do (which is mostly rag with a little bit of content generation) GPT 3.5-turbo-0125 with a 16k context window is the sweet spot for me. I started using the api when it was only a 4k context window, so the extra breathing room provided by the 16k context window feels cavernous. Plus the fact that it's $0.50 per 1Million tokens means that I can augment my software with LLM capabilities at a cost that is attractive to me as a small time developer.

The way I rationalize it is that using 3.5-turbo is like programming on an 8-bit computer with Kilobytes of Ram, and gpt-4o is like programming on a 64bit computer with a 4080 ti and 32gb of ram. If I can make things work on the 8-bit system, they will work nicely on the more powerful system.

segmondy3• 2 years ago

yeah, it's called phi-3-medium-4k-instruct

https://huggingface.co/microsoft/Phi-3-medium-4k-instruct

moffkalast2• 2 years ago

It's not called anything until the lmsys leaderboard ranks it. Microsoft's blatant benchmark overfitting on Phi-2 makes for very little trust in what they say about performance. As a man once said, fool me once, shame on you, fool me twice-can't get fooled again.

Gaazrukk• 2 years ago

.999936565

bunderbunder• 2 years ago

They keep calling them 1-bit LLMs, but they're really 1-trit LLMs. If you can have 3 states, it's not a BInary uniT, it's a TRInary uniT.

I don't think that this is just a nit, it implies a real mismatch between how these models work, and how modern computing hardware works. People have built ternary computers in the past, but to the best of my knowledge nobody's made a serious attempt at it in half a century.

You can always use two bits to store a ternary value, of course. But then it wouldn't be a 1-bit LLM, now, would it? And that doubling in compute resources required would make a tangible difference in how one has to think about potential efficiency improvements. Also, ternary tritwise logic implemented on binary hardware is unlikely to be anywhere near as efficient as binary bitwise logic on hardware that was specifically engineered for the purpose. This leaves me thinking that the research teams involved continuing to refer to these as 1-bit LLMs must be interpreted as knowingly engaging in dishonest marketing tactics.

freeone3000• 2 years ago

It’s stored in and calculated using a floating-point value, and uses log(2) bits. So it’s ~1.2 bits with float packing over 32m parameters.

IanCal• 2 years ago

I don't think it's quite so clear cut, and the papers and names are often a bit more precise anyway.

BitNet is (I think) actually one bit per parameter. BitNet 1.58b isn't, but then to be fair isn't describing itself as a 1 bit llm. I'm less sure but it seems OneBit is 1 bit. One of the processes mentioned in the article is a mix of one and two bits for the weights.

orlp• 2 years ago

Perhaps in the future we will see MsoTriState neural networks: https://learn.microsoft.com/en-us/dotnet/api/microsoft.offic...

For those unaware, the MsoTriState is fairly self-explanatory: it is a tri-state boolean type with five possible values.

neolefty3• 2 years ago

The article describes both. The 1,0,-1 networks are "1.58 bits" — that is, log2(3). But yeah it mostly focuses on 1-bit networks.

corysama• 2 years ago

Not long ago I managed to get someone working on sub-1-bit models to come out of the woodwork.

https://news.ycombinator.com/item?id=39865855

celltalk• 2 years ago

You all run on 2-bit (DNA) and you all seem pretty stable to me. You run your brain with couple of watts per day and add big numbers without much hassle.

neolefty3• 2 years ago

DNA could also be considered 6-bit, since codons are 3 base pairs: https://www.genome.gov/genetics-glossary/Codon

JoeAltmaier• 2 years ago

Lots of folks have the intuition that, 'slightly worse' LLM models means unacceptable rates of nonsense answers.

What's an acceptable rate? Is it 99%? 99.9%?

The closer it gets to 99.999% good answers, the more damaging the wrong ones become. Because people have been trained, too. Trained to trust the answers, which makes them lazy and vulnerable to lies.

gslin• 2 years ago

This makes me watch https://en.wikipedia.org/wiki/11001001 again.

sleepybrett• 2 years ago

smaller bucket of shit just as effective at getting you stinky as a large bucket of shit

blurbleblurble• 2 years ago

I know there are 1-bit experiments out there but from what I can tell it's BitNet 1.58 that's really exciting.

So why do people keep focusing on "1-bit" when the whole reason 1-trit models are so successful in the first place just might have everything to do with the ternary weights and the symmetry they're able to encode?

I'm not sold on the suggestion that "imprecise language models" in general are "nearly as accurate" when it might actually be that ternary weights are more precise in a totally difference sense: in that they just might be capturing a minimal representation of the very symmetries responsible for making all those parameters effective in the first place.

lavelganzu• 2 years ago

Make a tiny language model and then use it for autocorrect & autocomplete. Current autocorrect & autocomplete on smartphones is so bad that it spawns countless jokes. This use case doesn't require a large language model able to write whole paragraphs -- it only needs a model that's just barely big enough to make contextually appropriate suggestions of one or a few words.

“Imprecise” language models are smaller, speedier, and nearly as accurate

Related Stories