om8

This project needs webgpu -- I did it on cpu about a year ago.

My demo uses 2 bit quantization to run llama3 models on any device with enough ram.

https://galqiwi.github.io/aqlm-rs/

manufacturing companies making the flimsiest, cheapest, plastic crap to save 1/3 of a cent on every mop they produce. Designed to work for the least amount of time before needing replaced

We live in a world with such companies, and we can still buy quality things. If there is a demand for the purely-human generated texts, they will be around. Perhaps a lot of people around you will read ai text instead, and you'll get upset because of it, but it's their choice. You'll still have your thing

Oh, yes, you are right about watts. My bad

volt-amperes are technically the same unit as watts

volt-amperes are joules

Cool way to think about GWh/year:

  1 GWh/year = (10 ** 9) / 24 / 365.25 / (10 ** 6) MW = 0.11 MW

  70 GWh/year = 8 MW
  1755 GWh/year = 200 MW
  252 GWh/year = 29 MW

cargo/uv/go have lock files though

https://docs.vllm.ai/en/v0.20.0/api/vllm/model_executor/laye...

`vllm.model_executor.layers.quantization.turboquant`

The technique implemented here consists of the scalar case of the HIGGS quantization method (Malinovskii et al., "Pushing the Limits of Large Language Model Quantization via the Linearity Theorem", NAACL 2025; preprint arXiv:2411.17525): rotation + optimized grid + optional re-normalization, applied to KV cache compression. A first application of this approach to KV-cache compression is in "Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models" (Shutova et al., ICML 2025; preprint arXiv:2501.19392). Both these references pre-date the TurboQuant paper (Zandieh et al., ICLR 2026).

These are very different media types with very different goals.

Is there a way to disable it? Sometimes I value agent not having knowledge that it needs to cut corners

https://steveklabnik.com/writing/does-unsafe-undermine-rusts...

... Yet.

FYI you can run just `uvx pdm`

Also got it, found this thread by googling "ycombiinator"

Also got it, found this thread by googling "ycombiinator"

Have a similar project. Also written in rust, runs in a browser using web assembly

In-browser demo: https://galqiwi.github.io/aqlm-rs

Source code: https://github.com/galqiwi/demo-aqlm-rs

https://web.archive.org/web/20250914222638/http://ewaste.fka...

Oh, ok. Thanks, I agree

Why? They acquired books, that’s what they do

Good rationalism includes empiricism though

llama.cpp is a mess and ollama is right to move on from it

Of course it is. GPT-5 is one of the most anticipated things in AI right now. To live up to the hype, it needs to be a reasoning model.

It’s unfortunate that llama.cpp’s code is a mess. It’s impossible to make any meaningful contributions to it.

Sure, but integrated graphics usually lacks vram for LLM inference.

To have a gpu inference, you need a gpu. I have a demo that runs 8B llama on any computer with 4 gigs of ram

https://galqiwi.github.io/aqlm-rs/about.html

I'm currently in Russia trying to get a US visa for my CS PhD. Because I do CS, I got into a thing called administrative processing. For 95% of people, it takes days -- weeks, tops. Because of the colour of my passport, it has already lasted for 3 months. I know people who are waiting for 2 years to pass it.

Why do you think I shouldn't have access to this website in Russia?

I believe he knows

What would you do if some of the deps started to have conflicts in them? Also, what are your plans for migration when you'll need to move from one os version to another?

Implicit solutions like yours have lower cost of entrance, but larger cost of support. uv python scripts just work if you set them up once

I have a demo that runs llama3-{1,3,8}B in browser on cpu. It can be integrated with this thing in the future to be fully local

https://galqiwi.github.io/aqlm-rs

a local LLM is going to be the way to go

Non-technicals don't know how LLMs work, and, more importantly, don't care about their privacy.

For a technology to be widely used, by definition, you need to make it appealing to the masses, and there is almost zero demand for private LLM right now.

That's why I don't think that local llms will win. There are narrow use cases where regulations can force local llm usage (like for medical stuff), but overall I think that services will win (as they always do)

50 tokens is not really very much Yes! And also llama3.1’s tokens are different from Qwen and llama1 tokens. That’s the first model where meta started to use very large vocab_size.