HN user

Kostic

487 karma
Posts3
Comments107
View on HN

Sure but that holds true for most new tech. Expensive and slow but it can only get better or stay the same. I'm thinking it's only going to get better.

Auth with Github Copilot and then point it to localhost[0]. Hopefully the auth to Copilot requirement will be dropped for local models at some point. Would love to use a fully open stack (VSCodium and everything) in the future. My config:

``` [ { "name": "http://127.0.0.1:8888/v1", "vendor": "customendpoint", "apiKey": "llama.cpp", "models": [ { "id": "gemma4-31b", "name": "Gemma 4 31B", "url": "http://127.0.0.1:8888/v1/chat/completions", "toolCalling": true, "vision": true, "maxInputTokens": 65536, "maxOutputTokens": 8192 }, { "id": "qwen3.6-27b", "name": "Qwen 3.6 27B", "url": "http://127.0.0.1:8888/v1/chat/completions", "toolCalling": true, "vision": true, "maxInputTokens": 180224, "maxOutputTokens": 8192 } ] } ] ```

[0] https://code.visualstudio.com/blogs/2025/10/22/bring-your-ow...

For personal needs I connected VSCode with llama.cpp running Qwen 3.6 27B or Gemma 4 31B and it's good enough to cancel my cloud subscription.

Qwen running on my 1st GPU at q4@176k context from 70 to 50 tok/s with MTP, pretty good for coding.

Gemma on the other hand is using both GPUs, running q8@64k context, doing document sentiment analysis, summarization, proofreading and translating, at consistent 25 tok/s. Somewhat slow but usable for batched workflows. Might get some more once llama.cpp starts supporting MTP with tensor split mode.

Still using frontier LLMs at dayjob since I'm not paying it and those are obviously better. Hopefully we'll have a Sonnet 4.6/Opus 4.5 level 30B model in a year or so.

EDIT: Prompt processing starts from 800 t/s and drops to 400 t/s. In most cases my starting prompts are around 16k-24k of tokens and require from 60 to 90 seconds to be processed. Not great but acceptable.

"I'm sorry" is not gaslighting but an admission of fault it learned from our texts. And if an LLM managed to delete your database, it's time to slow down the vibe train and put up some guard rails.

LLMs are awesome but not without supervision.

Taalas showed that you could make LLMs faster by turning them into ASICs and get 10k+ token generation. It's a matter of time now.

The MacBook Neo 4 months ago

For now. These will be pretty cool Linux machines if Asahi starts supporting them at some point.

Claude 4 1 year ago

It's relevant from an engineering perspective. They have a way to develop a new model in months now.

I've stopped using search engines actively 18 months ago. My first stop is an LLM. Once I understand what I actually need, I do a web search to go to the product/tool website. I do this not because LLMs are that good but because web search result quality went way down in the same period.

One interesting trend that I like is that I started using local LLMs way more in the last couple of months. They are good enough that I was able to cancel my personal ChatGPT subscription. Still using ChatGPT on the work machines since the company is paying it.

Starship Flight 7 2 years ago

On a long-enough scale, this planet will die anyways. Let's first try to expand life and unlock the secrets of the Universe, as much as possible.

Default Ubuntu is still pretty good. I have been using the default Fedora Workstation for almost a decade and it works great for me. I also don't spend time on tweaking it anymore (used to be the complete opposite). The defaults work, I concentrate on my work, side projects and games.

Just after I moved my Cloudflare task to `in progress`.

Folks who manage their Cloudflare stuff via Terraform, are you able to make changes? Or does this mean that their API is also down?