senko

FWIW my observation was about the models I tested, but I can see how it could be taken as a general statement.

For the web app task I mentioned:

* Kimi K3: 9532k input (9172k cached), 114k output - cost $5.5

* Qwen 3.8 Max: 18020k input (17836k cached), 114k output - cost $6.3

* Fable: ~14m input (all cached??), 196k output - cost $30

Correction on my earlier post, Kimi was through Pi, not Kimi Code. For Qwen I used Qwen Code and for Fable I used Claude Code.

Not sure wtf is going on with the Fable stats (a lot tokens, virtuall all of them were cached - I guess heavy system prompt?) but both claude code stats and ccusage tool output match.

Having tested K3, Qwen 3.8 max preview, Fable and Sol for the past few days, I partially agree. Don’t trust the benchmarks, and the Chinese models really are slow and token-inefficient.

However they do seem very close to SOTA : I’d say roughly equal to previous gen (Opus 4.8, GPT 5.5). It’s yet another silly benchmark, but compare them here: https://senko.net/vibecode-bench/

I also had K3, Qwen3.8 and Fable (using Kimi Code, Qwen Code and Claude Code harnesses respectively, and the official APIs) create a simple but far from trivial web app (zero shot, from a detailed spec). In user testing all three results looked/behaved more or less the same. I had Sol (via codex) do code reviews on all three and it concluded all three were solid, with some room for improvement. Fable was slightly ahead of the pack.

In my own work I still prefer Opus 4.8 (until Antrophic come to their senses and allow 100% Fable usage on Max plans) and Sol, but if I had to find an alternative, I could live with both K3 and Qwen3.8 just fine.

Here it is: https://senko.net/vibecode-bench/2026/rts-gpt-5.6-sol.html

Clearly much better than the Terra version. I'd say its on par with Fable, and the observed differencies are more due to random luck and open-ended prompt, rather than model capability. (Edit: after some more testing, perhaps not on par - somewhere between Opus and Fable, is a better description).

Fable did better pathfinding and has more terrain variety, visually the map looks better, especially soft edges of the fog of war. And the enemies.

Sol took more care with tiny ux details, added help, and more building varieties.

Here's the prompt I used:

Create a simple but functional real time strategy (RTS) game similar to old WarCraft, StarCraft or Command & Conquer games. The player should be able to build buildings, create units, gather resources and should uncover the whole map. No AI or multiplayer needed. Use simple but nice-looking graphics. No sound. Implement everything in HTML/CSS/JS, everything in a single file (you can use 3rd-party js or css libraries/frameworks via CDN).

I'd say Fable 5: https://senko.net/vibecode-bench/2026/rts-fable-5.html

It even has enemies! (I'm not too mad about it not following my instructions because it can be fun to play :) And I generated that from Claude Code on my phone.

Sonnet 5 also produced a pretty nice version. You can see all of them here: https://senko.net/vibecode-bench/

Well, it is a silly test, not a scientific benchmark.

However, I would say it is a measure (not the measure). If you look at the entries, there's a lot of variation - definitely not something they memorized outright.

And the test itself is deceptively simple. You need to do canvas rendering, there's pathfinding, command queueing, terrain generation, etc. There are some subtle click handler bugs (various LLMs often stumble on those). And I ask the model to do it all in one file, further increasing the complexity of the task.

And the result is something that you can instantly evaluate. And if the result is any good, even play! So yeah, I think it's a fair test.

I'm sure it'll get saturated at some point. Actually I started with Minesweeper and switched to RTS last December, because Minesweeper was being saturated. I'm expecting (hoping?) the RTS test will last until the end of this year...

You guys had filesystems??

I love testing the new models by asking them to code a toy RTS game. Here's what Terra did: https://senko.net/vibecode-bench/2026/rts-gpt-5.6-terra.html (one try, in codex app, xhigh effort)

Comparing this to other models, I find it similar to GPT-5.5 and a bit behind Sonnet 5. You can see how other models fared here: https://senko.net/vibecode-bench/ (you can also fetch the prompt and the the 5.6 Terra resulting code on from that page).

I don't have access to Sol yet (on a Plus sub, which should get it according to what I've read), so can't do the more interesting test. I'll update the above page as soon as I get access - hopefully soon.

'fraid so, the odds have risen too high.

We might need a white night on a fiery steed. One can dream.

I feel like EU could start a company,

That's not how market-based economies work...

feed 2bln a year into it and make a compelling almost SOTA model

...and the reason is, if you give a bunch of people €2b a year and tell them "go try and make something", they'll make a ton of paperwork covering their asses and very little actual output.

This is irrespective if those people are European ("european google killer"), American ("cost plus" old US aerospace companies) or Chinese (which is why they do it a little different).

If there are no incentives to really try really hard, they won't do it.

In many high-tech cases in Europe, the formula for "let's subsidise the hell out of research and hope a commercially-viable business comes out" has a really poor track record.

Your second option - and possibly the best bet - is to find an existing company that already showed they're capable, and shower them with money, which is what French are doing with Mistral.

The post mainly talks about coding from security point of view. Fair enough.

In my own (limited) testing so far, Fable is the most capable model (for coding in general), and the most expensive.

It pretty much saturated my "LLMCraft" benchmark to implement a mini RTS: https://senko.net/vibecode-bench/2026/rts-fable-5.html (prompt and results for other models here: https://senko.net/vibecode-bench/ )

That said, combined with workflows and high thinking effort, burns through tokens (and money) at an alarming rate.

It may be too good (snd too expensive) for most tasks - using it alongside cheaper models for grunt work is probably the winning strategy.

LinkedIn in particular is quite aggressively blocking any automated attempts to read or navigate through it.

I post quite a lot there and wanted to have a copy of my posts on my blog[0] to preserve them. For a few months I was able to use a headless browser + claude code, then LI wised up and started logging it out, so I had to use a regular Chrome, log in manually and then tell the LLM to take over and slowly go through my feed.

If you're accessing sites which are not actively blocking bots, or - gasp - have an API, it's much better.

[0] example: https://blog.senko.net/may-quick-takes

Kagi's only office/hub is in Belgrade, which may not be EU, but it's (literally) close enough. Employees are remote.

Freediver (founder) is US based and Kagi is an US entity, so must comply with any warrants there.

But I guess they could set up a Serbian subsidiary?

Yeah, I agree 24B-36B sizes are better in general.

I don't have unified RAM tho and offloading to CPU is dog slow, which is why I'm interested in 7b-12b models.

Good catch. I haven't looked deeply into it. This is with Vulkan backend on Linux which I understand should be roughly comparable to CUDA? Gfx is rtx 3060(ti?).

I should play a bit more with llama.cpp options and see what bappened there. Thanks!

I ran the Q4 quant (used with llama.cpp) though my "minesweeper" vibe-coding benchmark: https://senko.net/vibecode-bench/2026/minesweeper-gamma-4-12...

The result is decent, but it had a few bizzare/trivial syntax errors I had to fix manually: it would do an extra closing bracket or paren a few times, and wanted to separate function definitions with comma. Not sure what that was about, but otherwise the output run just fine.

So, with those qualifiers, I think it's a decent local coding model. It roughly compares with GPT-4.1 (!!), released 14 months ago, on the output: https://senko.net/vibecode-bench/2025/minesweeper-gpt-4.1.ht... (actually I'd call it better, but those syntax errors...)

I ran the quantized version (4-bit GGUF) on my consumer-grade card with 12G of VRAM and got 5t/s for output. Not for interactive use for coding, but fairly capable model.

To me, it's fascinating how much progress we got in over a year. GPT-4.1 was considered an extremely capable coding model. Now we got something with 12B of params performing roughly the same (in this specific benchmark, disclaimers, etc).

Lists of various models I tested: https://senko.net/vibecode-bench/

Europe definitely has a lot of problems! Some are similar, others are different than in the US.

Just comparing income, even on purchasing-parity basis, doesn't cut it.

Yeah, it's weird, nobody's saying "we should make all the data centres use closed loop cooling even if it's more expensive for them!", but a lot of voices are yelling "AI uses water!", referring to the same thing.

I mean, email and Hacker News and Netflix use water, too.

This post, like many others, confuses AI with Big Tech (or maybe that's intentional).

I can wholehartedly agree with everything said there, if I mentally replace "AI" with "big tech profitmaxxing using this new tech".

I however, don't want to throw the baby out with the bathwater: https://blog.senko.net/how-i-want-to-use-ai

I do believe there's going to be a lot of left-behinds, as a sort-of digital rust belt. Even though, as an industry, we've always been in the business of automating and replacing ourselves, the shift will hit too quick.

I lay none of the blame on AI the technology, and all the blame on AI as a mindset and excuse.

Layoffs are not due to AI, but it's a convenient excuse: "more productive, don't need people, we're firing on all cylinders and yeah, firing 20% of the workforce while we're at it". Everything else being equal, the "more productive so we earn 20%" counterfactual makes more sense - but of course, not everything else is equal.

Treadmill speed will increase, no doubt. We haven't lowered working hours from 40 to 20 when computers 2x'd us all, we for sure won't lower them now.

We'll manage the nondeterministic imperfections, but boy, will there be bumps on the road.

What I fear most: AI will give us all more power. This includes profitmaxxing no-holds-barred corpos, from preseed startups hustling 996-style to big multinacionals. Even now, with locked-down devices and subscriptions for everything and owning things replaced with "owning a limited nontransferrable revokable end-user license", it's not good. AI is going to multiply that.

Damn, now I need a drink.

And things like AC and clothes dryers are taken for granted.

Not sure where you get your impression of Europe, but if you feel amenities like these are not standard, it’s a few decades out of date.

North Europeans traditionally didn’t need AC, but everywhere where it gets hot - which is everywhere now - they got them installed. Very few buildings with integrated HVAC systems for the entire buildings tho, mostly independent units.

European here. Yes, houses are smaller, apartments can be comparatively tiny. Street parking can be a challenge.

However: I got stores, cinemas, cafes, restaurants within walking distance. My kids can roam around in the neighbourhood without someone calling social service on me. I can walk anywhere in the city at any hour day or night without someone robbing me. I can cheaply purchase free range eggs and organic vegetables. Tap water is fine, actually excellent. 30hour commute is considered too long. Coast is mere 3h away, people come from all over the planet to enjoy it, I spend 5 weeks a year there, just chillin and enjoying life. I get fast, cheap internet, and order groceries, do my taxes and doctors appointment online.

Tell me again how I’m suffering without poorly insulated detached houses, HSA, spam calls, an SUV to drive myself to the bakery, school shooting drills, healthcare bills, homeless people rejected by society, and that circus you have for a government right now?

I don't have global CLAUDE.md and the only non-default skill I have that was used here is the one to use rodney[0] headless browser. I didn't expressly tell Claude to do browser testing, it decided to do it on its own.

So no extra guidance beyond the prompt.

[0] https://github.com/simonw/rodney/

Correct, that's the "workflows" part they introduced in claude code alongside the new model.

(Correction: I'm on a Max ($100/mo) sub. Realized the mistake too late, so can't edit my comment.)

Token equivalent of ~ $20 (I'm on a $100 Max sub).

Usually just once (and I did just one test for this particular one), but I've found the overall quality to be relatively consistent.

There's too many confounding variables here, randomness just one of them. So I don't think of it as a definitive test (and reliable ordering), just another data point (along with actual benchmarks, pelicans, etc) to get a sense of the capabilities.

For example, I managed to get something out of DeepSeek 4 Flash quantized to 2-bit with Antirez' DwarfStar, used via Pi. Almost kinda worked! :) Which makes me optimistic for using local models for serious development soon - I'd say within a year.

I've updated the page with the prompts, c/p-ing here:

Minesweeper: Create a beautiful and fully functional Minesweeper clone in HTML/JS/CSS (all in one file).

RTS: Create a simple but functional real time strategy (RTS) game similar to old WarCraft, StarCraft or Command & Conquer games. The player should be able to build buildings, create units, gather resources and should uncover the whole map. No AI or multiplayer needed. Use simple but nice-looking graphics. No sound. Implement everything in HTML/CSS/JS, everything in a single file (you can use 3rd-party js or css libraries/frameworks via CDN).

Is that for bsky.app (BlueSky platform) or my personal site (senko.net) where I put up the list of tests? What browser/device was that?