HN user
I got usable token rates (10-20tps from memory, so marginal) with the Qwen A10B a while back, well before all the new speculative speedups landed in llama.cpp. There's an unmerged branch which allegedly supports this, but I don't know how well yet. Looks worth investigating but I might give it a few days to see what bugs get shaken loose.
What's your hardware? I've got 64GB RAM and a 4090 here, wondering if it's worth a play.
Couple of other more businessy reasons:
- SynthID hides the watermark in the sampling RNG. No randomness -> no watermark.
- If you want to distil on the model outputs, you want temp=0 outputs. No temp=0 -> worse distillation.
The team. The organisation. The reviewer, if they care about team throughput.
That is the context switch I'm referring to. It's worth the reviewer suffering a 15 minute (say) flow state loss to avoid the multiple hours the ticket would otherwise have been waiting to be picked up.
For us, review latency was obviously a problem so we eliminated it (for most tickets, anyway) by making the handover synchronous. Get on a call with the reviewer when it goes into the review column. The win from eliminating review latency massively outweighs the context switch cost for the reviewer. All the more so when the patch is small enough to actually read through on the call, so if there is anything immediate to address you can ping-pong it without anything waiting.
The reasoning trace would be far more interesting, and that's not exposed.
Yeah, it's only a chat template change. So if you don't fancy re-downloading the whole model and can just hack the new chat template into your process, it's a lightweight test to do.
That doesn't make the burden of proof magically mine.
Worth knowing that Unsloth have just put out another Gemma 4 release from Google's upstream updates which should improve reliability. Bugs in the chat template affecting tool calling and other issues, apparently. https://www.reddit.com/r/unsloth/s/MpC6Hzs4Wj
I suspect the hard bit is that it sometimes needs to back up and redo, and that's an interface they haven't got figured out. I'm fairly sure I remember Dragon Naturally Speaking doing it in Word years ago though, so the interfaces should be there.
Backwards. The government knows who has kids because they do things like issue birth certificates and operate state schools. In a lot of places it's legally very difficult not to be listed as a parent on government records.
So why should I have to prove I don't have kids when the government can know I'm not on any of those lists?
I get my team to review my code, AI generated or not. Goes through the same process as anything they produce. Two pairs of eyeballs on everything.
Parsing used to be "AI". If you look at proceedings of old AI conferences you get this impression that anything interesting you might program a computer to do has passed through the field at some point.
Tagged Netstrings (tnetstrings) was a related proposal from 15 years ago or so. It replaces the comma with a single-character type definition so you can do JSON-like objects with a couple of recursive types: you had ',', '#', '^', '!', and '~' for strings, integers, floats, booleans, and nulls, then ']' and '}' for lists and dictionaries.
Most of the links have bitrotted and I don't think it ever got much traction, but I did always like how simple it was. There's a copy someone grabbed of the original spec here: https://raw.githubusercontent.com/ged/tnetstrings.info/refs/...
The fanfare over multi-touch in phones is something I've never understood. It's flashy but phone use (certainly my phone use) is optimised for one thumb, not two fingers.
Resistive touchscreens get a lot of criticism but even back then that was partly down to Apple propaganda which caught on because people were used to cheap screens with rubbish resistive sensors. The touchscreen on the N900 was good: high resolution, rigid, fast, accurate, sensitive enough for fingertip use.
I'm yet to be convinced that number of PRs as a productivity metric is any less flawed than counting lines of code.
I can believe - easily - that there's a real uplift here, but attaching any meaning to the 24% number at all is a massive overstep.
That's a stretch but it doesn't change the outcome.
On 1 and 3, the obvious move is to shift the bulk of the harness behind a new API that's not based on raw LLM access. Then they get to hide secret sauce behind that API and all three go from commodity to premium while simultaneously being able to try out whatever tricks they can get away with to reduce their own inference costs. I'm almost surprised this hasn't happened already.
Terrible ideas get executed all the time, despite the problems with them being well understood.
Bedrock is really out of date with the models it offers, to the extent that I'm not sure they even have plans to update what's on there now they have the deal with Anthropic. They're still offering Qwen 3, not even 3.5 and certainly not 3.6. GLM 5 is the newest z.AI model they have, when it's 5.2 that would be the one to worry Sonnet.
There are some ok models on there (Qwen 3 Coder Next is usable and fast, for instance) but the lack of updates in a fast-moving field makes it something I don't want to recommend to my org.
Wheels in a different configuration solve the stairs problem. https://www.sacktrucks.co.uk/stair-climber-trucks/
Either
1) the company has device-level control to the degree that they can not only restrict which API endpoints people can connect to but which accounts they use to do so (in which case this already isn't an issue); or
2) they don't, and all bets are off anyway, open weights or not.
Seems weird. A 9B model would normally fit unquantised on a 24GB GPU.
With this model size I've found that the harness seems to matter more. I've moved on to little-coder rather than raw pi with qwen3.6 27b personally, it might be worth taking a look.
Nah. There are already models at every size on the scale. If you want to run an open 1T model today, you can.
What's going to happen is that the capability at any given size point is going to get better over time as new training regimes cram more into the available space. A 27b model released next year will be better than a 27b model this year (else why release it?). Hardware will get more useful, not less.
That ends up begging the question, because the next step is "how high do you have to drop it from so that it's travelling twice as fast?" and you're immediately going round in circles.
"Eventually" doing a lot of work. Micron (and implicitly anyone signing this deal) are betting demand is going to outstrip capacity for several years, taking into account what new capacity can be brought online and when.