HN user
Pi. The benchmark is local, mostly stuff from my work, I run it everytime a new model comes up. The top model rn is gpt 5.6 sol, followed by fugu ultra, fable, opus 4.8, gpt 5.5 and glm 5.2 (which is the REAL IMPRESSIVE one still). Kimi-k3 is 14th in the list.
I'm disappointed. After all the buzz and benchmarks, I've tested with my personal benchmark that simulates real-world day-to-day specs for agentic coding, following instructions across long time walls, changing several files and code requirements with separation of concerns to build a complete Saas e2e - it reaches a similar rating as DeepSeek V4 Flash.
I've posted a recent article about the future of software development https://saturnino.substack.com/p/out-of-the-loop?r=7eqhw&utm...
Basically, in a decade or so, we'll be completely out of the loop in software development; even this title won't exist anymore (like the 2000's webmaster). We'll still be around, but with different roles.
so as everything in life. your relationships, your car, your job. Everything requires maintenance, and from the brand-new version, it is ONLY and ALWAYS downhill unless YOU keep things up.
GLM 5.2, with the help of GPT 5.5, presented results on par with the Fable model.
GLM 5.2 had wall times that were 3x longer than Fable's. It also stored ~10GB of inference-thinking logs on my hard drive, compared to ~1 GB for GLM 5.1. This suggests GLM 5.2 has a much deeper thinking iteration (10x more) than its predecessor.
Fable stored KBs, but Claude is known to be hiding inference logs, so it's unclear if there's an efficiency/inference gap between these models.
GLM 5.2 run cost me ~US$3, while Fable run cost me ~US$9.
GLM 5.2 is 3x cheaper but takes ~4x longer to generate results. This suggests a correlation between the data on why one might be cheaper but slower, the other is faster but costlier (e.g., datacenter hardware tier, availability, services). So, even though the results are on par, this doesn't mean there's an efficiency leap for GLM 5.2 - they might have cost the same if GLM 5.2 had the same wall time; and if that's the case, Fable is far ahead.
Nonetheless, this shows we can have GLM 5.2 working on production-grade codebases, combined with an SOTA model as a reviewer.
Thanks - fixed.
Thanks - yes, I agree, and already I knew this was a thing. I just haven't yet concretely made the actual math, and it left me baffled - Anthropic and OpenAI are charging 1/3 of the COST tops.
The cost of building the first version of FB has dropped 90% The cost of building the next FB stays the same
More sophisticated tools mean more refined products.
If an easier and cheaper method for working carbon fiber becomes broadly available, it won't mean you get less money; it means you'll now be cramming carbon fiber in the silverware, in the shoes, in baby strollers, EVERYWHERE. The cost of a carbon fiber bike will drop 90%, but teams will be doing a LOT more.
You could say the cost per line of code has dropped 90%, but the number of lines of code written will 100x.