HN user
And on creativity at least visually, Gemini 3.1 pro is somehow still up there. But its really hindered by its inability to use tool calls effectively or make a long term plan.
My personal benchmark for new models has been to compare video making skills with something like remotion. Usually reveals if they have any "taste" or outside the box thinking.
I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.
And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one. Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.
https://mesmer.tools/benchmarks/ai-video-generation , I usually put basic ones here.
Hadn't tested out any of the image models for training loras since Flux 1 dev, so I was curious which one is the current best. Results were pretty interesting
Models tested: Ideogram v4, Flux.1 Dev, Flux.2 Dev, Klein, Krea 2, Z-Image
I'm on max $200, I had this idea for around a month since Fable got banned. For the usage, not exactly sure since I had like 4 agent heavy tasks running at the same time.
I think personally you can do first half of this with the direction with fable and then render + checking using sol since thats what I ended up doing. Guesstimate I'd say it takes around 10-20% of the 50% fable usage
Had some fable usage to waste yesterday cause my reset was gonna happen. Ended up burning all my usage so had to finish the last part with Sol itself.
Imo Remotion based videos feel better to watch than just actual GenAI videos, like Seedance.
the exact prompt for reference:
"I wanna make a plan for making good fireship videos automatically. so my thinking is, you need to first need to find a good way to insert memes like gifs, short videos, images stuff like that. another part of a good fireship video is the voice itself, so you need to find a way to voice clone fireship's audio(probably download a sample video and use our existing higgs audio api, check airoleplay and search "higgs" for the api endpoint details and the api keys and stuff if you need). another part of it is the animations and creative text animations and extra touches that he puts, maybe for this you could analyze a couple of his vids in general using gemini 3.1 pro with openrouter in detail and you can ask multiple questions abut a video to explain in detail exactly what happens.
now remember you're the orchestrator and this is likely going to be a very long task so you need to be using your context very carefully, and assign various research tasks to opus or fable subagents depending on the complexity.
the video itself should be about claude fable 5 and make another one for gpt 5.6 sol. some more reference for how we recreated antoher vidoe with a single prompt before: '/Users/test/Documents/openmotion/recreate-video-user-prompts.md'"
As long as they have the best model they can afford to lose goodwill.
People who don't wanna spend too much on LLMs and are trying to optimize whats subsidized even on the Max plans are customers Anthropic is honestly better off without.
Ok thats a one month clock to the next Opus model at least, so thats a silver lining to a meh model.
Spoiler: more than half of it is just Meta ads spending cause its just addictive.
Seems really good at frontend work, and as a result on remotion programmatic videos. Not the best yet, thats still Gemini 3.1 pro(trained on actual videos) or Fable, but often better than what Opus can come up with
noticed it cause of ad manager, the main facebook site being down is kinda weird tho. I don't remember when was the last time that happened
Overall an improvement over Opus 4.8, but I'd still say Gemini 3.1 Pro has more of an artistic vision even tho it fails tool calls and writes buggy code sometimes.
Ik almost everyone is interested just in the SWE stuff, but this has been a good eval for me to think about how big the model is, how "creative" it is for generating new ideas etc.
More results from fable, with comparisons for Gemini, opus and some open source models: https://mesmer.tools/benchmarks/ai-video-generation
I think gpt 5.6 is coming out today so might wanna wait
/model claude-opus-4-8
seems to work but idk why they never set it so you can see it in the /model list.
"what model are you
I'm Claude Opus (claude-opus-4-8), running in Claude Code."
Yep sorry was just pulling it out my rear, not like a market trend that nearly every enterprise uses Anthropic or Openai models for coding or that Anthropic has had such ridiculous growth that they're 10x-ing year over year
My point was that even openrouter, the one place people who are looking for open source SOTA models go to, doesn't definitively have opensource models at the top. Esp considering quite a lot of the closed models usage is through AWS, GCP , Azure etc, probably dwarfing the usage on openrouter by a huge factor
As long as closed source is 6 months ahead in terms of current difference. Although this is hard to figure out using simple percent based coding benchmarks, you def. notice it when you're actually trying to do a long task. Even simple things like UI "taste" is enough for me to use opus instead of 5.5 though even though 5.5 is strictly better for anything that doesn't have a UI, ie backend, scripts, making agent workflows etc
As long as closed models are 6 months ahead I won't be switching from them to prev. 6 month SOTA open source models. Maybe its just a different calculation if you're in a job, but as an indiehacker I'll take any edge I can get
Ofc again, can be convinced to switch if there's however a clear speed difference, like 5x+ for a open source sota even if it was SOTA for 6 months ago
Based on current market for LLMs I'd say my use of "you" in the general is fine. Even openrouter which doesn't capture all of the SOTA closed models but nearly all of opensource model usage has Opus as 1st(on last week) on "Programming" category and 3rd in overall rankings
Cost for the value delivered. Like if you offered the current SOTA open source models at $0.1/M, I still think I'd be using Opus or 5.5 at $30/M. Or say GPT 5 which was released Aug 25, I don't think I'd use it for coding for even $0.1. I'd def find other uses for it(translations, agentic workflows, prompt guards etc), but for coding I don't think I'd ever completely switch to a SOTA open model
Unless ofc there was an actual speed difference, only reason I'd be willing to go with a worse model couple of percent worse than current best model is if the speed was at least 5x higher. Looking forward to kimi k2.6 offered publicly by Cerebras
For coding you always want to go with the best model in the category, not something that would be the best model if we went 1 year back which GLM 5.1 is, and I'm saying that as a big fan of GLM cause I run a translation site where GLM is good enough for the price.
Most of the money right now is in coding. Openai and Anthropic just have to be 6 months ahead of SOTA open source models and they'll capture most of the enterprise and dev market
If nothing else this blog did give me the idea that I should split my $200 claude max plan into two $100 CC max and $100 codex plan, esp because Claude is now offering 1.5x weekly limits so its the 5x usage is now more like 7.5x usage.
I'm just hoping they release Mythos soon now that it seems like they have enough compute to do promotions like this
I think that was a typo on my end, its "/model claude-opus-4-7" not "/model claude-opus-4.7"
I'm on the max $200 plan, so maybe its that?
I think its just a visual/default thing, cause Opus 4.0 isn't offered on claude code anymore. And opus 4.7 is on their official docs as a model you can change to, on claude code
Just ask it what model it is(even in new chat).
what model are you?
I'm Claude Opus 4 (model ID: claude-opus-4-7).
https://support.claude.com/en/articles/11940350-claude-code-...
Not showing up in claude code by default on the latest version. Apparently this is how to set it:
/model claude-opus-4-7
Coming from anthropic's support page, so hopefully they did't hallucinate the docs, cause the model name on claude code says:
/model claude-opus-4-7 ⎿ Set model to Opus 4
what model are you?
I'm Claude Opus 4 (model ID: claude-opus-4-7).
We went from "Peak hours" meaning 2x usage plus slower to now it just does 500 error
Was playing around with it earlier and the results were actually good with it being opensource and for how cheap the model is to run.
The tool page itself is smth I vibecoded in like an hour. Feel free to give it a try
turns out its for an April fools tomorrow: https://x.com/mesmerlord/status/2038938888178135223
Also saw this on twitter earlier, thought someone was just making a fake hype post thing. But turns out to be an actual prompt for capybara huh: https://github.com/chatgptprojects/claude-code/blob/642c7f94...