We made Grok 4.5, GPT-5.5, and Claude build the same apps

https://www.tryai.dev/blog/grok-4.5-vs-gpt-5.5-vs-claude-build-off

by hershyb_ • 14 days ago

173 93 14 days ago

maxdo2• 14 days ago

Tried at work , this release def a moment I will remember. My work is not the same . The model is the first model that offer exactly as I want :

For hard tasks , that needs precision I will wait and pay expensive tokens

For everything else , query data , logs, rolling out releases , I’m using grok and it’s much better vs other tools and much cheaper too .

Tostino• 14 days ago

Who the hell tries something that's been out a few hours and says "My work is not the same"?

maxdo2• 14 days ago

I am , because it’s smart and extremely fast . Same way as opus 4.6 was an obvious game changer

paxys• 14 days ago

Why not wait one more day for GPT-5.6?

m4rkuskk• 14 days ago

That will be in the Part 2 article.

fluidcruft• 14 days ago

And why not Sonnet?

faitswulff• 14 days ago

Also throw in GLM 5.2 for good measure

foxfired• 14 days ago

If we wait for the next models, we will never test anything because there will always be another model. Like the Ai Scotsman:

"Nay, laddie, that’s no’ the real AI Scotsman! He’s grander still! More powerful! Just wait for the next model!"

looksjjhg2• 14 days ago

I think he’s being sarcastic

acters• 14 days ago

I worry that GPT 5.6 will be heavily restricted and have the same feature to fallback to another model like Claude fable 5 does all too often. That fallback shenanigans mess up actual benchmarks and I don't like it.

trollbridge5• 14 days ago

Well, I'm probably not on the list of special people who will get to see GPT-5.6 Terra.

mlmonkey2• 14 days ago

I am 99% sure the post was written by AI

dwa3592• 14 days ago

i will give you the remaining 1% because i felt the same way.

jszymborski• 14 days ago

The honest takeaway: this is 100% written by an LLM.

mlmonkey2• 14 days ago

That was the honest giveaway ... :-D

trollbridge5• 14 days ago

Pretty much any harness with a halfway decent model should be capable of running the experiment including making the screenshots, summarising the results, and so forth.

I sort of do this whenever an interesting new model comes out.

t0mas88• 14 days ago

That number is not just a statistic, it's load bearing. ;-)

walrus013• 14 days ago

But just think of the amazing advancements over the past 12-18 months in fully automated slop-posting! We've reached a new high water mark.

ks2048• 14 days ago

Blog post idea:

We made Grok 4.5, GPT-5.5, and Claude write a blog post about using Grok 4.5, GPT-5.5, and Claude to build the same apps.

userbinator• 14 days ago

Either that or a human that has started writing like an LLM, having been "trained" on LLM output itself by sufficient exposure.

kristopolous2• 14 days ago

I'll actually put in spelling errors and grammar mistakes intentionally these days to show human

HeartStrings• 14 days ago

Role reversal, two figures, one file.
“snappy stylist”

Funny you can tell its slop just by this

RickS• 14 days ago

Barring the retry thing, n=1 on all models? Am I misreading, or is this a joke?

Variance in quality on these things is so, so high.

jeffgreco• 14 days ago

So strange to write a whole post with Claude giving the best results and Grok consistently the worst, but awarding Grok the winner because at least it did the worst fastest?

singingtoday2• 14 days ago

GPT was the worst on the Rubik's cube

GaggiX• 14 days ago

Grok did not render anything, they had to prompt it again.

cuvinny• 14 days ago

They gave grok 2nd try on that one. Now one shotting a webpage is a dumb metric but if that is what you are testing it did the worse.

trollbridge5• 14 days ago

GPT-5.5 isn't really a fair comparison to Fable or Opus. GPT-5.5-Pro would be a better comparison (and yes, I know how much more expensive it is).

I really wish they'd thrown in something like GLM-5.2 into the comparison.

sceptic1232• 14 days ago

GPT doesn't render but does actually work, Grok does not fully work Two scrambles in a row and the thing is broken.

dadoum• 14 days ago

I tried to one-shot the first test (the Rubik's Cube test) with LucidQuery's Swift model, to test it, as there are not much benchmarks about it and that they brag a lot about it, and I was pleasantly surprised to see it achieving a result similar to Grok 4.5 but in one shot (there is the same issue that if you scramble twice the solve button does not work anymore, but it got it in one shot).

Though it crunched most of the free quota, 47111 tokens, so I couldn't make multiple attempts.

singingtoday2• 14 days ago

Love the idea, I think more complex games would show the gap in ability better.

Do it again but this time get them to make a multiplayer online Jetmen REVIVAL game. Online play is key, because it's very complex. Jetmen is a good game for this since it has physics and customization that's complex enough but still simple.

krauq• 14 days ago

Too nice to Grok, if there are really cost savings it should say how much each of the three demos cost so we can judge if it's worth the lower quality (probably not). The time to complete each would also be interesting.

Kuyawa3• 14 days ago

I'd like to see the comparisons with DeepSeek, Qwen, Mimo, Kimi and GLM

Kuyawa3• 14 days ago

I just did the tests, Mimo and GLM delivered working cubes but GLM was the only visually perfect with smooth movements and great effects.

GLM is the clear winner:

https://chat.z.ai/space/t19sx5kvw631-art

trollbridge5• 14 days ago

GLM-5.2 just keeps blowing me away. I pretty much use it in the mode I used to use Opus for where I have it drive a lot of the reasoning with DeepSeek or other models for the smaller subtasks/subagents.

Kuyawa3• 13 days ago

Tried the recent Tencent Hy3 and it produced a working cube, but plain visuals.

GLM still wins by a landslide

* I ran all agents in their web version with their default settings, I don't remember now if they were set to deep thinking or what but that's the result they produced by default.

thorum• 14 days ago

Interesting that all four models converge on such similar designs, for such short prompts.

HeavyStorm2• 14 days ago

They were trained pretty much on the same data.

HeavyStorm2• 14 days ago

[flagged]

hyperhello• 14 days ago

It’s still science if you say “this appears to be a specimen of X” even if you don’t do a genetic test. Things don’t automatically graduate to science either by repetition or by formal verification. What makes a rubics cube is obvious enough that you can pass or fail.

sinuhe69• 14 days ago

It said “3D-looking Rubrik cube”. Maybe your cube looks different but I’m pretty sure for everyone else, the GPT result doesn’t look like a 3D-looking Rubrik cube.

looksjjhg2• 14 days ago

Show us how it’s done then, talk is cheap

remus• 14 days ago

This feels like a kid trying to do science. The will is there, but lacks experience.

It's funny, when I saw the title I was hoping the article would include some sort of blind ranking, where you could see the outputs (without knowing which model they came from) and score them on some criteria. Could have been a fun way to get a better ranking of the results.

coldtea• 14 days ago

Comparison are completely subjective

Nothing wrong with that.

In this case, the requirement is clear enough, and the result is similar enough to judge it subjectively.

Even code-wise, if tests pass and the features are there in all cases, the rest of what matters (architecture, code quality, style, readability) is all subjective too.

drcongo• 14 days ago

GPT somehow wins the particle sandbox thing too despite its heavy attractors not attracting anything.

logicalappeals• 14 days ago

This feels like a kid trying to do science.

lol. These are the comments that keep bringing me back here. Great feedback delivered with a dash (or more) of salt.

6510• 13 days ago

I wouldn't be surprised if multiple versions of both the cube and Breakout code are available some place online.

https://upload.wikimedia.org/wikipedia/en/c/cd/Breakout_game...

https://upload.wikimedia.org/wikipedia/commons/1/1a/Screensh...

That the tiles are much to thick makes Grok the most reasonable result.

dang• 12 days ago

You broke the HN guidelines badly with this, first because of the name-calling and personal attack, and second because it is a shallow dismissal in exactly the sense we ask people not to post here:

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." - https://news.ycombinator.com/newsguidelines.html

On the plus side, your comment does contain some specific observations which could have made for a good comment—one that communicates interesting information respectfully. If you'd please post that way in the future, we'd appreciate it.

Zebfross• 14 days ago

Isn’t the number of turns most important? Some agents take repeated input, while others can mostly one-shot what I’m looking for.

kelvinjps10• 14 days ago

Chat got ones were slow on Firefox mobile

nrightnour• 14 days ago

This is disgustingly biased. The conclusion is that Grok holds its own?! There was zero evidence of that.

deanalyzer• 14 days ago

Yes, I wonder how the verdicts would hold under a blinded test. This analysis read like the authors going out of their way to be supportive of Grok.

steve-atx-7600• 14 days ago

I have not used grok 4.5 yet, but the other pictures match my experience doing anything graphical with the other models that it cracks me up. gpt 5.5 has no design sense whatsoever. It cannot even make terminal output not look terrible. I've asked it to use colors and formatting in various ways and got goofy randomly colored output. opus 4.7 and later seemed to have an inuitive design sense by comparison - 2d or 3d. Fabel 5 is just rock solid.

Yes, subjective. But it matches my repeated experiences with these models for what it is worth.

trollbridge5• 14 days ago

I get better results with Opus than Fable 5 on various oneshots (including our old friend of "generate an SVG of a pelican riding a bicycle"). (Opus is also far and away better than pretty much any other SOTA or near-SOTA model.)

RazorBucksICO• 14 days ago

“The honest headline:”

Written by Claude. Ugh. If it’s worth publishing, it’s worth proofreading, folks.

Yiin• 14 days ago

especially when using /humanizer is one prompt away

dirteater_• 14 days ago

The receipts: speed and cost

I don't get why cost per reply is at all relevant here?

Why do so few who attempt comparisons actually compare dollars per task.

baxtr• 14 days ago

Tokens per task might be the better choice

Computer0• 14 days ago

Why do I care how many tokens were used? 50,000 or 5,000,000 all I care about is the output quality, speed, and the cost.

unanonymousanon• 14 days ago

Could you elaborate on why tokens over dollars as a metric?

NewJazz• 14 days ago

I think these were all one shots, so it was 1 reply per task?

brainless• 14 days ago

I get the point of this demo but if instructions are clear, tech stack related resources are available, then the models do not differ as much.

I use different models all the time. And mostly lower cost ones. I do not know how people write software these days, but I have clean instructions, usually in Epics and they have Tasks.

I have been using DeepSeek V4 Flash for much of my coding in https://github.com/brainless/akar for example. Planning is mostly done by Qwen latest (in opencode) or Sonnet.

For my commercial, client work I use Claude but barely use Opus. Sonnet does most of the work. For a recent project, I went through a 35 page PRD in about 4 weeks, that includes client calls, changes, Ecpi/Task generation, a massive test suite, deployment.

sgk2842• 14 days ago

If you like this kind of comparison, we have an arena of 52 apps one-shotted across 21 models here: https://arena.logic.inc/

I keep it pretty up to date (tomorrow Grok 4.5 and Sonnet 5 should be pushed).

kristopolous2• 14 days ago

I want this with smaller models as well like Gemma 4 or Qwen 3.6

sgk2842• 14 days ago

Awesome - will work on getting those in.

theturtletalks• 14 days ago

Really nice site! From your experience, what’s your go-to model for nice storefronts?

coopykins• 14 days ago

Impressive, specially the amount of models used for the comparison.

Otterly99• 14 days ago

This is impressive and, I think, complements well whatever benchmark is the hottest right now.

Madmallard• 14 days ago

Could probably do this with a much older model given that it's something that probably has thousands of github repositories for source code to do so.

owenbrown• 14 days ago

I’m spending a significant portion of my day waiting for agents to execute.

What’s more interesting to me than time-to-first token or latency is the time it takes for the agent to execute, from starts to finish, excluding when it’s waiting on a human.

madaxe_again• 14 days ago

Subagents can help enormously…

On which note I recently convinced codex to use the ChatGPT web client to run subagents. Means I only pay for the slavemaster, and all the slaves I can eat for $20 a month. Actually works surprisingly well - I currently have it crunching through a large dataset, which would have taken weeks on a single thread - started last night, nearly done this morning. $20.

wwind123• 14 days ago

Half year ago I tried to use Codex, Claude and Gemini build the same scripts to automate various things on my machine. Claude was the clear winner back then, making the most reasonable assumptions, presenting results in the easiest-to-read format, writing runnable script with minimum dependency. Half year later I think Codex and Claude models have both advanced a lot, but Gemini is still lackluster. Gemini could catch problems when reviewing Claude/Codex's design plans and code, but it's hard to make Gemini make complex plans or implement complex code by itself.

JoyfulPanda2• 14 days ago

Grok failed the Rubik's cube. I pressed Scramble twice and then solve and it didn't solve the cube. Opus did.

JoyfulPanda2• 14 days ago

I'm not sure, but it looks like none of them solve the cube. They just memorize what happened to the cube and do the reverse.

sehw• 14 days ago

look guys, we burned money! Upvote pls.

millsau• 14 days ago

Claude seems to have the best text generation out of all of them.

reenorap• 14 days ago

How can grok create a coding LLM at the same level as OpenAI or Anthropic when they don’t have the same amount of AI talent as the other companies by an order of magnitude? Is it really that easy to train a coding model like that?

inemesitaffia• 14 days ago

Cursor

kev009• 14 days ago

Those two companies spend all their effort patting themselves on the back about "peer reviewed studies" and posturing immeasurables like security theatre and "trust and safety". It's really not hard to believe that Grok or Chinese models can show there is no moat.

throwa356262• 14 days ago

1. Elon throw money on Gemini folks to get them switch ship

2. Dario is an idiot for not realising his dataset, workflow and model are going to be copied when he uses spacex datacenters

3. Grok has a special fan base that promote it everywhere they go

dminik• 14 days ago

The fact that the breakout previews included exactly zero gameplay is so weird to me. It shows that there was exactly zero extra effort put into anything here.

sceptic1232• 14 days ago

Grok doesn't even solve the cube if you scramble more than once, that is NOT the AI you wan't building your software, no matter how cheap.

deterministic• 13 days ago

Please ask AI to simplify x 3 before posting AI generated articles.

We made Grok 4.5, GPT-5.5, and Claude build the same apps

Related Stories