ponyous

Interesting way to show off a model last on every benchmark. Not sure any other lab is doing this

As someone who just had success with equivalent system in EU my recommendation is to get someone who's done it before to do it for you. I hired an agency. They took 8% fee, which is pretty low, usually it's between 10% and 15%.

Absolutely. Running it now, will update this comment in about 30 mins.

Edit: Surprisingly very good results with 3.0 flash with high thinking.

Cost: $0.06

Duration: 3.22 min

Code Errors: 1.3 per attempts (meaning on average it had to retry 1.3 times)

Adherence was on par with 3.5 flash Low thinking

I don't have the eval results live yet, so I cannot share them yet.

I was benchmarking using a soon to be released new version of my AI CAD modeling software[0]. It's basically an agent that has access to tools that can execute build123d scripts, get sculpted models, blender to combine sculpts + parametric models, tools to inspect the model (visually and with code), search datasheets, ...

I tried what you recommend a while ago (asking an AI to evaluate using different angles) and the AI evaluations were extremely bad - barely any correlation to what I scored. Things have gotten better, but I don't trust it enough yet.

Here is how I score adherence (and how AI did as well, but I tried methods where it would just give back a boolean "pass" or not):

    <0.2 → Poor – Misses core intent; largely irrelevant or incorrect.
    <0.4 → Weak – Partially relevant; significant omissions or errors.
    <0.6 → Fair – Covers main points but lacks completeness or precision.
    <0.8 → Good – Mostly accurate; minor gaps or deviations.
    <=1.0 → Excellent – Fully aligned; precise, comprehensive, and faithful to intent.

    dragon-bottle-stopper
    editing-param-mid-conv
    editing-parametric-enclosure
    editing-swap-material-param
    editing-text-edit-cube
    multi-turn-bird-house
    multi-turn-dice-tower
    multi-turn-modular-planter
    multi-turn-phone-stand
    multi-turn-shelf
    one-shot-bookend
    one-shot-cable-clip
    one-shot-chess-queen
    one-shot-coaster
    one-shot-coffee-cup
    one-shot-dog-tag
    one-shot-dragon-figurine
    one-shot-hex-bracket
    one-shot-keychain-fob
    one-shot-low-poly-tree
    one-shot-pegboard-hook
    one-shot-pi4-case
    one-shot-threaded-jar

Just ran and scored 63 3d model generations (via code) across high and no reasoning. 3D Modeling benchmark quickly shows spatial, logic and code performance of the model so I think it's a very good indicator of the quality.

Here are the results compared to Gemini 3.5 Flash:

    Model + config          CodeErr/gen   Cost/gen   Median time   Quality
    gemini-3.5-flash, low      0.71        $0.18        68s       baseline
    GLM 5.2, reasoning high    0.61        $0.18       289s         -6.0%
    GLM 5.2, reasoning off     1.52        $0.10       126s        -13.6%

Edit: I recently ran evals with Kimi 2.7 and MiniMax-M3 and this is clearly open source SOTA model, by far.

Basically double from Opus 4.8 IIRC

Surprisingly written by a human :)

Article ends with this

Bun is(was?) a lot about performance. How does it compare to zig?

I've run a tons of benchmarks for OpenSCAD for all kinds of models and setups, and what I realised is:

- Models are very jagged (might excel in one type of 3d model, but not another)

- Gemini models are the least jagged in my experience and have the best image understanding

- Gemini models are also the most creative (which may be undesirable if you want precise CAD part)

- Overall this benchmark doesn't prove much because one 3d model (and one attempt) is just not enough. I am usually testing on at least a dozen models each generated 3 times, but should really do much more, but it's too pricey for a solo dev.

Still, thanks for publishing this. Will be definitely run flash 3.5 soon to see how it performs.

I've seen this and other attempts like this[0] while exploring improvements for my CAD AI[1]. And I think these are potentially powerful solutions, but none of the current projects/weights have enough training (data or time) to make it work on arbitrary models. MeshCoder pretty much works only on models based of their training data. I haven't tried GenCAD but other commenters have confirmed my suspicion.

[0] https://daibingquan.github.io/MeshCoder/

[1] https://grandpacad.com

Seems like Chinese labs are the only ones that are trustworthy (at least when it gets to this specific issue). This feels so ironic haha

The site is currently unavailable 503 so I can't read it. But I wonder, what should you consent to? Every dependency? Every dependency above 1GB?

Kimi is nowhere near GPT or Opus unfortunately. I really wish it was. I’m running evals where models have to generate code that produces 3D models and it’s obvious that it lacks spatial understanding and makes many more code errors before it succeeds.

Maybe it’s better in one particular case here and there and I think this blog post is example of that.

Gregor! Not sure I want to say more on here.

Been following you guys a while, seems like you've been gaining some traction recently, lets goo and congrats!

I have been working on GrandpaCAD[0] for a while, a very similar product. I thought of you as my biggest competitors but noticed recently you are focusing more and more on professionals while I am focusing on total noobs in modeling who just want to whip out a quick model. So I guess we are not competitors anymore?

My evals[1] show that Opus 4.7 and GPT 5.5 are very comparable in terms of generation quality, but GPT 5.5 is slower and costs sooo much more in my harness. And the original breakthrough model was Gemini 3.1. I'm curious do you have more written about your benchmarks setup?

If you want to chat email is in my profile. Btw, just met "your"(?) neighbour on a plane a couple of days ago. World is small.

[0] https://grandpacad.com

[1] https://grandpacad.com/en/blog/public-benchmarks-misled-me-o...

Another library I have to integrate and benchmark against OpenSCAD for my AI SaaS[0]. I am really curious how constructive solid geometry compares to sketching and extruding that CadQuery is build on.

Anyone curious in the writeup? I have a pretty good harness for evaluating 3d generation performance.

[0] https://grandpacad.com

Why does M1 Max project significantly higher revenue than M3 Max with double the ram?

http://grandpacad.com - 3D Modeling tool intended for creating 3D prints. AI based. Allows for dimensionally accurate parts as well as organic shapes.

I massively improve it every month. Pretty proud of it.

As far as I know reddit does the same thing. If you don’t follow normal human patterns you will quickly get banned. I read some guy’s report where he got banned 8 times detailing each behaviour and indeed it was nothing crazy.

I export it as .3mf file and display it with threejs on the web. Compilation seemed fast enough - few seconds tops.

Does this use its own backend/engine? I've been working on LLM to CAD tool[0] and have realised there are so many backends and options to choose from. Since the realisation I'm trying to find the best representation for an LLM. I think OpenSCAD is currently the best and most feature complete choice, but I definitely need to dig a bit deeper. If anyone has any pointers I welcome them!

[0] https://GrandpaCAD.com

Thank you!

This "screenshot -> refine loop" is a great strategy and I have built it into my 3D Modeling product as well[0], but had to disable it because it would often quadruple the costs and the product is already expensive.

I am on standby to enable it though, just need a price to drop a bit more!

[0] https://grandpacad.com

In the coworking I am in people are hitting limits on 60$ plan all the time. They are thinking about which models to use to be efficient, context to include etc…

I’m on claude code $100 plan and never worry about any of that stuff and I think I am using it much more than they use cursor.

Also, I prefer CC since I am terminal native.

3D AI Modeling software intended for 3D printers.

https://grandpacad.com

Originally I made it for my grandpa, but I got a lot of interest so I made it into a full commercial product.

Just yesterday I published a set of 3 mini tutorials if you want to see how it works - https://youtube.com/playlist?list=PLKt1F5TvOjAHE07oBDlPXcrHc...

Ran a bunch of 3D Modeling benchmarks on Gemini 3.1 vs Gemini 3.

Unsurprisingly 3.1 performs a bit better. But surprisingly it costs 2.6x as much ($0.14 vs. $0.37 per 3D Model Generation) and is 2.5x slower (1m 24s vs. 3m 28s).

To me it feels like "lets increase our thinking budget and call it an improved model!"

I am building pretty much the same product as OP, and have a pretty good harness to test LLMs. In fact I have run a tons of tests already. It’s currently aimed for my own internal tests, but making something that is easier to digest should be a breeze. If you are curious: https://grandpacad.com/evals

https://grandpacad.com/

Dimensionally accurate AI 3D modelling. My grandpa has a 3D printer but struggles to use any complex tools. So I am working on this chat interface to allow him to do some simple models.

So far he has triggered more than 150 generations. It’s getting better every model cycle and gives me something I enjoy working on.

I see no mention of that, but OpenAI already has "service tier" API option[0] that improves the speed of a request by about 40% according to my tests.

[0] https://openai.com/api-priority-processing/

I think models are smart enough for most of the stuff, these little incremental changes barely matter now. What I want is the model that is fast.