AI for generative design: Plain text to 3D Designs

https://blog.insightdatascience.com/ai-for-3d-generative-design-17503d0b3943

by tf_12 • 6 years ago

239 48 6 years ago

blog.insightdatascience.com

zarmin• 6 years ago

This is amazing.

heyitsguay3• 6 years ago

Amazing! I do work in 3D computer vision and I've really wanted to get into 3D generative asset design as a side project at least. If the author or anyone else is interested in collaborating, I'd be down to talk and shop something.

gaogao2• 6 years ago

Sure! I've been wanting to incorporate this exact thing for a "And let there be light simulator!"

Essentially, people in VR would be able to point and speak to create objects in a shared space.

heyitsguay3• 6 years ago

Cool! I've got some Unity VR experience, though i don't know much about incorporating ML pipelines into it. I'm guessing there's resources out there? So basically you'd want speech -> text -> object?

starstorms5• 6 years ago

This would be really incredible! Could maybe be a way to quickly make video game assets for testing and such. Although to make the meshes performant in a game engine you'd need to decimate them quite a bit as the marching cubes I used here produces really dense meshes.

JabavuAdams3• 6 years ago

Oh wow, count me in. You could use Google Cloud Speech to go from voice -> text, and then Google Cloud Natural Language to go from text -> simple command interface.

I've got both working in Python, but don't do anything with the parsed text.

I want to do things like "create sphere" "move that up" "no, bigger"

etc.

Lichtso• 6 years ago

Really cool idea, I wonder why nobody tried this earlier.

However, the main problem I have with this approach is the voxels. They model the geometry and only the geometry. The far more important aspect of topology is left out. Thus, the results suffer the same problems like 3D scanning / photogrammetry does: It is practically unusable as it can not even be textured, yet alone animated or used in fabrication (except for Lego-models I guess). So point 5 of the future work is the biggest one in my opinion.

AndrewKemendo• 6 years ago

Well they have at least for certain parts. I'd guess based on what they use, the whole workflow here was really only possible in the last couple of years.

I'll update this comment another time but back in 2016 my company needed something like this and there was already research using GANs to generate objects from basic parameter inputs. The text portion wasn't there.

I believe that research came out of Stanford actually.

Jack000• 6 years ago

Voxels are directly analogous to 2d pixels, which means they benefit from the myriad of CNN based techniques developed over the last few years (GAN in particular)

3d manifolds are sparse, and more analogous to 2d vector graphics. There are approaches for dealing with this type of data (eg. spectral graph NN) but they don't work as well for 3d topology as CNNs do for dense pixel data, as far as I know.

In the near term, it might be better to explore approaches that use voxels, then generate the topology heuristically.

abj• 6 years ago

Electronic Arts used 3D scanning / photogrammetry in the new Star Wars Battlefront II.

You're right - the topology is a huge problem - but we're starting to see specialized tools that can take minimal input from the artist and automate the topology creation workflow.

Here's the GDC talk where the E.A devs go over how they converted the scans to game usable 3D models. https://youtu.be/U_WaqCBp9zo

masto• 6 years ago

Nobody tried what, generative design? https://www.autodesk.com/solutions/generative-design/manufac...

Or did you mean the way the model is specified?

starstorms5• 6 years ago

Ya using voxels was definitely a big drawback to my approach here but was more intended to just see what's possible.

I think a better approach would be something similar to the StructureNet paper I mention in the post and use graph based models to actually attempt to capture the topology. But they did it with super explicitly defined part trees as training data the hard part would be finding a way to do that in a unsupervised manner so you could actually make use of the massive amount of unlabelled 3D models available.

camillovisini• 6 years ago

Stunning work, congratulations. That was a pleasure to read.

mikehollinger• 6 years ago

Nifty!

mendeza• 6 years ago

Great work! I wonder if using SDF as a 3d model representation would result in higher resolution model generation (rather than voxels), some recent work in that area here: https://github.com/marian42/shapegan Adversarial Generation of Continuous Implicit Shape Representations : https://arxiv.org/abs/2002.00349

starstorms5• 6 years ago

I hadn't seen that paper, looks really interesting! It definitely would improve the resolution but I think fundamentally it still runs into the issues I had with voxels where it mostly just learns low level correlations and not high level topology.

airstrike2• 6 years ago

I was literally thinking about this in bed before falling asleep last night, but I was imagining scenes rather than objects.

I'm in no way a ML developer (I'm not even a developer), but I was wondering how hard it would be to generate an apartment / house layout from a (somewhat detailed) description.

I'm sure the first use that springs to mind is for architects and real estate developers / agents / brokers, but one of my pet-projects-I'll-never-really-do is to recreate memories in 3D from descriptions and old photos. Imagine being able to relive your childhood memories! Just throwing it out there in case one of you smart folks want to take that and run with it...

adfm• 6 years ago

Somebody will eventually build it in Grasshopper/Wallacei.

Wallacei X Cybertruck: https://youtu.be/bLZf-MNRoyg

tarsiel• 6 years ago

That exists! The graphics lab at my old university published a paper on this exact thing: https://manyili12345.github.io/Publication/2018/T2S/index.ht...

airstrike2• 6 years ago

Thanks! That is really cool and I appreciate you sharing it!

FYI I tried clicking on "Slides" but got a 404 error, so thought I'd flag in case you know the authors.

Agebor• 6 years ago

I've been thinking along similar lines on high level - this approach will likely be very general and can be used to create different kinds of media, as well as potentially behaviours in a later stage.

https://metapresent.org/creation-engine

Will be interesting to base it on a decentralised open platform that could be "built-in" in the Internet.

sgt101• 6 years ago

There were a lot of evolutionary design algorithms in the 1990's with similar results. This is fun - but I don't think it is going to go anywhere.

SiempreViernes• 6 years ago

yeah, since all they wouls show were rather cruddy voxel shapes and didn't start the text with "It actually includes the physics from the start" I didn't bother read the whole thing.

But it very much looks like a very advanced way to produce something slightly less useful than a sketch on a napkin, since the latter was made with an actual understanding of what the description is supposed to mean.

JabavuAdams3• 6 years ago

Very short sighted. The underlying technology is fundamentally more powerful. Also the whole tool ecosystem has improved, so going from voxels -> surfaces is better.

WFHRenaissance• 6 years ago

Woot for Insight Data Science!

raidicy• 6 years ago

This is fantastic. Much better than my idea of just using text recognition to search a database of models. I hope this research continues so in the years to come I can slap on a VR headset and generate scenes like the holodeck.

tlack• 6 years ago

Here's the github repo for this interesting work: https://github.com/starstorms9/shape

(I couldn't find it in the linked article, might have missed it)

hyperion2010• 6 years ago

Fun. Here are some sentence that come to mind.

Flat surface with bevelled edge and 4 Ionic columns as legs that fade into nymphs at the base.

Table with caryatid legs.

Crescent wrench that doubles as a corkscrew.

Stapleremover.

afpx• 6 years ago

Very cool. But, seems like a lot of work to order to conclude that tables are rectangular flat surfaces and cups have handles and hold liquids.

ExSoax• 6 years ago

I thinked about it so many times... Really coil. I hope someday will be possible convert a sentence to a full scene

qayxc• 6 years ago

That's possible even today, kind of. [1] [2]

[1] https://t2i.cvalenzuelab.com

[2] https://towardsdatascience.com/text-to-image-a3b201b003ae

noajshu• 6 years ago

Is anyone else getting ModuleNotFoundError: No module named 'plotly'?

ebg13• 6 years ago

Yes

whichquestion• 6 years ago

It would be interesting to train the 3D Designer on more classes. Then have the text generated from something like AI Dungeon 2 feed into it and see what 3D designs it generates.

rkagerer• 6 years ago

Tea, Earl Grey, Hot!

I really want to try the live demo and make some melty furniture of my own, but I got this error:

    ModuleNotFoundError: No module named 'plotly'

starstorms5• 6 years ago

Thanks! It got overloaded with the traffic from my post but I restarted it and it's back online now so feel free to give it a try!

raobit• 6 years ago

ModuleNotFoundError: No module named 'plotly' Still getting this error

cheschire• 6 years ago

Honestly surprised I had to scroll this far to find a Star Trek reference.

The scene where the crew describes their mass dream into a horror dentist chair is forever burned into my brain as a technology I want in my lifetime.

starstorms5• 6 years ago

I loved that scene! It was actually a direct inspiration for my project here. For anyone looking it was Star Trek TNG S06E05 (Schisms)

systemvoltage• 6 years ago

Design is not just about how a thing looks or what kind of shape it has, but the intelligence that goes into how it works.

vsskanth• 6 years ago

This will be extremely useful in product search.

"sofa with 3 cushions and a round arm" should generate a model and match with similar looking products

abhinai• 6 years ago

This work is inspiring and ground breaking. I can see a lot of text description => X solutions coming up in the future. Absolutely awesome work!

britmob• 6 years ago

Can’t wait till this becomes applicable to something like AIDungeon!

windyfly• 6 years ago

The ultimate tool for story-telling.

bufferoverflow• 6 years ago

    Importing libraries...
    ModuleNotFoundError: No module named 'plotly'
    Traceback:
    File "/usr/local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 314, in _run_script
        exec(code, module.__dict__)
    File "/app/shape/streamlit_app.py", line 15, in <module>
        import plotly

AI for generative design: Plain text to 3D Designs

Related Stories