A working implementation of text-to-3D DreamFusion, powered by Stable Diffusion

https://github.com/ashawkey/stable-dreamfusion

by nopinsight • 4 years ago

286 71 4 years ago

jrmann1002• 4 years ago

[Edited] The original Dreamfusion project was discussed here a few days ago: https://news.ycombinator.com/item?id=33025446

naet• 4 years ago

This is a different project/implementation, based on the open source stable diffusion instead of proprietary google imagegen

jrmann1002• 4 years ago

Edited; thanks for clarifying!

That makes this Stable-Dreamfusion adaptation even more promising.

nobbis4• 4 years ago

Took a week, as predicted: https://twitter.com/DvashElad/status/1575614411834011651

nobbis4• 4 years ago

Key step in generating 3D – ask Stable Diffusion to score views from different angles:

  for d in ['front', 'side', 'back', 'side', 'overhead', 'bottom']:
    text = f"{ref_text}, {d} view"

https://github.com/ashawkey/stable-dreamfusion/blob/0cb8c0e0...

shadowgovt3• 4 years ago

I'm modestly surprised that those few angles give us enough data to build out a full 3D render, but I guess I shouldn't be too surprised, as that's tech that has had high demand and been understood for years (those kind of front-cut / side-cut images are what 3D artists use to do their initial prototypes of objects if they're working from real-life models).

dwallin• 4 years ago

Given the way the language model works these words could have multiple meanings. I wonder if training a form of textual inversion to more directly represent these concepts might improve the results. You could even try teaching it to represent more fine grained degree adjustments.

bergenty• 4 years ago

I can’t wait to generate novel 3D models to CNC/3D print! Can these be exported out as STL/OBJs?

moron4hire• 4 years ago

In the usage notes, there's a line that mentions

  # test (exporting 360 video, and an obj mesh with png texture)
  python main_nerf.py --text "a hamburger" --workspace trial -O --test

So I guess so. That's pretty awesome.

samstave2• 4 years ago

Heckin A!

I just got my 3D printer and was a bit too tipsy to assemble it the day it arrived - and have several things I want to print…

It will be interesting to experiment with describing the thing I want to print with text instead of designing it in SolidEdge and see what AI thinks….

I wonder if you can feed it specific dimensions?

“A holder for a power supply for an e bike with two mounting holes 120mm apart with a carry capacity that is 5 inches long and 1.5 inches deep”

wongarsu2• 4 years ago

Well, here's 16 attempts of regular Stable Diffusion with that prompt [1], and here's what it things a technical drawing of it might look like [2].

Maybe two papers down the line :D For now you might have more luck with something less specific.

1: https://i.imgur.com/RPNCwyM.png

2: https://i.imgur.com/c9pfM8U.png

wongarsu2• 4 years ago

There's a small gallery of success and failure cases here [1].

It certainly doesn't look as good as the original, yet. I wonder if that's due to the implementation differences noted, less cherry picking in what they show, or inherent differences between Imagen and Stable Diffusion.

Maybe Imagen just has a much better grasp of how images translate to actual objects, where Stable Diffusion is stuck more on the 2d image plane.

1: https://github.com/ashawkey/stable-dreamfusion/issues/1

nopenopenopeno• 4 years ago

One cannot help but notice the success cases are expected to have symmetry along at least one axis, whereas the failure cases are not.

namarie• 4 years ago

Aren't squirrels and frogs expected to have an axis of symmetry? I think the reason for the failures is the presence of faces; it seems to be trying to make a face visible from all angles.

shadowgovt3• 4 years ago

I feel like the Cthulhu head is extra-successful, given the subject matter.

Non-Euclidean back-polygon imaging? Good work, algorithm. ;)

codeflo• 4 years ago

I guess a lot can be done to force the model to create properly connected 3D shapes instead of these thin protruding 2D slices. But I noticed something else. Some of the angles “in between” the frog faces have three eyes. I wonder if part of the issue might be that those don’t look especially wrong to Stable Diffusion. It’s often surprisingly confused about the number of limbs it should generate.

Cook4986• 4 years ago

Currently working with a student group to build out a 3D scene generator (https://github.com/Cook4986/Longhand), and the prospect of arbitrary, hyper-specific mesh arrays on demand is thrilling.

Right now, we are relying on the Sketchfab API to populate our (Blender) scenes, which is an imperfect lens through which to visualize the contents of texts that our non-technical "clientele" are studying.

Since we are publishing these scenes via WebXR (Hubs), we have specific criteria related to poly counts (latency, bandwidth, etc) and usability. Regarding the latter concern, it's not clear that our end users will want to wait/pay for compute.

*copyedited

sirianth• 4 years ago

wow

GistNoesis2• 4 years ago

Stable-Diffusion is a latent diffusion model, which diffuses in a latent space instead of the original image space. Therefore, we need the loss to propagate back from the VAE's encoder

There is also an alternative way to handle this latent difference with the original paper that should also work :

Instead of working in voxel color space, you push the latent to the voxel (Aka instead of having a voxel grid of 3d rgb color, you have a voxel grid of dimlatent latents, (you can also use spherical harmonics if you want as it works just the same in nd) ).

Only the color prediction network differ, the density is kept the same.

The NERF then directly render to the latent space (so there are less rays to render) which mean you need to decode it with the VAE only for visualization purposes and not in the training loop.

hwers4• 4 years ago

This sounds really interesting but I’m not sure I follow. Having a hard time expressing how I’m confused though (maybe its unfamiliar nerf terminology) but if you have the time I’d be very interested if you could reformulate this alternative method somehow (I’ve been stuck on this very issue for two days now trying to implement this myself).

taneq• 4 years ago

NERF is neural radiance fields (the neural 3D reconstruction method that nVidia published recently.)

Basically if I’m reading it right, this does the synthesis in latent space (which describes the scene rather than rendering vocals) then translates it into a NERF. It sounds kind like the Stable Diffusion description that was on here earlier.

going_ham• 4 years ago

Do you have refereneces to NeRF papers that can directly compute the field in the latent code? Because it NeRF based methods are essentially performing solving the rendering equation to learn the mapping, what could be alternative equation for directly learning latent code? Your idea is interesting, could you elaborate on it!

GistNoesis2• 4 years ago

Sorry no references for NeRF papers. But the idea is not new. My experience was originally with PointClouds and 3d keypoint-feature slam.

In these, you represent the world as a sparse latent representation, collection of 3D coordinates, and their corresponding SIFT feature descriptor. You rotate and translate these keypoints to obtain a novel view of the features in 2D image. (The descriptor can be taken as an interpolate of the descriptors weighted by the difference in orientation between views as features only match for small viewing angle difference (like 20°) ). And then you could invert features to retrieve a pixel-space image (for example https://openaccess.thecvf.com/content_cvpr_2016/papers/Dosov... ) (although it's never needed in practice.

Coming back to NERF, it's the same principle. When your NERF has converged, if you don't have transparent object, along the ray the density will be 0 except when you intersect the geometry where only a single voxel will be hit in which case you fetch the latent stored in the voxel latent (spherical harmonics) from the direction given by the ray during the training with a latent image.

The rendering equation is still the same but instead of rendering a single ray, it would be analog to rendering a group of close rays in order to render a patch of image, of which the latent is a compressed representation. You have to be careful not to make the patch too big, because like with a lens in the real world, spherical transform flip the patch-image upon translation, but neural network should transparently handle this.

The converged representation is an approximation based on linear approximation and interpolation along positions and ray direction, provided that you have enough resolution, you can construct it manually from the solution and see how it behaves in the rendering.

Will the convergence process work ? It will depend on how well latent mix, along a ray. The light transport equation is usually linear, and latent usually mix well linearly (even more so when weighted by a density), But in the case it doesn't mix well you can learn a mixing of latent rule that help it converge.

Also once you have a latent Nerf, it won't allow you directly to obtain a STL/obj directly but you should have 3d consistent views from which you could render a classical NERF, but you can/should also instead optimize for the classical voxel grid, that fit the latent voxel grid (aka that give the same image patches).

sdwvit• 4 years ago

All these news on image/3d/video generation just show how we live in the middle of ai/ml breakthrough. Incredible to see news with extreme progress in the field like this popping everyday.

dmingod666• 4 years ago

Browse Lexica.art your mind will be blown by the range and amount of detail on some of the art.

Like this (nsfw content): https://lexica.art/?q=Intricate+goddess

There is an addictive and trippy quality to this and it is yet to hit mainstream -- The art itself is stunning but it goes beyond that, the ability to nudge it around and make variations to it is incredible. now add the fact that you can train it with your own content. people are going to go bonkers with this and it's going to open up a lot of debates too.

egypturnash5• 4 years ago

Jesus. Well thanks for your contribution to putting the entire creative industry out of work, I guess, little anime girl icon person. Ugh.

ccity88• 4 years ago

How very pessimistic. We should never shirk technological progress for fear of upsetting the status quo or established agenda. All of this is only a matter of time away from emerging. Have fun being on the forgotten side of history

axg11• 4 years ago

A new technology is developed with the potential to make you 100x more efficient at your job. Today, a creative artist can only contribute to a project through a narrow slice. Tomorrow, the same creative artist can single-handedly orchestrate an entire project.

astrange• 4 years ago

It's more like there were tasks that were previously so unproductive they couldn't be done at all, and now they're productive enough you might be able to be employed doing them.

Automation creates jobs rather than destroying them. What destroys jobs is mainly bad macroeconomic conditions.

uni_rule• 4 years ago

This is a force multiplayer. It doesn't take the place of artistic intent, dingus. Besides you can't accomplish much with just "a model". This is an asset generator, hardly a threat to anyone especially when these things will likely need some weight painting to touch up anyway.

mrtranscendence2• 4 years ago

I don’t think the commenter is upset that this particular model will be deployed, putting creative professionals out of work. It’s clearly a janky proof of concept. I think they’re upset about what follow on work could eventually mean.

cercatrova• 4 years ago

Another day, another AI media generation project, and yet another comment by egypturnash lamenting the "death of the creative industry."

egypturnash5• 4 years ago

representative of the industry currently under threat of disruption is not happy about this and continues to be vocal about her unhappiness, film at 11

tluyben24• 4 years ago

I have a friend who is an old school professional artist from before affordable computing who has been using AI (and computers before that) to aid him with creations for many years now and runs everything himself on his own machines (which is a pretty expensive setup) experimenting and training and he loves every iteration.

But I guess what creative industry means to you? Pumping out web UIs or 3d gaming models were never, for the most part, the creative industry; learning to see what people like and copying that for different situations is not necessarily creative and thus what AI easily does; anything that doesn’t come with a lot of learning and practice and talent outside manual work will be replaced by AI soon; the other stuff will take somewhat longer.

If you think this can replace you, you weren’t/aren’t in the creative industry. Same goes for coders afraid of no code.

egypturnash5• 4 years ago

So how many shitty "not creative industry" jobs did your friend take on the way to where he could have "a pretty expensive setup" to do this? What did he crank out solely to earn a paycheck with his art skills?

yieldcrv• 4 years ago

tent cities by the beach can use the showers

Geee• 4 years ago

Would be cool to see it adapted to img2img scenario, using one or more 'seed images'. It would be closer to a standard NERF, but also would be able to imagine novel angles and guidance with prompt.

etaioinshrdlu2• 4 years ago

How long does it take? 5 hours?

etaioinshrdlu2• 4 years ago

I think it would be interesting to convert to a polygon mesh periodically in-the-loop. It could end up with more precise models.

xwdv• 4 years ago

Feels like AI generated art is approaching a sort of singularity at this point. Progress getting very exponential.

antegamisou• 4 years ago

Only if you have low standards of what constitutes research on artificial intelligence and art perception.

hwers4• 4 years ago

These things take months and months to train (hardly fast progress). Any new model that’s coming out is generally known in the atmosphere (not unpredictable) and these applications were pretty expected the day stable diffusion came out.

bpodgursky• 4 years ago

"months and months"

At the beginning of this year, most technical people would have told you that graphic design was a decade from being automated, and creative video production more.

Now we are at "months and months".

whazor• 4 years ago

I would love a AI model that does text-to-SVG.

fsiefken• 4 years ago

that would be lovely! perhaps it can also dream up interesting game of life patterns

WASDx• 4 years ago

OpenAI Codex can do that.

baxtr• 4 years ago

Can someone explain the significance of this? I am not familiar what DreamFusion is.

hwers4• 4 years ago

Only downside with this is that each mesh takes like 5 hours to generate (on a v100 too). Obviously it’ll speed up but we’re far from the panacea

thehappypm• 4 years ago

Omg, imagine how useful this would be for video games or movies. Whipping up an asset in a matter of hours of computer time? Amazing

verbify• 4 years ago

Or any sort of CAD

gpm2• 4 years ago

Maybe I just don't understand what goes into CAD (probably even), but I don't see it? CAD needs to spit out precise dimensions for things at the end, while this sort of AI is just going to spit out "something more or less like this", it doesn't seem likely to be usable?

dannyw• 4 years ago

Incredible for indie developers especially. It's truly democratising.

intelVISA• 4 years ago

Finally I can become a one man AAA game factory...

A working implementation of text-to-3D DreamFusion, powered by Stable Diffusion

Related Stories