Stable Diffusion 2 Depth Guided model: architecture photos from dollhouse

https://twitter.com/justlv/status/1605276764598665217

by tosh • 4 years ago

340 50 4 years ago

liuliu2• 4 years ago

depth2img is underrated for SDv2 release, probably because AUTOMATIC1111 integration is done only 1 or 2 weeks ago.

Later this week, Draw Things will have a release with depth2img model as well, initially only supporting iOS-provided depth map, but will do MiDaS inferred depth map soon. Going to be exciting to see what people comes up with.

Der_Einzige• 4 years ago

Does this work with dreambooth automatic webUI extension yet? 2.1 doesn't work

liuliu2• 4 years ago

I think, we may be able to apply LoRA trick to transfer custom dreambooth models (trained with SD v2) back to depth guide models.

DethNinja2• 4 years ago

Depth Guided models are amazing for game art texturing: https://hackaday.com/2022/12/18/image-generating-ai-can-text...

I already incorporated this to Houdini to generate 3D distance meshes and it works wonders. If anyone is interested in experimenting with this, I really recommend Hugging Face diffusers library.

I wonder when we will be able to generate studio quality 3D models from a single 2D image. I know there are solutions that can do this but it is still nowhere near professional quality.

robertlagrant• 4 years ago

It looks like the artwork from that game Psychonauts. Awesome.

corysama• 4 years ago

Really obvious use that took too long to occur to me: Make a bunch of textures for your game. Label them and use them to train a specialized model file. Use that to make more textures for your game.

It’ll get you 80% of the way there for each texture really fast. Would work really well if you are making a ton of NPCs and made a head model and a separate clothes model.

dwallin3• 4 years ago

This is really cool, and I can immediately see a lot of way to take this further and improve some of it's weaknesses.

To solve the issue with hidden faces (right now it just projects out copies of the visible textures, you could combine this with inpainting: Rotate the scene / model to expose untextured faces sides that get masked out and inpainted, then rinse and repeat until the model is fully textured.

Another interesting (albeit much harder) thing to try would be to take the image output from SD, run it through MiDaS to get a new depth map with additional details. Diff the depth maps in 3d space and update the geometry to match before projecting the image. Combine it with the first suggestion and you would have a process for progressively refining a detailed, fully textured model.

bryced• 4 years ago

Another interesting (albeit much harder) thing to try would be to take the image output from SD, run it through MiDaS to get a new depth map with additional details. Diff the depth maps in 3d space and update the geometry to match before projecting the image. Combine it with the first suggestion and you would have a process for progressively refining a detailed, fully textured model.

I made a go at this exact approach last night. I'm new to working with 3d data but at least got a mesh rendered from the depth map. Spent most of my time fighting with tooling. Let me know if you want to help out.

matthewfcarlson2• 4 years ago

Do you have that written up anywhere? That sounds amazing as I'd love to see mesh generation in Houdini.

DethNinja2• 4 years ago

It is commercial code, so I can’t share it. However, algorithm is really simple to implement in Houdini:

* Generate depth image from 3D meshes

* Send depth image to local HTTP server running Hugging Face diffusers library. Generate textured image.

* Cast textured image to a 2D plane surface that covers the camera extends.

* Traverse the UV surface of the 3D meshes and find closest point on the 2D stable diffusion image plane. Copy the RGB value to UV texture and you’re done.

CyanBird• 4 years ago

Seems that the texture generation is still being relegated to image generation rather than sampling the curvature, AO or normal maps?

Thank you for sharing, this is just incredible

joewhatkins• 4 years ago

We got this working in Three.js - it works great for painting scenery and backgrounds.

FarhadG• 4 years ago

Care to share more? Would love to see how it's being leveraged in that space

matthewfcarlson2• 4 years ago

I'd love to see this as well

isoprophlex• 4 years ago

The pace and extent of innovation in this space is breathtaking. Almost every day there's a new super original idea showing up, that builds on something we just simply didn't have four months ago.

The open sourcing of these models was of course instrumental to this. So thanks Huggingface et al., I guess!

charcircuit• 4 years ago

Stable Diffusion is not an open source model.

bandana• 4 years ago

It is

IanCal2• 4 years ago

It's very clear how the discussion goes from here so it's really worth short circuiting it and providing the detail.

The models are openly available but the license places some restrictions on the use.

You agree not to use the Model or Derivatives of the Model:
- In any way that violates any applicable national, federal, state, local or international law or regulation;
- For the purpose of exploiting, harming or attempting to exploit or harm minors in any way;
- To generate or disseminate verifiably false information and/or content with the purpose of harming others;
- To generate or disseminate personal identifiable information that can be used to harm an individual;
- To defame, disparage or otherwise harass others;
- For fully automated decision making that adversely impacts an individual’s legal rights or otherwise creates or modifies a binding, enforceable obligation;
- For any use intended to or which has the effect of discriminating against or harming individuals or groups based on online or offline social behavior or known or predicted personal or personality characteristics;
- To exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
- For any use intended to or which has the effect of discriminating against individuals or groups based on legally protected characteristics or categories;
- To provide medical advice and medical results interpretation;
- To generate or disseminate information for the purpose to be used for administration of justice, law enforcement, immigration or asylum processes, such as predicting an individual will commit fraud/crime commitment (e.g. by text profiling, drawing causal relationships between assertions made in documents, indiscriminate and arbitrarily-targeted use).

https://github.com/CompVis/stable-diffusion/blob/main/LICENS...

For many, this distinction is more of either an academic one or one where we are OK with those kind of restrictions. Where I live if I did many of these things I would be either criminally or civilly (the right phrasing?) liable so having a license that tells me I can't break the law is a little redundant. I think 4 is possibly legal but an edge case and the last one is nation state level.

You are probably technically correct, but the context here is very important IMO.

prvc3• 4 years ago

I wish Stable Diffusion had focused on more fine art and professional photography in their training corpus, like DALL-E did in its own one. All outputs I've seen have a decidedly low-quality or amateurish look, which must have been picked up from somewhere.

orbital-decay3• 4 years ago

Thing is, the primary power of large models is not in the styles they memorized, but in the ability to be fine-tuned and guided by reference content. The focus is your choice, you can shift the model bias with any available finetuning method and a certain amount of reference images. 99% of the training has already been done for you, so all you need is a relatively subtle push achievable on a single GPU (or a few GPUs).

Take a look at Analog Diffusion outputs, for example.

dwringer• 4 years ago

With patience the guidance can even be done simply, without training, with enough prompts, blending of prompts, input images to img2img, etc. I'm not an expert on art history, but those who are seem able to get just about any style on any image with just the base stable diffusion model. Adding specific photography terms can emulate some of the results from the analog diffusion model in regular SD as well.

prvc3• 4 years ago

It has nothing to do with "style". DALL-E 2 consistently astonishes with its composition, framing, variety of visual elements, and so on, which tend to also be good in oil paintings and professional photographs, but not at all in the sort of content which predominates on art sharing apps with user-generated content.

andybak• 4 years ago

That doesn't match my observations. I've seen a very wide range of styles out of Stable Diffusion (and I've probably viewed mulitple thousands)

l33tman2• 4 years ago

You're just prompting it wrong. Try adding qualifiers that have well-known associations with pro-level photography. For example these two makes your prompts incredibly much better, append either "Canon EOS 5D Mark IV, Sigma 85mm f/1.4" or "photography by Annie Liebovitz" (for a bit more artsy photos) to the end of your prompt and see the difference.

If you're not doing portrait photography, replace Sigma 85mm f/1.4 with something else for example Sigma 24mm for more wide-angle photography.

lelandfe• 4 years ago

Note that in my extensive testing specific lenses and cameras tend to make broad, random changes to the output unless they tend to have a specific aesthetic associated with them.

E.g. 85mm vs 24mm make no specific changes to a photo. SD appears to just interpret these as "make a photo look realistic", and any changes to the photo as you switch between them are simply incidental.

nickthegreek2• 4 years ago

You need to look at a better source. SD 2.x is trained on alot of professional photography. There is plenty of output which is not low-quality/amateurish. There is also the Analog Diffusion model is the sd1.5 fine tuned for film photography.

https://huggingface.co/wavymulder/Analog-Diffusion

andrewon• 4 years ago

A primer to Depth model in Stable Diffusion 2: https://stable-diffusion-art.com/depth-to-image/

monkeydust• 4 years ago

Can this be used for virtual 2d try on of clothing?

bzzzt• 4 years ago

Looked impressive until I saw the guitars ;)

pea• 4 years ago

On a similar note, has anyone had any luck using these models to try different paint colourings in their house (from uploading photos of their rooms)? I've struggled to do this with SD, and the non-AI apps aren't great at it either.

herendin2• 4 years ago

When you afford SD img2img influence with a high strength parameter, so it can change the colors of a photo, you also are letting it have enough influence to redesign the room at random and not just changing the colors

Dreambooth trained on the ground truth photos might help a bit

AuryGlenz• 4 years ago

I’d love for there to be a form of frequency separation, where we could keep the details but mess with the colors/less major details.

Photographers have been attempting to emulate various films in different ways forever. Stable diffusion could do it absolutely perfectly.

CuriouslyC• 4 years ago

Try converting the images to grayscale, then setting the denoising strength very low and prompting for the exact colors you want things to be.

nickthegreek2• 4 years ago

This seems much easier in an app like photoshop/affinity if you want to keep everything really static. Or by using a combination of photoshop and then SD to clean up.

usrusr• 4 years ago

Take the depth field from life size furniture instead, all covered in chroma key surface and make "we never use the same crazy virtual studio twice, not even the arrangement" the identity of your fixed camera show?

ramoz• 4 years ago

If you like using Colab, here’s a link to a notebook that spins up a nice simple demo of the depth model.

https://github.com/backnotprop/Colab-Stable-Diffusion-2-Dept...

itronitron• 4 years ago

coming soon to an AirBnB listing near you :)

chx2• 4 years ago

Isn't all AI art stolen art?

xwdv2• 4 years ago

Good artists borrow, great artists steal.

chx2• 4 years ago

Yeah trivialize away ruining many artists livelihoods with this bullshit. But what did I expect from HN bros.

xwdv2• 4 years ago

When will AI artists get fast enough that they can generate imagery as you type?

Stable Diffusion 2 Depth Guided model: architecture photos from dollhouse

Related Stories