Explaining the SDXL Latent Space

https://huggingface.co/blog/TimothyAlexisVass/explaining-the-sdxl-latent-space

by thesephist • 2 years ago

163 33 2 years ago

huggingface.co

Sabinus• 2 years ago

That's very cool. I had no idea the latent space was that accessible and obviously manipulatable.

Also interesting is how the way sdxl structures latents affects how it thinks about images.

ttul5• 2 years ago

I’ve been playing with diffusion a ton for the past few months, writing a new sampler that implements an iterative blending technique described in a recent paper. The latent space is rich in semantic information, so it can be a great place to apply various transformations rather than operating on the image directly. Yet it still has a significant spatial component, so things you do in one spatial area will affect that same area of the image.

Stable Diffusion 1.5 may be quite old now, but it is an incredibly rich model that still yields shockingly good results. SDXL is newer and more high tech, but it’s not a revolutionary improvement. It can be less malleable than the older model and harder to work with to achieve a given desired result.

fpgaminer• 2 years ago

It can be less malleable than the older model and harder to work with to achieve a given desired result.

That has been my experience as well. It's frustrating because SDXL can be exquisite, but SD 1.5 is more "fun" to work with and more creative. I can throw random ideas into a mish-mash of a prompt and SD 1.5 will output an array of interesting things while SDXL will just seem to fall back to something "reasonable", ignoring anything "weird" in the prompt. SDXL also seems to have a lot more position bias in the prompt. SD 1.5 had a bit of that, paying more attention to words earlier in the prompt, but SDXL takes that to a new level.

But SDXL can draw hands consistently, so ... it's a tough choice.

designium• 2 years ago

With comfyui, you can do SDXL > SD1.5 or SD1.5 > SDXL, it makes more sense to generate basic image in SDXL Turbo and apply the effects of a checkpoint later.

lrasinen• 2 years ago

But SDXL can draw hands consistently, so ... it's a tough choice.

Looking at the article photos it still has some way to go. I counted 3 cases of missing fingers, two cases of extra fingers (on the cartoon girl), and a few arm poses that in real life would need medical attention.

countWSS• 2 years ago

SDXL(and possibly 2.1) switched to different CLIP implementation that is geared for sentence-level understanding, SD1.5 uses old CLIP that works with tag-cloud type prompts.

3abiton• 2 years ago

I am curious what do you mean by high tech?

ttul5• 2 years ago

SDXL is a more sophisticated model architecture. It has more layers. The CLIP model is bigger.

wruza• 2 years ago

Entered this thread to write your comment. I find SDXL inferior to 1.5 and yes, much harder to work with.

My another issue is that sdxl images that you can see on the web always have that “from a movie/ads”-?ish? coating. Can’t explain it, but it feels even more uncanny than 1.5.

SDXL is too resource-hungry for what it produces. 3x+ model sizes, 12GB vram is barely enough for it, 40 steps is the minimum, and I don’t think training loras will turn out feasible at all. I can’t lower the resolution without distortions, and even proportions are hard to deal with. It feels much less flexible than 1.5 in this regard.

I’m sticking with 1.5, no sdxl plans.

ttul5• 2 years ago

Particularly considering the rich world of SD1.5 fine-tunes, SDXL leaves so much to be desired. I'm sure it will all be sorted out eventually, but right now, the momentum in the community just isn't there with SDXL the way it is with 1.5.

nomel2• 2 years ago

What's the reason for using RGB rather than, say, HSV? RGB seems like it would be fairly discontinuous. Or, do I have that backwards?

Lerc2• 2 years ago

I think there might be an opinion that since most colour space conversions can be expressed with relatively small neural nets (since they are mostly accumulations of variously scaled values), the autoencoder can dedicate a negligible proportion of its parameters towards that job and that gives it the potential to choose whatever color space training dictates.

I'm not entirely convinced by this idea myself. I have seen a few networks where a range of -1..1 inputs do a lot better than inputs in the range of 0..2 even though translation should be an easyish step for the network to figure out. The benefit from preprocessing the inputs, to me seem to be more advantageous than my common sense tells me it should be.

shenberg• 2 years ago

I suspect that weight initializations are geared towards inputs being normal random variables with mean 0 and variance 1. Deviating from that makes the learning process unhappy.

Dwedit• 2 years ago

It's more like YCbCr or YCgCo color space than RGB.

godelski• 2 years ago

The format isn't explicit to the network. But the data trained on is usually in RGB format, so probably the reasoning. I found a repo where someone tried different formats but it's wroth noting that this was for discrimination so just because it can discriminate doesn't mean it does the same thing. Maybe I'll run some experiments. You could use a UNet for classification and then look at the bottom layer and do the same thing. Be hard to do with SD (or SDXL) because you'd need to retrain with the format. Tuning could possibly work but the network would likely be biased to understand the RGB encoding.

Edit: ops, forgot the link

https://github.com/ducha-aiki/caffenet-benchmark/blob/master...

viraptor2• 2 years ago

But the data trained on is usually in RGB format, so probably the reasoning.

It's trivial to convert the values for training - basically 0% of the cost of the process. But there's likely more "meaning" in HSV than in RGB. So I don't think that would account for the difference.

Lerc2• 2 years ago

I don't think it's as simple as this naive approach suggests, but it's a good preliminary analysis. It's a good lesson that while being absolutely correct might be quite difficult, diving in and having a go might get you further than you think.

01HNNWZ0MV43FF• 2 years ago

All the patterns and textures are expressed by only one dimension? Bizarre.

l33tman2• 2 years ago

It's only patterns and textures for 8x8 images, so I guess it could make sense, you're not going to need every conceivable pattern of 8x8 pixels in normal images..

If you quantize those 4 floats per 8x8 block, is that encoding better than say the old venerable JPG 8x8 DCT + quant?

SV_BubbleTime2• 2 years ago

That was an excellently written article.

I for sure thought a discussion about latent spaces would instantly be over my head. It was, but took a few paragraphs.

Der_Einzige• 2 years ago

Anyone know if the work shown here has been implemented in Automatic1111 or ComfyUI as an extension? If not, than that might be my first project to add since these are quite simple (relatively speaking) in the code to implement.

mzz80• 2 years ago

What’s your github username? I’d be interested in using that when it’s released.

HanClinto• 2 years ago

Thank you for the excellent article! Top notch work!

rgmmm• 2 years ago

Enhance.

l33tman2• 2 years ago

Just a terminology comment here. "Latent space" means a lot of different things in different models. For a GAN for example it actually means the "top concept" space where you can change the entire concept of the image by moving around in the latent space, which is notoriously difficult. For SD/SDXL it refers to the bottommost layer just above pixelspace, which expands the generated image from 64x64 to 512x512 pixels in the case of SD1.5.

This allows the rest of the network to be smaller while still generating a usable output resolution, so it's a performance "hack".

It's a really good idea to explore it and hack into it like in the article, to "remaster" the image so to speak!

Explaining the SDXL Latent Space

Related Stories