Diffusion Models

https://andrewkchan.dev/posts/diffusion.html

315 31 2 years ago

ilaksh6• 2 years ago

What's the best Apache or MIT-licensed python library for Diffusion Transformers?

HuggingFace Diffusers is Apache and supports Diffusion Transformers: https://huggingface.co/docs/diffusers/en/api/pipelines/dit

ilaksh6• 2 years ago

They are actually based on the Attribution - Non-commercial Facebook code and have the same license.

eli_gottlieb2• 2 years ago

Besides huggingface there's also the DDPT repo: https://github.com/lucidrains/denoising-diffusion-pytorch/

ilaksh6• 2 years ago

Nice. But is that a diffusion transformer?

ilaksh6• 2 years ago

I found this:

https://paperswithcode.com/paper/scalable-diffusion-models-w...

https://github.com/mindspore-lab/mindone/tree/master/example...

sidcool• 2 years ago

This is a great post

b33j0r• 2 years ago

I really like how readable the layout is. (Why do I waste so much time making hard-to-read layouts?)

The only disappointment came when I hit Reader View—almost to prove “this page is semantically perfect!!”—and alas, the nav list has a line height less than one in that medium, and scrunches up all crazy. I’ll let it slide ;)

sashank_1509• 2 years ago

Good post, I always thought diffusion originated from score matching, today I realized diffusion came before score matching theory, so when OpenAI trained on 250 million images, they didn’t even have great theory explaining why they were modeling the underlying distribution. Gutsy move

reasonableklout6• 2 years ago

The original Dickstein 2015 paper [1] formulated diffusion as maximizing (a lower bound of) log-likelihood of generating the distribution, so there was some theory. But my understanding is that the breakthrough was empirical results from Ho [2] and Nichol [3] showing diffusion could produce not only high-quality samples but better than GANs in some cases.

[1] https://arxiv.org/abs/1503.03585 [2] https://arxiv.org/abs/2006.11239 [3] https://arxiv.org/abs/2105.05233

Tao3300• 2 years ago

I spent 2022 learning to draw and was blindsided by the rise of AI art models like Stable Diffusion. Suddenly, the computer was a better artist than I could ever hope to be.

I hope the author stuck with it anyway. The more AI encroaches on creative work, the more I want to tear it all down.

ctippett• 2 years ago

Conversely I've become more motivated to draw things and try my hand at digital art since being exposed to Stable Diffusion, Midjourney et. al. I take the output from these tools and then attempt to recreate or trace over them.

arvinsim• 2 years ago

People who do art for art's sake will do it regardless of AI.

After all, photography didn't stop people from drawing or painting.

davidguetta3• 2 years ago

The train loop is wrong no ? neither x0s and eps are used in the expression of xts so it loons like your training to predict random noise

fisian• 2 years ago

Yes, should be the same as the equation before. Like this:

  xts = alpha_bar[t].sqrt() * x0s + (1.-alpha_bar[t]).sqrt() * eps

Additionally, the code isn't consistent. In the sampling code a time embedding is used, while in training it isn't.

reasonableklout6• 2 years ago

Oops, you're right. Fixed, thanks.

kmacdough2• 2 years ago

Not sure which eq you refer to, but from what I understand, the network never network "sees" the correct images. Rather, the network must learn to infer the information indirectly through the loss function.

The loss function encodes information about the noise and, because the network sees the noised up image exactly, this is equivalent to learning about the actual sample images. It's worth noting that you could design a loss function measuring the difference between the output and the real images. This contains equivalent information, but it turns out that the properties of gaussian noise make it much more conducive estimating the gradient.

But point being, the information on the true images is in the loop albeit only through the lense of some noise.

kmacdough2• 2 years ago

Thanks for sharing. This has given me much more insight into how and why diffusion models work. Randomness is oddly powerful. Time to try and code one up in some suitably unsuitable language.

Not much to TL;DR for the comment lurkers. This post is the TL;DR of stable diffusion.

davidguetta3• 2 years ago

How is doing classifier-free guidance where you

"Train a single diffusion model on every training sample x0x0 twice: once paired with its class label yy, and once paired with a null class label."

not doing exactly the same, and having the same problem that was deemed bad in the first paragraph of the same section:

"However, the label can sometimes lead to samples that are not realistic or lack diversity if the model has not seen enough samples from p(x∣y)p(x∣y) for a particular yy. So we often want to tune how much the model “follows” the label during generation."

Awesome post btw

reasonableklout6• 2 years ago

It's not quite the same thing because you are able to control the tradeoff of realism and class strength, since sampling is formulated as pushing an unconditional sample in the direction of the conditional one for a tunable amount, rather than simply sampling from the conditional distribution. However, you are right that we get similar problems with realism if we apply guidance with too much strength.

davidguetta3• 2 years ago

Mm. I understand the argument (tho this subletly is not present in the article, maybe you should add it if that's actually what people do).

That said I am not 100% convinced : first the underlying model is still the same and because you are effectively learning a eps(x, t, y). secondly the equivalence suggests (if you actually use the difference as gradient) you should end up in the same global min/max as if you were doing this directly no ?

Diffusion Models

Related Stories