Diffusion models explained simply

https://www.seangoedecke.com/diffusion-models-explained/

by onnnon • 1 year ago

168 36 1 year ago

https://lilianweng.github.io/posts/2021-07-11-diffusion-mode...

Philpax• 1 year ago

Notably, Lilian did not explain diffusion models simply. This is a fantastic resource that details how they actually work, but your casual reader is unlikely to develop any sort of understanding from this.

Y_Y2• 1 year ago

your casual reader is unlikely to develop any sort of understanding [from this]

"Hell, if I could explain it to the average person, it wouldn't have been worth the Nobel prize." - Richard Feynman

kmitz• 1 year ago

Thanks, I was looking for an article like this, with a focus on the differences between generative AI techniques. My guess is that since LLMs and image generation became mainstream at the same time, most people don't have the slightest idea they are based on fundamentally different technologies.

cubefox3• 1 year ago

That's a nice high-level explanation: short and easy to understand.

cubefox3• 1 year ago

It's nice that this contains a comparison between diffusion models that are used for image models, and the autoregressive models that are used for LLMs.

But recently (2024 NeuIPS paper of the year) there was a new paper on autoregressive image modelling that apparently outperforms diffusion models: https://arxiv.org/abs/2404.02905

The innovation is that it doesn't predict image patches (like older autoregressive image models) but somehow does some sort of "next scale" or "next resolution" prediction.

In the past, autoregressive image models did not perform as well as diffusion models, which meant that most image models used diffusion. Now it seems autoregressive techniques have a strict advantage over diffusion models. Another advantage is that they can be integrated with autoregressive LLMs (multimodality), which is not possible with diffusion image models. In fact, the recent GPT-4o image generation is autoregressive according to OpenAI. I wonder whether diffusion models still have a future now.

earthnail• 1 year ago

From what I can tell, it doesn't look like the recent GPT-4o image generation includes the research of the NeurIPS paper you cited. If it did, we wouldn't see a line-by-line generation of the image, which we do currently in GPT-4o, but rather a decoding similar to progressive JPEG.

I'm not 100% convinced that diffusion models are dead. That paper fixes autoregression for 2D spaces by basically turning the generation problem from pixel-by-pixel to iterative upsampling, but if 2D was the problem (and 1D was not), why don't we have more autoregressive models in 1D spaces like audio?

famouswaffles2• 1 year ago

From what I can tell, it doesn't look like the recent GPT-4o image generation includes the research of the NeurIPS paper you cited. If it did, we wouldn't see a line-by-line generation of the image, which we do currently in GPT-4o, but rather a decoding similar to progressive JPEG.

You could, because it's still autoregressive. It still generates patches left to right, top to bottom. It's just that we're not starting with patches at the target resolution.

cubefox3• 1 year ago

From what I can tell, it doesn't look like the recent GPT-4o image generation includes the research of the NeurIPS paper you cited.

Which means autoregressive image models are even ahead of diffusion on multiple fronts, i.e. both in whatever GPT-4o is doing and in the method described in the VAR paper.

rudedogg• 1 year ago

From what I can tell, it doesn't look like the recent GPT-4o image generation includes the research of the NeurIPS paper you cited. If it did, we wouldn't see a line-by-line generation of the image, which we do currently in GPT-4o, but rather a decoding similar to progressive JPEG.

Going off my bad memory, but I think I remember a comment saying the line-by-line generation was just a visual effect.

famouswaffles2• 1 year ago

The innovation is that it doesn't predict image patches (like older autoregressive image models) but somehow does some sort of "next scale" or "next resolution" prediction.

It still predicts image patches, left to right and top to bottom. The main difference is that you start with patches at a low resolution.

porphyra• 1 year ago

Meanwhile, if you want diffusion models explained with math for a graduate student, there's Tony Duan's Diffusion Models From Scratch.

[1] https://www.tonyduan.com/diffusion/index.html

bcherry• 1 year ago

"The sculpture is already complete within the marble block, before I start my work. It is already there, I just have to chisel away the superfluous material."

- Michelangelo

jdthedisciple• 1 year ago

Not to be that guy but an article on diffusion models with only one image ... and that too just noise?

ActorNightly2• 1 year ago

The thing to understand about any model architecture is that there isn't really anything special about one or the other - as long as the process differentiable, ML can learn it.

You can build an image generator that basically renders each word on one line in an image, and then uses a transformer architecture to morph the image of the words into what the words are describing.

They only big difference is really efficiency, but we are just taking stabs at the dark at this point - there is work that Google is doing that eventually is going to result in the most optimal model for a certain type of task.

noosphr• 1 year ago

Without going into too much detail: the complexity space of tensor operations is for all practical purposes infinite. The general tensor which captures all interactions between all elements of an input of length N is NxN.

This is worse than exponential and means we have nothing but tricks to try and solve any problem that we see in reality.

As an example solving mnist and its variants of 28x28 pixels will be impossible until the 2100s because we don't have enough memory to store the general tensor which stores the interactions between group of pixels with every other group pixels.

joefourier• 1 year ago

While true in a theoretical sense (an MLP of sufficient size can theoretically represent any differentiable function), in practice it’s often the case that it’s impossible for a certain architecture to learn a specific task no matter how much compute you throw at it. E.g. an LSTM will never capture long range dependencies that a transformer could trivially learn, due to gradients vanishing after a certain sequence length.

ActorNightly2• 1 year ago

You are right with respect to ordering of operations, where recurrent networks have a whole bunch of other computational complexity to them.

However, for example, a Transformer can be represented with just deeply connected layers, albeit with a lot of zeros for weights.

g42gregory• 1 year ago

One of the key intuitions: If you take a natural image and add random noise, you will get a different random noise image every time you do this. However, all of these (different!) random noise images will be lined up in the direction perpendicular to the natural images manifold.

So you will always know where to go to restore the original image: shortest distance to the natural image manifold.

How all these random images end up perpendicular to the manifold? High dimensional statistics and the fact that the natural image manifold has much lower dimension than the overall space.

yubblegum• 1 year ago

TIL.

Generative Visual Manipulation on the Natural Image Manifold

https://arxiv.org/abs/1609.03552

For me, the most intriguing aspect of LLMs (and friends) are the embedding space and the geometry of the embedded manifolds. Curious if anyone has looked into comparative analysis of the geometry of the manifolds corresponding to distinct languages. Intuitively I see translations as a mapping from one language manifold to another, with expressions being paths on that manifold, which makes me wonder if there is a universal narrative language manifold that captures 'human expression semantics' in the same way as a "natural image manifold".

Ey7NFZ3P0nzAe• 1 year ago

I think this is related: https://news.ycombinator.com/item?id=44054425

fisian• 1 year ago

I found this course very helpful if you're interested in a bit of math (but all very well explained): https://diffusion.csail.mit.edu/

It is short, with good lecture notes and has hands on examples that are very approachable (with solutions available if you get stuck).

woolion• 1 year ago

Discussed on hn: https://news.ycombinator.com/item?id=43238893

I found it to be the best resource to understand the material. That's certainly a good reference to delve deeper into the intuitions given by OP (it's about 5 hours of lectures, plus exercises).

IncreasePosts• 1 year ago

Are there any diffusion models for text? I'd imagine they'd be very fast, if the whole result can be processed simultaneously, instead of outputting a linear series of tokens that each depend on the last

imbnwa• 1 year ago

Need a text diffusion model to output a version of Eden!Eden!Eden!

woadwarrior01• 1 year ago

Diffusion for text is a nascent field. There are a few pretrained models. Here's one[1], AFAIK it's currently the largest open weights text diffusion model.

[1] https://ml-gsai.github.io/LLaDA-demo/

intalentive2• 1 year ago

This explanation is intuitive: https://www.youtube.com/watch?v=zc5NTeJbk-k

My takeaway is that diffusion "samples all the tokens at once", incrementally, rather than getting locked in to a particular path, as in auto-regression, which can only look backward. The upside is global context, the downside is fixed-size output.

orbital-decay• 1 year ago

That a not a good intuition to have. That backwards-looking pathfinding process is actually pretty similar in both types of models - it just works along a different coordinate, crude-to-detailed instead of start-to-end.

intalentive2• 1 year ago

Good point.

petermcneeley• 1 year ago

This page is full of text. I am guessing the author (Sean Goedecke) is a language based thinker.

JoeDaDude• 1 year ago

Coincidentally, I was just watching this explanation earlier today:

How AI Image Generators Work (Stable Diffusion / Dall-E) - Computerphile

https://www.youtube.com/watch?v=1CIpzeNxIhU

bicepjai• 1 year ago

>>CLASSIFIER-FREE GUIDANCE … During inference, you run once with a caption and once without, and blend the predictions (magnifying the difference between those two vectors). That makes sure the model is paying a lot of attention to the caption.

Why is this sentence true ? “That makes sure the model is paying a lot of attention to the caption.”

noodletheworld• 1 year ago

Mmm… how is a model with a fixed size, let’s say, 512x512 (ie. 64x64 latent or whatever), able to output coherent images at a larger size, let’s say, 1024x1024?

Not in a “kind of like this” kind of way: PyTorch vector pipelines can’t take arbitrary sized inputs at runtime right?

If you input has shape [x, y, z] you cannot pass [2x, 2y, 2z] into it.

Not… “it works but not very well”; like, it cannot execute the pipeline if the input dimensions aren’t exactly what they were when training.

Right? Isn’t that how it works?

So, is the image chunked into fixed patches and fed through in parts? Or something else?

For example, (1) this toy implementation resizes the input image to match the expected input, and always emits an output of a specific fixed size.

Which is what you would expect; but also, points to tools like stable diffusion working in a way that is distinctly different to what the trivial explanation tend to say does?

[1] https://github.com/uygarkurt/UNet-PyTorch/blob/main/inferenc...

swyx• 1 year ago

That last point indicates an interesting capability that diffusion models have: you get a kind of built-in quality knob. If you want fast inference at the cost of quality, you can just run the model for less time and end up with more noise in the final output2. If you want high quality and you’re happy to take your time getting there, you can keep running the model until it’s finished removing noise.

not quite right... anyone who has run models for >100 steps knows that you can go too far. whts the explanation of that?

Diffusion models explained simply

Related Stories