Diffusion with Offset Noise: Finetuning SD to generate very dark or light images

https://www.crosslabs.org//blog/diffusion-with-offset-noise

by siraben • 3 years ago

133 29 3 years ago

This is really interesting. Another thing I noticed in my fun with SD is that it is extremely stubborn about colors during the denoising process.

That is, whatever color a region of the image has during denoising step 3, it will almost surely have that color at step 50, even if it makes no logical sense for the thing in that location to have that color.

This may not seem bad, but it's annoying when doing anything image-to-image, because regardless of the prompt you give it, the colors are "sticky".

If you have an image of an apple, and you use image-to-image with the prompt "an image of an orange", you will get a very reddish orange (in my experience at least).

sp332• 3 years ago

Does it work to make a greyscale image and let the denoiser find a color by descent?

danielvf• 3 years ago

From experience in related things - you are going to get a fairly grey orange, and fairly grey overall image.

Vecr• 3 years ago

I've heard putting a fuzzy multicolored noise (perlin in RGB space or something) over your IMG2IMG input. Not sure what scale or what opacity you need, but that's something you can try.

CuriouslyC• 3 years ago

You can use this to your advantage too though. I've manually generated "noise" for img2img by taking a base image, hitting it with a bunch of filters, then putting it back into img2img with a low-ish denoising strength. This method works quite well for ensuring composition while still letting the image be styled, but it's probably obsolete for that purpose now that we have controlnet.

markisus• 3 years ago

This could possibly be fixed by generating training image sequences which, along with random noise pixes, also have random hue shift for subregions of the image.

tangjurine• 3 years ago

An easy extension would be to have random patches of the image be offset by a random color, of varying sizes.

Pretty cool!

Jack000• 3 years ago

I'm curious if this generalizes to mid frequencies (ie. add some blurred noise in addition to the offset) and what effect that might have on the generations.

ChrisFoster• 3 years ago

Exactly my thought too. The offset is just the zero frequency. But in general, the need to do this for the zero frequency would suggest that there's a scaling problem for all the longer frequency Fourier components? And that, perhaps, the effective spatial Fourier spectrum of noise which is used in Sd is not optimal?

sebastiennight• 3 years ago

Hi Jack, I've been fascinated with your work on Colormind and Fontjoy and tried to reach out to you on your @colormind email to ask about your API. Is there a better way to reach you?

sschueller• 3 years ago

The whole thing with SD is extremely interesting and difficult to keep up with as there are so many new things coming in almost daily.

Just last week I saw ControlNet[1] which ads a lot more control.

Today I saw what corridor crew[2] did to stabilize the randomness when you want to make videos. Very exciting.

[1] https://github.com/lllyasviel/ControlNet

[2] https://www.youtube.com/watch?v=_9LX9HSQkWo

cwkoss• 3 years ago

The A1111 webUI and extension ecosystem is a beautiful example of what FOSS can be.

- User can install with a few clicks, no thinking about command line

- webUI has hover tooltips on everything, so user can most figure out what's going on without ever needing to touch documentation

- A1111 has a tab which can load a list of extensions from a github page

- Click to install, then refresh UI and extension just works

- Users are getting new incredibly powerful extensions every week or two - deforum lets you sequentially generate as many frames as you want and stitch them into a video, controlnet lets you copypaste features from a source image to your target image(s). Controlnet was added to A1111 ~2 weeks ago and is already integrated into the deforum tab so you can use both together.

Truly beautiful. I'd love to see more FOSS projects that felt so user friendly, generous with features, and rapid. Really fun to play with new cutting edge tech every couple weeks.

lelandfe• 3 years ago

a beautiful example of what FOSS can be

I was going to respond by saying it's not actually FOSS, but the author added an AGPL license last month: https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob...

(I suspect they did not ask contributors to relicense, but hey)

p1esk• 3 years ago

In the corridow crew video you posted - the most interesting part is the beginning (interesting to ML practitioners).

braingenious2• 3 years ago

There is already a publicly-available model that uses this!

https://civitai.com/models/11193/illuminati-diffusion-v11

SV_BubbleTime3• 3 years ago

As a ckpt that's cool, but I'd like to see it as a LORA so you can use any checkpoint you already have. That would (let's face it... will next week) be amazing.

jupiterelastica• 3 years ago

What is LORA in this context? Google only brought me to wifi networks :D

BudaDude• 3 years ago

Turns it out it came faster than that.

https://civitai.com/models/8765/theovercomer8s-contrast-fix-...

Lerc• 3 years ago

Is the noise function used for just the starting data provided to the first iteration of denoising, or does it get called repeatedly throughout the iterations?

ec109685• 3 years ago

Diffusion is so interesting. Unlike LLM that have some parallels to how the human mind works, it's not as obvious that reverse engineering from noise to a prompt has any similar parallels.

Will this cause us to hit walls at some point or actually exceed what a human can create?

albertzeyer• 3 years ago

I would actually say the opposite.

We know that the biological brain does a lot of iterative refinement via recurrent processing (attractor dynamics), which is very similar to how diffusion works.

However, while prediction is also a core functionality of the brain, it's not really that you always auto-regressively generate word-by-word.

vlovich123• 3 years ago

Do we know the process through which dreams and memory visualization works?

nyanpasu64• 3 years ago

I don't fully understand SD, but what would happen if you used brown or pink noise rather than a DC component added to white noise?

Abecid• 3 years ago

This is one of the coolest things I've read about diffusion

mgraczyk• 3 years ago

What if you do this with no fine tuning?

indeedmug• 3 years ago

Does anyone know how are they taking wavelengths from an image or what exactly "long-wavelength features" means?

I googled "wavelength of images" it doesn't seem like I am going in the right direction because it's about finding the wavelength of light from images rather than "wavelength of features" that this blog is talking about.

Diffusion with Offset Noise: Finetuning SD to generate very dark or light images

Related Stories