Stable Diffusion PR optimizes VRAM, generate 576x1280 images with 6 GB VRAM

https://github.com/basujindal/stable-diffusion/pull/103

by avocado2 • 4 years ago

192 143 4 years ago

github.com

avocado2• 4 years ago

can run with gui: https://github.com/basujindal/stable-diffusion/blob/main/opt...

jacobn• 4 years ago

Interesting that “del” helps here - could eg torch.jit.script/trace be more of a “sledgehammer” solution?

ceeplusplus• 4 years ago

Unfortunately the author of this PR decided to include garbage like this [1] in it, so this PR is pretty useless.

[1] https://github.com/basujindal/stable-diffusion/pull/103/file...

kevingadd• 4 years ago

You're still able to run it locally even if the license change prevents it from being merged, and it seems to be a proof of concept that may inspire other people to optimize it in different ways

jinseokim• 4 years ago

I just reminded "clean room design" technique: https://en.m.wikipedia.org/wiki/Clean_room_design

javchz• 4 years ago

Yeah, reminds me of the old freedoom WADs situation

Matheus28• 4 years ago

Seems like author did his work on a different fork and pushed to this one, which included all changes from the other fork...

cercatrova2• 4 years ago

How is that garbage? That license is part of the original SD repository, the creator Emad even talks about it in his initial post, about the OpenRAIL M License [0]:

i) The model is being released under a Creative ML OpenRAIL-M license [https://huggingface.co/spaces/CompVis/stable-diffusion-licen...]. This is a permissive license that allows for commercial and non-commercial usage. This license is focused on ethical and legal use of the model as your responsibility and must accompany any distribution of the model. It must also be made available to end users of the model in any service on it.

[0] https://stability.ai/blog/stable-diffusion-public-release

zarzavat• 4 years ago

It’s worth to point out that there’s no consensus yet on whether ML models (as in the weights) are actually copyrightable. If you’re using them, you should probably assume that they are. If you’re distributing them, you should probably assume that they aren’t.

whywhywhywhy2• 4 years ago

they mean the entire thing has been run through some sort of "beautifier" so there are thousands of irrelevant whitespace changes across things unrelated to the commit, which itself is around 6 lines.

sercand• 4 years ago

This is the worst type of PR. Just submit a different PR for the code format changes.

dr_dshiv• 4 years ago

How much VRAM was previously required?

temp_account_322• 4 years ago

I am able to generate a maximum resolution of 512x768 on my 11GB 1080Ti. This seems to use almost 100% of the available RAM.

frognumber• 4 years ago

That's weird. I cap out at 512x512 on my 16GB Ampere card. Even stepping down precision doesn't help. I wonder what's different.

I use it directly from Python.

gpt52• 4 years ago

I can generate a 768x896 pixels image on an RTX 3090, using 23.4/24GB

moneycantbuy• 4 years ago

How does the content of the images look compared to the same prompt and seed for 512x512? I ask because when I make 768x896 with M1 MAX the resulting image is less coherent, more noisy, and or has repeating subject matter, nothing really usable unlike 512x512.

jameshe• 4 years ago

Looks like 10GB in the README[0] for a 512x512 image - though the fork[1] from the repo in this PR claims to be able to work with as little as 4GB VRAM.

[0] https://github.com/CompVis/stable-diffusion#stable-diffusion [1] https://github.com/basujindal/stable-diffusion#txt2img

jpeter2• 4 years ago

512x706 on rtx 2060s with 8gb vram

oefrha3• 4 years ago

Almost impossible to pinpoint what changed thanks to thousands of lines of completely irrelevant changes and shitty commit messages. It seems the only changeset that might be relevant out of +2,273 -1,531 is the +11 -7 from https://github.com/basujindal/stable-diffusion/pull/103/comm...? Does it even work?

piyh• 4 years ago

As a learning opportunity for people like me, what does a good PR look like for a large change?

adastra22• 4 years ago

You don’t make large change PRs.

MontagFTB• 4 years ago

Branching in git is very cheap, so the typical path a large change takes to the main branch is by a series of smaller, incremental PRs. This keeps your work close to the main branch as much as possible.

oefrha3• 4 years ago

I would say large contributions from non-members generally work rather terribly in open source. If you absolutely have to,

- Communicate ahead of time; don't surprise maintainers with sweeping architectural changes or huge features no one wants or would like to review;

- Try to break up changes into logical units that can be understood and reviewed independently;

- Write useful and detailed commit messages (some bad examples: "Update attention.py", "various clean-ups, code now beautified");

- Don't sneak in anything unrelated to the PR; don't sneak anything unrelated into a commit;

- Absolutely don't use a code formatter to format the entire code base if the repo wasn't already using one. You can suggest that separately. And changes like that are best done by a trusted member.

lbotos• 4 years ago

in this case a lot of the large change is actually "upstream".

We Have Neon, who is making a PR.

We have Upstream, SD source.

We have basu, repo maintainer.

Neon pulled in upstream changes, and are trying to merge those into basu's repo AND then on top of that, apply neon edits to the code.

Ways to make this cleaner:

- Basu pulls in upstream changes, Neon just puts theirs on top. (Slow, you have to wait for Basu)

- Neon makes two PRs, one to pull in upstream changes, another for his edits on top of those changes. (More work, and coordination, but each PR is "one unit of work")

- Better commit messages: https://github.com/basujindal/stable-diffusion/pull/103#comm... <- notice the repeated and non-descript commit messages? That makes it hard for non-experts to cherry pick out the bits that are really relevant. (Fastest)

imron• 4 years ago

Follow this: https://cbea.ms/git-commit/

and this: https://www.aleksandrhovhannisyan.com/blog/atomic-git-commit...

That way your "large change" becomes a collection of small, easy to reason about and well explained changes.

jsnell• 4 years ago

Does it even work?

It has an effect, but nothing like what's claimed in the submission title. On a 1070Ti (8GB), I managed to go up from 512576 to 576640.

ipaddr3• 4 years ago

Has anyone gotten to run on a 4gig card?

operator-name• 4 years ago

It looks like the first commit is the important change.

qdot763672• 4 years ago

For everyone about to comment on the garbage in the commit:

It looks like the committer made their changes in the top commit, then merged the updated CompViz StableDiffusion change set on top of it for some reason. That's where the license change, rick astley image, etc come from.

And yes, StableDiffusion from the original repo will rick roll you if you try to generate something that triggers its NSFW filter.

Here's the code that does it:

https://github.com/CompVis/stable-diffusion/blob/main/script...

And here's what it looks like:

https://twitter.com/qDot/status/1565076751465648128

NHQ2• 4 years ago

But what is the correct git command to ignore all that?

cercatrova2• 4 years ago

There's no git command to ignore it, the main repo should merge their changes and then the other person can make a PR on the updated changes.

oefrha3• 4 years ago

Do a git-cherry-pick.

p-e-w7• 4 years ago

And yes, StableDiffusion from the original repo will rick roll you if you try to generate something that triggers its NSFW filter.

It goes without saying that the authors of a piece of software have the right to make the software do whatever they want, but that shouldn't stop us from recognizing that AI engineers are starting to act like megalomaniac overseers who consider it part of their mission to steer humanity onto the "right" path.

Who exactly do these people think they are?

Imagine this behavior from a web browser. "The URL of the file you were trying to download triggered my NSFW classifier, so I'm going to replace the file with this funny image."

This isn't funny, it's creepy.

esjeon2• 4 years ago

Yeah, but, when it comes to generating NSFW contents, it is illegal to distribute pornographic software in some countries, and doing so can lead to domain blockage and such (i.e. South Korea, believe or not). This is something you gotta understand.

Also, if the author wanted to block NSFW contents at all, I'm pretty sure one can actually make the filter inseparable from the main network. This isn't the case here AFAIK.

0cVlTeIATBs• 4 years ago

Devil's advocate, this output could have unexpected results, making a filter possibly desirable. Browsers do filter by default with safe browsing URL lists for malware, and they gate access to sites with bad certificates. But agreed users should have control, leaving such filters as safe defaults only.

Arainach4• 4 years ago

What's creepy is the very real world that we're approaching where anyone can create porn of anyone else given a few pictures without their consent. That can cause all sorts of lasting psychological harm and other societal negative consequences. Even if this probably can't be entirely avoided, that doesn't mean that the folks developing these models are obligated to make it easy or are incorrect to take steps to avoid it.

qdot763672• 4 years ago

Nah it’s just basic due diligence for releasing an open source app of this nature. Turning it off is a simple one line change, because everything is obviously named. This is not some high wall to scale.

As someone that manages nsfw open source projects, this move seems fine to me.

And actually kinda hilarious.

ThrowawayTestr• 4 years ago

It's literally one line of code so you don't generate unexpected boobs. Save your ire for the nannies that won't let you generate NSFW images at all.

greesil2• 4 years ago

Yes, the megalomania of someone who makes their code and models freely available.

throwaway5959• 4 years ago

I’m sure you’re great at parties. A number of people have explained how easy this is to turn off (I did it myself in minutes without outside help by literally commenting out a line of code and changing a variable name). Getting offended at everything doesn’t solve anything and just makes half the US think we’re all snowflakes.

jannes• 4 years ago

Remember the Terminator movie? It’s about an AI that went too far.

yieldcrv• 4 years ago

Don’t worry, intentionally warped faces and NSFW filters have like a month, tops

Angostura2• 4 years ago

Who exactly do these people think they are?

Software engineers who took the time to think about the ways that their work could be used.

How many times have we seen on HN people pleading with developers to think about the ethical dimensions of their work?

vimy• 4 years ago

Read the tweets of the Stability AI, company behind Stable Diffusion, founder. He agrees with you. The image is purely to protect themselves against criticism, takedowns, possible legal action, stuff like that.

https://mobile.twitter.com/emostaque

aortega2• 4 years ago

StableDiffusion from the original repo will rick roll you if you try to generate something that triggers its NSFW filter.

Just replace in scripts/txt2img.py:

- x_checked_image, has_nsfw_concept = check_safety(x_samples_ddim)

+ x_checked_image = x_samples_ddim

And be done with it.

rrobukef• 4 years ago

And you save some more RAM too!

nl3• 4 years ago

I've been using the HuggingFace diffuses repo[1] with 6GB of VRAM fine.

It's well engineered, maintainable and with decent installation process.

The branch in this PR[2] adds M1 Mac support with a one line patch and it runs faster than the CompVis version (1.5 iterations/sec vs 1.4 for CompVis on a 32 Gb M1 Max,

I highly recommend people switching to that version for the improved flexibility.

[1] https://github.com/huggingface/diffusers

[2] https://github.com/huggingface/diffusers/pull/278

JimDabell4• 4 years ago

I wouldn’t recommend using that as-is. MPS doesn’t give deterministic random number generation, which means that seeds become meaningless and you won’t ever be able to reproduce something. You can work around it by generating random numbers on the CPU and then moving them to MPS, but that probably requires a fix in PyTorch.

The MPS support issue for diffusers is here:

https://github.com/huggingface/diffusers/issues/292

…and it links to the relevant PyTorch issue here:

https://github.com/pytorch/pytorch/issues/84288

nl3• 4 years ago

That's really annoying, even though I hadn't noticed it until now. I can confirm this is an issue.

But the situation seems to be the same on the CompVis derived repos, right?

So this is no worse off, but with better engineered and faster code.

whywhywhywhy2• 4 years ago

https://github.com/magnusviri/stable-diffusion/commit/d0b168...

Copying this change fixed seeds on M1 for me.

andrethegiant• 4 years ago

Wild how fast this is moving

bongobingo1• 4 years ago

Just wait until the AI is writing the code too. You wont be able to `git pull` fast enough.

jpeter2• 4 years ago

The power of open source

switchers• 4 years ago

Slightly off tangent, has anyone got it running on an AMD card? Sad 6600xt Windows user reporting in :(

universa1• 4 years ago

Not sure about the 6600, but there is a guide for Linux at least:

https://m.youtube.com/watch?v=d_CgaHyA_n4&feature=emb_logo

And this is somehow relevant (possibly), as I kept the link open.

https://github.com/RadeonOpenCompute/ROCm-docker/issues/38

mpaepper• 4 years ago

Here is a guide for AMD, but I don't have such a card, so haven't tried.

https://rentry.org/sdamd

saurik• 4 years ago

I was under the impression that generating images where both dimensions were larger than 512 didn't merely require a lot of resources but didn't work well as the model was trained exclusively on 512 by 512 images and while it sort of worked ok to stretch one dimension a bit you got weird repetitions by making the overall canvas too large (as this isn't going to generate higher dpi images, merely ones with greater square inches).

prettydeep• 4 years ago

This is accurate in my experience. Changing the res. to anything but 512x512 produces inferior results.

TaylorAlexander2• 4 years ago

In case anyone is confused by the clashing repos, here is how I was able to easily run this updated code.

Clone the original SD repo, which is what this code was built off of, and follow all the installation instructions:

https://github.com/CompVis/stable-diffusion

In that repo, replace the file ldm/modules/attention.py with this file:

https://raw.githubusercontent.com/neonsecret/stable-diffusio...

Now run a new prompt with a larger image. Note that the original model was trained on 512x512 and may lead to repetition especially if you try to increase both dimensions (this is mentioned in the SD readme) so just run with one dimension increased.

For example try the following example:

python scripts/txt2img.py --prompt "a person gardening, by claude monet" --ddim_steps 50 --seed 12000 --scale 9 --n_iter=1 --n_samples=1 --H=512 --W=1024 --skip_grid

I confirmed that if I run that command with the original attention.py, it fails due to lack of memory. With the new attention.py, it succeeds.

That said, this still uses 13GB of ram on my system.

I suppose you can check out the full repo with the updated code, which seems to have other changes, if you want to give that a try.

https://github.com/neonsecret/stable-diffusion/

I have already been using the original SD repo so I found benefit by just changing attention.py

macrolime2• 4 years ago

I've seen people claim that it's better to generate 512x512 images since that's what it's trained on, and then upscale, rather than generating higher resolution images directly. Anyone tried any systematic investigation of this?

lionkor• 4 years ago

This is one of the worst PRs i've ever seen.

If I was involved, this PR would be closed and conversation locked, simply because its a ton of formatting changes. The only "memory efficiency" changes are like 6 lines of python where the author uses `del`. Thats it. Everything else is formatting bullshit, and some other merge stuff. Yikes.

Also, some of those PR comments are written by kids or something. Wtf

MayeulC• 4 years ago

I wonder if GTT (GART size) wouldn't help with limitted VRAM amounts, at the cost of some performance?

WithinReason• 4 years ago

The diff that apparently does the RAM optimisation, quite simple and something to learn form:

https://github.com/basujindal/stable-diffusion/commit/47f878...

liuliu• 4 years ago

Yeah, stable diffusion's PyTorch code is not optimized for inference memory usage at start. I am looking at the code now and it seems that if it is converted to static graph, there are probably a bit more opportunities (I only looked at CLIP model and UNet model it uses today, not sure about the Autoencoder yet).

LarsDu88• 4 years ago

Started using this branch and it works on my old ass 1060 gpu. Maybe I'll do a blog post on the whole process

Stable Diffusion PR optimizes VRAM, generate 576x1280 images with 6 GB VRAM

Related Stories