A Web UI for Stable Diffusion

https://github.com/AUTOMATIC1111/stable-diffusion-webui

by feross • 4 years ago

284 143 4 years ago

This is the one I've been using https://github.com/sd-webui/stable-diffusion-webui . docker-compose up , works great.

wyldfire2• 4 years ago

I like this one but had some trouble with using img2img. Maybe my image was too small (it was smaller than 512x512). Failed with the same signature as an issue that was closed with a fix.

ghilston• 4 years ago

I am mobile, but there's an issue reported on github about img2img

jyrkesh• 4 years ago

I've also been using this one (wasn't sure at first, they just migrated from the /hlky/ namespace on GitHub), but I have no idea at first glance what the differences are.

I will say that this one has had REALLY active development as new features have been coming out, and is pretty polished at this point (albeit I'm using it more as a toy than anything, but it's awesome to have a quick way to use the new features that have been shipping out).

TekMol4• 4 years ago

Is there a way to run this in the cloud? On Google Colab or elsewhere?

fragmede10• 4 years ago

Colab: https://colab.research.google.com/github/WASasquatch/StableD...

To run it elsewhere in the cloud, grab a GPU (spot) instance and SSH in.

KronisLV• 4 years ago

Seems like it fails if you have an AMD GPU instead of an Nvidia one (at least that's my guess, based on the error contents):

  ERROR: for stable-diffusion  Cannot start service stable-diffusion: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown

mach1ne• 4 years ago

Indeed, Stable Diffusion does not currently run on AMD graphics cards.

sharps_xp• 4 years ago

if someone can dockerize this, please reply with a link!

stephanheijl• 4 years ago

This is the dockerized version of this repo: https://github.com/AbdBarho/stable-diffusion-webui-docker

geerlingguy• 4 years ago

The main one I've been testing (https://github.com/sd-webui/stable-diffusion-webui) has a Docker Compose file.

Grab that, install Docker, install Nvidia's Docker integration, copy the example Docker-env file, and docker-compose up is all you need.

Edit: here's a gist with exact steps I used: https://gist.github.com/geerlingguy/384ed4aba35e3118f2a0f358...

user-one1• 4 years ago

It can be run directly into google colab: https://colab.research.google.com/drive/1Iy-xW9t1-OQWhb0hNxu...

TekMol4• 4 years ago

When I run it, I get "Your session crashed for an unknown reason."

fragmede10• 4 years ago

Just click runtime -> run all again. There’s a weirdness where Python’s loader gets confused and the most effective fix is to crash the interpreter

smrtinsert• 4 years ago

I have a 6gb 1660ti, barely holding on. Is a new 12gb card good enough for now, or should I go even higher to be safe for a few years of sd innovation?

wyldfire2• 4 years ago

It sounds like there's forks that are able to work with <=8GB cards. And I'm not sure but I think the weights are using f32, so switching to half might make it yet easier still to get this to work w/less memory.

But yeah the next generation of models would probably capitalize on more memory somehow.

filiphorvat• 4 years ago

People have reported that this repo even works with 2gb cards if you run it with --lowvram and --opt-split-attention.

fassssst• 4 years ago

The GeForce 4000 series is about to release and should make Stable Diffusion wayyyyy faster based on related H100 benchmarks posted today.

drexlspivey• 4 years ago

How is M1/M2 support for SD? Is there a significant performance drop? Presumably you would be able to buy a 32GB M2 and be future proof because of the shared memory between CPU/GPU.

jjcon• 4 years ago

In my setup at least it runs essentially in CPU mode since there is no CUDA acceleration available and metal support is really messy right now. So while quite slow I don't run into memory issues at least. It runs much faster on my desktop GPU but that has more constraints (until I upgrade my personal 1080 to a 3090 one of these days).

totoglazer2• 4 years ago

There was a long thread last week. It’s honestly pretty good if you follow the instructions. 30-40 seconds/image.

jwitthuhn• 4 years ago

I recently switched from a CPU-only version to this repo release 1.13: https://github.com/lstein/stable-diffusion

The original txt2img and img2img scripts are a bit wonky and not all of the samplers work, but as long as you stick to dream.py and use a working sampler, I have had good luck with k_lms, then it works great and runs way faster than the cpu version.

Works great on 32gb ram but I'm honestly tempted to sell this one and get a 64gb model once the m2 pros come around. This is capable of eating up all the ram you can throw at it to do multiple pictures simultaneously.

CitrusFruits2• 4 years ago

I'm using it with a 2070 (4 year old card with 8gb vram) and it takes about 5 seconds for a 512x512 image. It's been plenty fast to have some fun, but I think I'd want faster if it was part of a professional work flow.

totoglazer2• 4 years ago

What settings? That seems faster than expected.

poisonarena• 4 years ago

im on my 2020 macbook air m1 ... 512px image takes 2-3 minutes :(

hwers• 4 years ago

People recently figured out how to export stable diffusion to onnx so it’ll be exciting to see some actual web UIs for it soon (via quantized models and tfjs/onnxruntime for web)

password43213• 4 years ago

Very cool! Can you link to where this is taking place?

A commenter mentioned today it might be possible to pre-download the model and load it into the browser from the local filesystem rather than include such a gigantic blob as an accompanying dependency, fighting different caching RFC's, security/usage restrictions, and anything else that might inadvertently trigger a re-download.

https://news.ycombinator.com/item?id=32777909#32779093

minimaxir3• 4 years ago

Support for ONNX export was just added to diffusers, but no runtime logic for scheduling yet.

https://github.com/huggingface/diffusers

ducktective2• 4 years ago

It seems Midjourney generates better results than SD or Dall-E.

What's with the "hyper resolution", "4K, detailed" adjectives which are thrown left and right, while we are at it?

Ckalegi• 4 years ago

The metadata and file names of the images in the source data set are also inputs for the model training. These keywords are common tags across images that have these characteristics, so in the same way it knows what a unicorn looks like, it also knows what a 4k unicorn looks like compared to a hyper rez unicorn.

schleck8• 4 years ago

Those are prompt engineering keywords. SD is way more reliant on tinkering with the prompt than midjourney

https://moritz.pm/posts/parameters

throwaway1851• 4 years ago

MidJourney needs a lot of prompt engineering too. And Dall-E also. If you look at the prompt as an opportunity to describe what you want to see, the results are often disappointing. It works better to think backwards about how the model was trained, and what sorts of web caption words it likely saw in training examples that used the sorts of features you’re hoping it will generate. This is more of a process of learning to ask the model to produce things it’s able to produce, using its special image language.

cma• 4 years ago

Midjourney uses SD under the hood (you can see in their license), but they augnment the model in various ways.

qudat• 4 years ago

The results in midjourney are significantly better than SD. I find it much easier to get to a good result in MJ and I've been trying to understand why. Anymore insight you could share?

jrm43• 4 years ago

So, and this is an ELI5 kind of question I suppose. There must be something going on like "processing a kazillion images" and I'm trying to wrap my head around how (or what part of) that work is "offloaded" to your home computer/graphics card? I just can't seem to make sense of how you can do it at home if you're not somehow in direct contact with "all the data?" e.g. must you be connected to the internet, or "stable-diffusions servers" for this to work?

codefined• 4 years ago

All those 'kazillion' images are processed into a single 'model'. Similar to how our brain cannot remember 100% of all our experiences, this model will not store precise copies of all images it is trained off of. However, it will understand concepts, such as what a unicorn looks like.

For StableDiffusion, the current model is ~4GB, which is downloaded the first time you run the model. These 4GB encode all the information that the model requires to derive your images.

ducktective2• 4 years ago

As someone with ~0 knowledge in this field, I think this has to do with a concept called "transfer learning" in which you once train with that kazillion of images, then use that same "coefficients" for further run of the NN.

palunon• 4 years ago

Nah, transfer learning is when you take a trained model, and train it a little more to better fit your (potentially very different) problem domain. Such as training a cat/dog/etc recognition model on MRI scans.

The goal is usually to have the more fundamental parts of your model already working and you thus need way less domain specific data.

Here, you're not training anything, you're running the models (both the CLIP language model and the unet) in feedforward. That's just deploying your model, not transfer learning.

sC3DX• 4 years ago

What you interact with as the user is the model and its weights.

The model (presumably some kind of convolutional neural network) has many layers, every layer has some set of nodes, and every node has a weight, which is just some coefficient. The weights are 'learned' during the model training where the model takes in the data you mention and evaluates the output. This typically happens on a super beefy computer and can take a long time for a model like this. As images are evaluated the output gets better the weights get adjusted accordingly.

Now we as the user just need the model and the weights!

dwohnitmok• 4 years ago

This is the main reason why attempts to say that these sorts of AI are just glorified lookup tables, or even that they are simply tools that mash together a kazillion images together are very misleading.

A kazillion images are used in training, but training consists of using those images to tune on the order of ~5 GB of weights and that is the entire size of the final model. Those images are never stored anywhere else and are discarded immediately after being used to tune the model. Those 5 GB generate all the images we see.

juliendorra2• 4 years ago

That’s the interesting part: all the images generated are derived from a less than 4gb model (the trained weights of the neural network).

So in a way, hundreds of billions of possible images are all stored in the model (each a vector in multidimensional latent space) and turned into pixels on demand (drived by the language model that knows how to turn words into a vector in this space)

As it’s deterministic (given the exact same request parameters, random seed included, you get the exact same image) it’s a form of compression (or at least encoding decoding) too: I could send you the parameters for 1 million images that you would be able to recreate on your side, just as a relatively small text file.

amelius3• 4 years ago

As it’s deterministic (given the exact same request parameters, random seed included, you get the exact same image) it’s a form of compression (or at least encoding decoding) too: I could send you the parameters for 1 million images that you would be able to recreate on your side, just as a relatively small text file.

For any input image? Or do you mean an image generated by the model?

nope96• 4 years ago

So it's like a compiler which produces a 4GB executable file? And that 4GB is all the "logic" which can produce infinite possible images?

l33tman• 4 years ago

SD has 860M weights for the main workhorse part. At 16-bit precision that is only 1.6 GB of data, which in some very real sense has condensed the world's total knowledge of art and photography and styles and objects.

It's not a search engine, it's self-contained and the closest analogy is that it's a very very knowledgable and skilled artist.

password43213• 4 years ago

Is there a smaller version of the model available (<4gb) intended for use with 16 bit precision?

bootsmann• 4 years ago

You can think of it more like this: If I do 100 experiments of dropping stones at variable heights and measuring the time it takes for the stone to land on the ground I have enough datapoints to make a linear estimation of gravity by using linear regression. So based on my data I create a model that the time it takes for a stone to fall is sqrt(2h/9.81). Now if you want to figure out how long it takes for your stones to fall, you don’t need to redo all the experiments and can instead rely on the parameters I give you (say 9.81 in this case) to calculate it yourself.

With these models it works exactly the same way. Someone dropped millions of rocks and created a formula of unbelievable complexity and what they now did is they released that formula with all their calculated parameters into the world. What you do when you ultimately use Stable Diffusion is you just calculate the result of this formula and that is your image. You never have to process those images.

MattRix5• 4 years ago

This is exactly it. It’s pretty remarkable that it was trained on over 100 terabytes of images and yet the model has been distilled down to only 4gb.

culi• 4 years ago

A kazillion images are used to create/optimize a neural network (basically). What you're working with is the result of that training. These are the "weights"

mirekrusin• 4 years ago

It’s all offline in 4gb file on your local computer. It’s like mini brain trained to do just one/few specific tasks. Just like your own brain doesn’t need Wi-Fi to connect to global memory storage of everything you experienced since birth, same way this 4gb file doesn’t need anything extra.

amelius3• 4 years ago

Regarding the opening image: if it can't correctly put the marks on dice, how can it put eyes, nose and mouth correctly on a human face?

bastawhiz• 4 years ago

Presumably the number of faces in the training set far exceeds the number of dice by more than a few orders of magnitude.

amelius3• 4 years ago

In one of the other posts I noticed this option:

GFPGAN Face Correction: Automatically correct distorted faces with a built-in GFPGAN option, fixes them in less than half a second

So apparently there is still an issue with faces.

eminence32• 4 years ago

In my experience, Stable Diffusion is also pretty bad about human faces and hands too

ShamelessC• 4 years ago

Regarding the opening image: if it can't correctly put the marks on dice, how can it put eyes, nose and mouth correctly on a human face?

It helps if you consider it all as effectively advanced compression. Everything the model can do is limited by its architecture, the number of parameters in the model, and the accuracy and size of the training data.

The underlying architecture is a transformer (e.g. GPT3) wired to a (denoising) diffusion model.

Current flaws with this approach:

- Transformers seem to approach a "bag-of-words" model, often ignoring the ordering of the words. Among other things, this means that text-to-image models are very bad at "binding attributes" [0]. This is why "a boy wearing a red shirt and a girl wearing a black jacket" may fail (putting the colors on the wrong items, for instance).

- Autoregressive transformers have no means to correct early mistakes.

- Training data is captioned images and the captions are likely noisy and under-specified. Every time it sees a face labeled as "face" - it tries to generate a face from the distribution of _all_ faces in the data. The same goes for the dice. If the dice are just labeled "dice", but don't have a description of how they landed - the model has to guess which angle you're referring to. As a sibling comment points out, this is exacerbated by the relative frequencies of examples of the data in the dataset.

[0] https://wikiless.org/wiki/Binding_(linguistics)

zenlikethat• 4 years ago

It can’t. :)

Well, I kid a bit. I’ve seen it produce some amazing results, but, generally, it has a hard time with that. Often faces end up looking blurry or having these creepy, dead white eyes. Hands likewise often end up malformed (seven fingers anyone?) and twisty. But, it seems to have a much easier time generating passable faces in close ups with the right key words. Especially if you give it an input image that already has a clear one. It also seems to have an easier time doing faces it already knows like a celebrity, presumably because it’s using a strong existing influence instead of inventing/hallucinating it.

Supposedly this is improved in their new 1.5 version which is in beta. The software is so compelling that I suspect this will be improved quite quickly. Also, I think either way workarounds will emerge, either by composing with other networks/software (some UIs have GANs for face correction) or the old fashioned way by photoshopping over the blemishes.

MattRix5• 4 years ago

It’s worth noting you also get MUCH better faces and hands if you’re willing to run it for more steps (100-150). It takes a lot longer to run it at higher step counts so a lot of people don’t do it.

fibonacc2• 4 years ago

a sigh of relief as I thought that this would generate instead of pictures, React UI code based on plain text description.

First they came for illustrators, then they came for UI designers.

MattRix5• 4 years ago

heh yeah it won’t be that long until we have A Stable Diffusion for Web UI

fragmede10• 4 years ago

You've seen https://twitter.com/sharifshameem/status/1282676454690451457 yeah?

fibonacc2• 4 years ago

yes what I mean was mixing stable diffusion into the UI styling without the fine atomic details. Like say I want a dashboard with color themes from cyberpunk complete with graphics/art/logo. It would be a "good enough" category if something can literally be told to produce a complete frontend.

offsky• 4 years ago

I’m waiting for someone to wrap this up into a desktop app that I can install and run on my Mac.

geuis2• 4 years ago

I've been looking into this for the last 2 days. Unless you're running an M1 Mac or newer, you're SOL.

Stable Diffusion is built on PyTorch. PyTorch mainly has been designed to work with Nvidia cards. However PyTorch added support for something called RocM like a year ago that adds compatibility with newer AMD cards.

Unfortunately RocM doesn't support slightly older AMD cards in conjunction with intel processors.

So my 32gb pretty powerful 2020 16in MacBook Pro isn't capable of running Stable Diffusion.

Any native app will likely have to rely on a remote cloud gpu. And boy, those are fucking expensive. Been researching what I need to stand up a service the last few days and it isn't cost friendly.

minimaxir3• 4 years ago

There is work on a CoreML version which may play nicer with older Macs w/sufficiently beefy dGPUs.

https://github.com/huggingface/diffusers/issues/443

fragmede10• 4 years ago

Apple's MPS drivers supports AMD GPUs on MacOS

danuker2• 4 years ago

And boy, those are fucking expensive.

Unless you want to train the model, Lambda Labs is somewhat cheap:

https://lambdalabs.com/service/gpu-cloud

dreamcompiler• 4 years ago

I've been running the Intel CPU version [0] for a while now on a 2013 MacMini. Works fine; it takes several minutes per image but I can live with that.

[0] https://news.ycombinator.com/item?id=32642255

kcartlidge3• 4 years ago

I've been looking into this for the last 2 days. Unless you're running an M1 Mac or newer, you're SOL.

And not just any old M1 Mac. Last week I got it running on my 2021 8GB M1 MacBook Air and it's slow. Images at 512x512 with 10 steps take between 7 and 10 minutes to generate.

It's the only thing I do that hits performance limitations on the 8GB machine so there's no regrets on that score, but with the way this stuff is progressing 16GB+ is a realistic minimum for comfortable use.

westoncb• 4 years ago

I’ve been working on a queue-centric desktop app GUI for SD: https://twitter.com/westoncb/status/1568114946235580418?s=46...

I plan to wrap things up and put out the source this weekend.

fragmede10• 4 years ago

https://www.charl-e.com/

There are a few bugs to iron out before it's ready for prime time. For now, create the folder `~/Desktop/charl-e/samples/` manually before you run it.

tasuki• 4 years ago

Funny, it probably does a whole lotta things, but it can't create the `~/Desktop/charl-e/samples/` directory? That seems like it should be relatively trivial...

kcartlidge3• 4 years ago

Great stuff, and works on an 8GB M1 Air taking between 5 and 10 minutes for between 5 and 15 steps. As a suggestion, perhaps add the option for setting the CFG too (I know, it's open source etc, but it's just a suggestion).

kierangill• 4 years ago

Same! I wrote a public web app so that I could access the model from my phone [0]. This is how I found Replicate [1]. Their SD model is very cheap to use. While we wait for a native Mac app, I recommend accessing the model straight from their web UI.

[0] https://www.drawanything.app/

[1] https://replicate.com/

cloudking2• 4 years ago

How does copyright work with output images? If someone runs the model on their own hardware, do they "own" the images generated? If 2 people generate the same image using the same prompt/seed, who "owns" the image?

skipnup• 4 years ago

Is it any different than Photoshop content aware fill? Or using a camera?

Nobody would ever think about Adobe or Nikon having copyright claims over your pictures. For me it's just a tool, the artistic part is providing a good description/base image, refining and choosing the best output.

Anyway, I'm not a lawyer and we probably live in different countries, so it'll be interesting to wait for the first lawsuit.

peoplefromibiza• 4 years ago

Nobody would ever think about Adobe or Nikon having copyright claims over your pictures

But it is illegal to share pictures of Eiffel Tower, for example.

People do it, but they shouldn't.

If I put a picture of Eiffel Tower at night in a book or any other kind of commercial product, I have to pay to use it. Doesn't matter that it's there for my eyes to see it.

The question is: are the images generated by an hyper accelerated learning machine using copyrighted material without the author's consent legal?

I think they shouldn't be and the data included in the training should be free or licensed.

cthalupa7• 4 years ago

https://www.smithsonianmag.com/smart-news/us-copyright-offic...

In the US, AI generated art cannot be copyrighted.

Edit:

Some additional details.

The US also denied a copyright for one where the creator listed themselves, with the AI just being a co-creator.

https://www.reddit.com/r/COPYRIGHT/comments/vshypc/the_us_co... (Original article is paywalled, reddit post contains the relevant bits)

In particular: >“Even though you argue that there is some human creative input present in the work that is distinct from RAGHAV’s contribution, this human authorship cannot be distinguished or separated from the final work produced by the computer program,” the office stated.

The US does seem to be a bit of an outlier here. The above work was granted copyright in Canada and India.

In the EU, AI generated artwork is likely copyrightable: https://link.springer.com/article/10.1007/s40319-021-01115-0

The same for the UK: https://www.kilburnstrode.com/knowledge/ai/ai-musings/respon...

Edit2: I'm not a lawyer, this isn't legal advice, go contact one if you actually need legal advice here.

cal85• 4 years ago

That’s one case, not a ruling about all AI generated art. It won’t be the same for every image involving AI in some way. What if you use AI to fill in a portion of an image, as with Adobe’s content aware fill? What if you use a series of SD steps but with a human selecting outputs and feeding them back in as inputs to get something else the AI could not have come up with on its own? The copyright conversation is only just beginning.

holoduke4• 4 years ago

What if I create a fully automated image site where people can purchase images. All the images are generated by SD based on keywords I scrape from competitor websites. Would be quite easy to create a website like that.

notfed• 4 years ago

Looks great but I use Linux and the README is fairly Windows-centric without warning. It'd be nice if there were clearly deliniated sections for Windows vs *nix.

There's a (very ironically named) "Manual installation" section which might seem to be the answer for Linux, but then it's not immediately obvious which preceding sections are Linux or Windows without doing critical thinking.

cthalupa7• 4 years ago

I've been having a lot of fun with Stable Diffusion and Midjourney.

One thing that is very powerful with Stable Diffusion is using text inversion ( https://textual-inversion.github.io/ ) - you can add additional input samples to further extend the possibilities beyond what is included in the original model.

adammarples2• 4 years ago

Can I run (train?) this textual inversion using the same consumer GPUs that work with stable-diffusion? Or does it require a much beefier machine

cthalupa7• 4 years ago

You can, though you might run into memory limitations running it on a GPU. There can be tuning done to lower the VRAM utilization, but I have been lucky enough to not need this - I do some CG work and ran into VRAM limitations there, so I'm on a 3090 with 24GB.

You can always run it on a CPU and utilize your RAM instead if needed, though the training might extend to 24+ hours that way.

Edit: Here's an example of someone successfully using textual inversion - https://www.reddit.com/r/StableDiffusion/comments/wz88lg/i_g...

Metricon• 4 years ago

Another free offline and easy to use model with a GUI can be downloaded from here: https://grisk.itch.io/stable-diffusion-gui

For some genuinely incredible results try this pattern for instruction:

Portrait of {Name of some type of identity such as "Faerie Princess" or "Dragon Queen"} {Name of a celebrity such as "Scarlett Johansson"}, beautiful face, symmetrical face, tone mapped, intricate, elegant, highly detailed, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and Greg Rutkowski and Alphonse Mucha and Boris Vallejo and Johannes Voss and Aleksi Briclot and Michael Komarck.

Run several iterations of the same query as some results will have anomalies.

fomine3• 4 years ago

nit: seekbar is terrible UI to select exact large value, like resolution. I don't know why is it accepted.

underlines2• 4 years ago

Nice to know tricks for /sd-webui/

- activate advanced: create prompt matrix and use

- add different relative weights for words in a prompt:

watercolor :0.5 painting :0.2 by picasso :0.3

- Generate much larger images with your limited vram by using optimized versions of attention.py and model.py

https://github.com/sd-webui/stable-diffusion-webui/discussio...

- Generate "Loab the AI haunting woman" if you can (Try using textual inversion with negatively weighted prompts)

https://www.cnet.com/science/what-is-loab-the-haunting-ai-ar...