Run Stable Diffusion on Your M1 Mac’s GPU

https://replicate.com/blog/run-stable-diffusion-on-m1-mac

by bfirsh • 4 years ago

1007 401 4 years ago

That was fast.

Is there a way to get it to run on an an Intel-based Mac? I've attempted several times, but quickly ran into dependency issues and other quirks.

jw12248• 4 years ago

I believe the branch which adds support for Apple Silicon also adds support for running on Intel chips (albeit extremely slowly). I haven't tested it myself, but I've seen several people in the GitHub issues saying this.

pja4• 4 years ago

The standard release works fine, if you tweak the code to use the CPU pytorch device instead of CUDA. It does take about an hour to generate a set of images with the standard options on my AMD 2600 CPU though!

vimy• 4 years ago

Comment from github: "By the way, i confirmed to work on my Intel 16-in MacBook Pro via mps. GPU (Radeon Pro 5500M 8GB) usage is 70-80% and It takes 3 min where --n_samples 1 --n_iter 1. My repo https://github.com/cruller0704/stable-diffusion-intel-mac"

For comparison, my RTX 2070 takes 10 seconds for one image (512x512)

amelius• 4 years ago

I'd rather see someone implemented glue that allows you to run arbitrary (deep learning) code on any platform.

I mean, are we going to see X on M1 Mac, for any X now in the future?

Also, weren't torch and tensorflow supposed to be this glue?

nathas7• 4 years ago

Broadly speaking, it looks like they are. The implementation of Stable Diffusion doesn't appear to be using all of those features correctly (i.e. device selection fails if you don't have CUDA enabled even though MPS (https://pytorch.org/docs/stable/notes/mps.html) is supported by PyTorch.

Similar goes for quirks of Tensorflow that weren't taken advantage of. That's largely the work that is on-going in the OSX and M1 forks.

davedx2• 4 years ago

I got stuck on this roadblock, couldn’t get CUDA to work on my Mac, was very confusing

scoopertrooper2• 4 years ago

If you look at the substance of the changes being made to support Apple Silicon, they're essentially detecting an M* mac and switching to PyTorch's Metal backend.

So, yeah PyTorch is correctly serving as a 'glue'.

https://github.com/CompVis/stable-diffusion/commit/0763d366e...

bfirsh11• 4 years ago

As mentioned in sibling comments, Torch is indeed the glue in this implementation. Other glues are TVM[0] and ONNX[1]

These just cover the neural net though, and there is lots of surrounding code and pre-/post-processing that isn't covered by these systems.

For models on Replicate, we use Docker, packaged with Cog for this stuff.[2] Unfortunately Docker doesn't run natively on Mac, so if we want to use the Mac's GPU, we can't use Docker.

I wish there was a good container system for Mac. Even better if it were something that spanned both Mac and Linux. (Not as far-fetched as it seems... I used to work at Docker and spent a bit of time looking into this...)

[0] https://tvm.apache.org/ [1] https://onnx.ai/ [2] https://github.com/replicate/cog

keepquestioning3• 4 years ago

Gamechanger!

keepquestioning3• 4 years ago

One beautiful thing I realized about all this progress in AI.

We will still need people to do the hard yards, and get dirt between their fingernails. I am firmly in the camp of those people.

Fancy algorithms won't dig holes, or lay out rail tracks of over hundreds of miles.. or build houses all across the world.

blagie3• 4 years ago

Are you following progress in robotics?

dougmwne2• 4 years ago

School us! What’s the latest in robotics that is going to knock our socks off?

keepquestioning3• 4 years ago

Once they combine the progress in generative algorithms and robotics, then we are truly done for.

avereveard• 4 years ago

How fast is it on a m1?

ebiester2• 4 years ago

It takes 1-2 mins for a 512x512 image. It's been a lot of fun since I did this last night.

smoldesu3• 4 years ago

For reference, inferencing the model on a 2070 takes 10-12 seconds for the same size at max-precision, and the 3070 can synthesize an image in almost 6 seconds.

If you extrapolate the power consumption (3070 @~300w vs M1 Pro GPU@~30-50w) the metrics make a lot of sense.

nathas7• 4 years ago

I haven't ran this fork yet, but about 1.3 sec/iter. Usually ~30-50 iters/sample (image).

pwinnski5• 4 years ago

The answer depends VERY MUCH on RAM.

My M1 with 8GB takes 70-90 minutes per image.

My M1 Pro with 16GB takes 3 minutes per image.

fifafu• 4 years ago

Another data point: M1 Max with 64GB takes ~40 seconds

astrange2• 4 years ago

Hopefully the library is intentionally going very slowly trying to fit in RAM and that's not just your 8GB machine totally falling over.

Birch-san4• 4 years ago

9.3 secs if you sample for 7 steps. 11 secs if you sample for 8 steps. Heun sampler, M1 Max. https://twitter.com/Birchlabs/status/1564792349221330944?s=2...

ebiester2• 4 years ago

Note: I ran this and haven't yet been able to get img2img working yet. I borked it up trying to get conda working.

It's been a lot of fun to play with so far though!

bravura• 4 years ago

Apparently, this release should include a Dockerfile for easier replicability.

bfirsh11• 4 years ago

Unfortunately this can't run in Docker because Docker for Mac can't access the M1 GPU. (Several layers of virtualization and emulation!)

nathas7• 4 years ago

Try the lstein fork: https://github.com/lstein/stable-diffusion/tree/fix-cuda-res...

You'll still need to play with modifying some of the code to get it to run, but `dream.py` works for me. Funny enough, I got only img2img effectively working with the lstein branch; it broke txt2img for me.

StapleHorse• 4 years ago

Yesterday I thought I broke it too. In my case, the solution was just to make sure that the input image (from the editor or otherwise) was the same size as the output image. Hope it helps.

caxco93• 4 years ago

could someone who has already done this please share how long it takes for a 50 steps image to be generated?

nathas7• 4 years ago

1.3 sec/iter on my M1 Mac, so ~39 seconds.

jw12248• 4 years ago

That was fast. I'm only getting 5.26s/iter on an M1 Pro MBP with 16GB RAM.

EDIT: Speed increased to 2.3s/iter after a reboot

blagie3• 4 years ago

How large an image will this handle (versus how much RAM you have)?

It seems the GPU memory requirements beyond 512x512 are obscene.

michaelchisari3• 4 years ago

I'm on a iMac M1 16gb and I can handle up to 768x768 but since it's shared memory I close out every other application and run things overnight.

The biggest issue with apple chips is that the --seed setting doesn't work. I should be able to set a seed to, for instance, 1083958 and if I re-run a command at the same resolution with that seed, I should get the same image every time. This would allow me to test different steps so I could generate a 100 images at 16 steps (which is quite fast) and pick the ones that are most promising and re-render at 64 or 128 steps.

But currently you can't do that on apple hardware because of an open issue in PyTorch. Genuinely hoping a fix comes soon, until it is this is more of a novelty than a tool on Apple hardware.

fragmede6• 4 years ago

There's a partial fix for the seed issue on Reddit.

pugio3• 4 years ago

Me at the end of last year: "Should I really go for the full 64GB on this M1 Pro? What could I possibly use this for? mmbml mumble... something about unified GPU... something Deep Learning, one day..."

Me now: "a red juicy apple floating in outer space, like a planet" --H 768 --W 768

Uses about 27GB. 1.81s/it.

Can't do 1024x1024 yet because of some hardcoded Metal issue (https://github.com/pytorch/pytorch/issues/84039.

michaelchisari3• 4 years ago

I was so geniunely happy with my 16GB iMac M1 until stable-diffusion released.

I was amazed at how fast and powerful it was. I thought this meant I could stop buying top-of-the-line Macs every 4 years and start buying bottom-of-the-line Macs every 5 years. And that would have been 100% true... if it weren't for stable-diffusion.

neurostimulant2• 4 years ago

Most people that want hires would just feed the resulting image into AI upscaler like gigapixel.

schleck88• 4 years ago

This model was mostly trained on 512x512 so you should stick to approximately that size.

Use external upscalers like RealESRGAN, SwinIR or BSRGAN or GFPGAN (faces).

Alternatively use hacks like txt2imghd to get it to natively create 1 MP images.

sxp• 4 years ago

Is there a good set of benchmarks available for Stable Diffusion? I was able to run a custom Stable Diffusion build on a GCE A100 instance (~$1/hour) at around 1Mpix per 10 seconds. I.e, I could create a 512x512 image in 2.5 seconds with some batching optimizations. A consumer GPU like a 3090 runs at ~1Mpix per 20 seconds.

I'm wondering what the price floor of stock art will be when someone can use https://lexica.art/ as a starting point, generate variations of a prompt locally, and then spend a few minutes sifting through the results. It should be possible to get most stock art or concept art at a price of <$1 per image.

skybrian• 4 years ago

So you’re estimating over a thousand generated images an hour and less than a tenth of a cent per image using the A100. If that turns out to be accurate, it seems like some online image generation will included in the price of the stock art.

(DreamStudio is charging a bit over one cent per generated image at default settings, depending on exchange rates.)

sowbug• 4 years ago

Related: I wrote up instructions for running Stable Diffusion on GCE. I used a Tesla T4, which is probably the cheapest that can handle the original code. If you're spinning up an instance to play with, rather than to batch-process, then cheaper makes more sense because most of the machine's time is spent waiting for you to type stuff and look at the results.

https://sowbug.com/posts/stable-diffusion-on-google-cloud/

fleddr2• 4 years ago

It can be even cheaper.

Midjourney, in case you appreciate their output, has an unlimited plan for 30$ a month. The only limitation is that if you're an extremely heavy user, they may "relax" you, which means results come in a bit slower.

Note that they've been also experimenting with a --beta parameter which basically means the algorithm uses StableDiffusion's algorithm behind the scenes, or you can use any of 4 versions of MidJourney's more stylistic algorithms.

So if you don't want to tinker or don't have a high-end GPU, it's a cheap way to play around. I have StableDiffusion running locally but still prefer MidJourney. I enjoy the stylistic output but it's also a highly social way to generate art. Everybody is doing it in the open.

Anyway, the stock art part is a hairy subject. You should assume that you AI image is not copyrighted. Which begs the question why they would pay at all.

esperent• 4 years ago

The only limitation is that if you're an extremely heavy user, they may "relax" you, which means results come in a bit slower.

You don't have to be an extremely heavy user. I used it for about an hour every evening and it took 11 days out of a month subscription for them to put me on relax mode.

The relax mode is based on how busy the service is. If usage is low, it's the same a fast mode. But other times its really slow.

That makes it unpredictable enough that it stopped being fun for me to use it. I've barely used midjourney since I got put on relaxed mode - it stopped feeling like I can jump on and play because I might hit a busy period and then it'll take 5 minutes to generate a prompt

That said, I could buy more hours of fast mode and I think it's still way cheaper than Dall-E or Dreamstudio

brokenodo• 4 years ago

My 3170Ti will create 512x512 image in about 5-6 seconds with 50 inference steps

butUhmErm• 4 years ago

Between this and efforts to add 3D dimension to 2D images, I don’t see much of a future for digital multimedia creator jobs.

Even TikTok could be an endless stream of ML models.

Fears of a tech dystopia may be overblown; the masses will just shut off their gadgets and live simpler if labor markets implode within the traditional political correct economic system we have.

Open source AI is on the verge of upending the software industry and copyright. I dig it.

johnfn10• 4 years ago

For those as keen as I am to try this out, I ran these steps, only to run into an error during the pip install phase:

ERROR: Failed building wheel for onnx

I was able to resolve it by doing this:

brew install protobuf

Then I ran pip install again, and it worked!

geerlingguy4• 4 years ago

In the troubleshooting section it mentions running:

    brew install Cmake protobuf rust

To fix onnx build errors. I had the same issue.

jonplackett• 4 years ago

What kind of speed does this run at? Eg. How long to make a 512x512 image at standard settings?

jw12248• 4 years ago

On my M1 Pro MBP with 16GB RAM, it takes ~3 minutes.

pwinnski5• 4 years ago

I haven't installed from this link specifically, but I used one of the branches on which this is based a few days ago, so the results should be similar.

On a first-gen M1 Mac mini with 8GB RAM, it takes 70-90 minutes for each image.

Still feels like magic, but old-school magic.

johnfn10• 4 years ago

Looks like I'm getting around 4s per iteration on my M1 Max. At 50 iterations, that's 200 seconds.

whywhywhywhy• 4 years ago

M1 Max (32gb) is around 35 seconds per image.

chemeng• 4 years ago

Getting around 4 minutes per image on M1 MacBook Air 16GB

moneycantbuy4• 4 years ago

For 512x512 on M1 MAX (32 core) with 64 GB RAM I'm getting 1.67it/s so 30.59s with the default ddim_steps=50.

_ph_2• 4 years ago

On my M2 Air, 16G, 10 CPU cores, the default command as in the installing instructions takes like 2m20s.

chime• 4 years ago

MacBook Air M2 8 CPU 8GB the example apple image took 35mins. Guess I'll wait for now.

matsemann• 4 years ago

Python dependency hell in a nutshell. Impossible to distribute ML projects that can easily be ran.

leetbulb• 4 years ago

I just had to:

brew link protobuf --overwrite

Don't blindly run this command unless you understand what you're doing.

mark_l_watson2• 4 years ago

Thanks for writing this up!! I enjoyed getting TensorFlow running with the M1, although a multi-headed model I was working on wouldn’t run.

I just made my Dad’s 101 year old birthday card using OpenAI’s image generating service (he loved it) and when I get home from travel I will use your instructions in the linked article.

Any advice for running Stable Diffusion locally vs. Colab Pro or Pro+? My M1 MacBook Pro only has 8G ram (I didn’t want to wait a month for a 16G model). Is that enough? I have a 1080 with 10G graphics memory. Is that sufficient?

Razengan• 4 years ago

101 years! Congratulations!! Does he own a suspiciously plain gold ring by any chance?

mark_l_watson2• 4 years ago

no :-)

ErneX2• 4 years ago

From the comments here 8GB is not enough, it will swap a lot and take way more time than a 16GB MacBook.

r3trohack3r2• 4 years ago

I've been playing with Stable Diffusion a lot the past few days on a Dell R620 CPU (24 cores, 96 GB of RAM). With a little fiddling (not knowing any python or anything about machine learning) I was able to get img2img.py working by simply comparing that script to the txt2img.py CPU patch. Was only a few lines of tweaking. img2img takes ~2 minutes to generate an image with 1 sample and 50 iterations, txt2img takes about 10 minutes for 1 sample and 50 generations.

The real bummer is that I can only get ddim and plms to run using a CPU. All of the other diffusions crash and burn. ddim and plms don't seem to do a great job of converging for hyper-realistic scenes involving humans. I've seen other algorithms "shape up" after 10 or so iterations from explorations people do online - where increasing the step count just gives you a higher fidelity and/or more realistic image. With ddim/plms on a CPU, every step seems to give me a wildly different image. You wouldn't know that steps 10 and steps 15 came from the same seed/sample they change so much.

I'm not sure if this is just because I'm running it on a CPU or if ddim and plms are just inferior to the other diffusion models - but I've mostly given up on generating anything worthwhile until I can get my hands on an nvida GPU and experiment more with faster turn arounds.

squeaky-clean7• 4 years ago

You wouldn't know that steps 10 and steps 15 came from the same seed/sample they change so much.

I don't think this is CPU specific, this happens at these very low number of samples, even on the GPU. Most guides recommend starting with 45 steps as a useful minimum for quickly trialing prompt and setting changes, and then increasing that number once you've found values you like for your prompt and other parameters.

I've also noticed another big change sometimes happens between 70-90 steps. It's not all the time and it doesn't drastically change your image, but orientations may get rotated, colors will change, the background may change completely.

img2img takes ~2 minutes to generate an image with 1 sample and 50 iterations

If you check the console logs you'll notice img2img doesn't actually run the real number of steps. It's number of steps multiplied by the Denoising Strength factor. So with a denoising strength of 0.5 and 50 steps, you're actually running 25 steps.

Later edit: Oh and if you do end up liking an image from step 10 or whatever, but iterating further completely changes the image, one thing you can do is save your output at 10 steps, and use that as your base image for the img2img script to do further work.

schleck88• 4 years ago

With the 1.4 checkpoint, everything under 40 steps can't be used basically and you only get good fidelity with >75 steps. I usually use 100, that's a good middleground.

auggierose• 4 years ago

How do you change these steps in the given script? Is it the --ddim_steps parameter? Or --n_iter? Or ... ?

Aeolun3• 4 years ago

I found I got quite decent results with 15-30 steps when generating children’s book illustrations (of course, no expectation for hyperrealism there)

wokwokwok2• 4 years ago

https://github.com/Birch-san/stable-diffusion has altered txt2img to support img2img and added other samplers, see:

https://github.com/Birch-san/stable-diffusion/blob/birch-mps...

That branch (birch-mps-waifu) runs on M1 macs no problem.

yoyohello132• 4 years ago

Has anybody had success getting newer AMD cards working?

ROCm support seems spotty at best, I have a 5700xt and I haven't had much luck getting it working.

geerlingguy4• 4 years ago

I've tried using this set of steps [1], but have so far not had luck, mostly because the ROCm driver setup is throwing me for a loop. Tried it with an RX 6700 XT and first was going to test on Ubuntu 22.04 but realized ROCm doesn't support that OS yet, so tried again on 20.04 and ended up breaking my GPU driver!

[1] https://gist.github.com/geerlingguy/ff3c3cbcf4416be2c0c1e0f8...

my1233• 4 years ago

Yes. That's expected.

AMD market segmented their RDNA2 support in ROCm to the Navi21 set only (6800/6800 XT/6900 XT).

It is not officially supported in any way on other RDNA2 GPUs. (Or even on the desktop RDNA2 range at all, that only works because their top end Pro cards share the same die)

switchers• 4 years ago

6600XT reporting in. Spent a few hours on Windows and WSL2 setup attempts, got no where. I don't run Ubuntu at home and don't want to dual boot just for this. From looking around I think I'd have a better chance on native Ubuntu.

my1233• 4 years ago

Buy an NVIDIA card. ROCm isn't supported in any way on WSL2, but CUDA is.

AMD just doesn't invest in their developer ecosystem. Also as you use a 6600 XT, no official ROCm support for the die that you use. Only for navi21.

haxxorfreak• 4 years ago

I have it working on an RX 6800, used the scripts from this repo[0] to build a docker image that has ROCm drivers and PyTorch installed.

I'm running Ubuntu 22.04 LTS as the host OS, didn't have to touch anything beyond the basic Docker install. Next step is build a new Dockerfile that adds in the Stable Diffusion WebUI.[1]

[0] https://github.com/AshleyYakeley/stable-diffusion-rocm [1] https://github.com/hlky/stable-diffusion-webui

tgtweak3• 4 years ago

The RX6800 seems like a great card for this - 16GB of relatively fast VRAM for a good price.

How long does it take to do 50 iterations on a 512x512?

ece• 4 years ago

I tried getting pytorch vulkan inference working with radv, it gives me a missing dtype error in vkformat. Fp16 or normal precision have the same error. I think it's some bf16 thing.

Daegalus3• 4 years ago

you can try my guide. got it working on a 6750XT

https://yulian.kuncheff.com/stable-diffusion-fedora-amd/

gonehome• 4 years ago

Thanks for this - it's rare to see a setup guide that actually works on each step!

I did need to run the troubleshooting step too, could probably just move that up as a required step in the guide.

bfirsh11• 4 years ago

It isn't required for some (most?) users. Weirdly sometimes pip is picking up the wheel for `onnx`, sometimes it isn't, and we can't figure out why.

Any Python packaging experts know what's going on? all macOS 12, arm64, Python 3.10. Can't think it wouldn't resolve the wheel.

But yes, good idea to move up. I'll stick it next to the `pip install`.

yboris4• 4 years ago

Please consider also adding a small note to help those few that get stuck with this bug:

    RuntimeError: expected scalar type BFloat16 but found Float

The solution is easy:

    append the execution command with `--precision full`

orf• 4 years ago

Different versions of pip.

jw12248• 4 years ago

Are we being pranked? I just followed the steps but the image output from my prompt is just a single frame of Rick Astley...

EDIT: It was a false-positive (honest!) on the NSFW filter. To disable it, edit txt2img.py around line 325.

Comment this line out:

    x_checked_image, has_nsfw_concept = check_safety(x_samples_ddim)

And replace it with:

    x_checked_image = x_samples_ddim

pja4• 4 years ago

That means the NSFW filter kicked in IIRC from reading the code.

Change your prompt, or remove the filter from the code.

johnfn10• 4 years ago

Haha, busted!

r3trohack3r2• 4 years ago

If you open up the script txt2img and img2img scripts, there is a content filter. If your prompt generated anything that gets detected as "inappropriate" the image is replaced with Rick Astley.

Removing the censor should be pretty straightforward, just comment out those lines.

nonethewiser2• 4 years ago

It bothers me that this isn't just configurable. Why would they not want to expose this as a feature?

joshmlewis• 4 years ago

When the model detects NSFW content it replaces the output with the frame of Rick Astley.

rhacker2• 4 years ago

It's kind of amazing that ML can now intelligently rick roll people.

I think it would be awesome to update the rickroll feature to the following:

Auto Re-run the img2img with some text prompt: "all of the people are now Rick Astley" with low strength so it can adjust the faces, but not change the nudity!!!1

_2d30• 4 years ago

Same thing happened to me which is especially odd as I literally just pasted the example command.

hackerlight• 4 years ago

It has a lot of false positives. A lot of my portraits of faces were marked as NSFW. Possibly detecting proportion of the image that's skin color?

ntr--• 4 years ago

Unrelated to stable diffusion, but I was showing DALL-E to my sister last night and a prompt with > Huge rubber tree set off the TOS violation filter.

AI alignment concerns are definitely overblown...

sp332• 4 years ago

Any chance of this running on an M1 iPad Pro?

boppo1• 4 years ago

Seconding this

liuliu• 4 years ago

Probably another week or two. running on M1 iPad Pro needs to get out of PyTorch, possibly export the model either through TorchScript and then do onnx conversion. From what I found so far, not many of these conversions done (except the OpenVINO one maybe?).

djhworld• 4 years ago

Note that once you run the python script for the first time it seems to download a further ~2GB of data

nonethewiser2• 4 years ago

Including a rick astley image for the first thing you gen -_-

ErneX2• 4 years ago

That’s the NSFW filter :D

johnfn10• 4 years ago

Hm, when I run the example, I get this error:

expected scalar type BFloat16 but found Float

Has anyone seen this error? It's pretty hard to google for.

nathas7• 4 years ago

Yeah. Try running with PYTORCH_ENABLE_MPS_FALLBACK=1 <script> --full-precision

johnfn10• 4 years ago

This worked!

yboris4• 4 years ago

For me `--full-precision` kept erroring out, but `--precision full` worked correctly.

bfirsh11• 4 years ago

Are you running macOS >=12.3?

johnfn10• 4 years ago

Oh, no I'm not, I'm on 12.0. Does this make a difference?

johnfn10• 4 years ago

Update: I solved this error more properly by upgrading to the latest version. Thanks bfirsh.

wuyishan• 4 years ago

I am having the same issue on MacOS 12.2.1 (21D62); Python 3.10.6 What did you upgrade to solve this? Thanks! (I can get it working with `--precision full`)

TekMol• 4 years ago

Does running it locally give you anything over using the web version?

bfirsh11• 4 years ago

You can hack on it, modify it, integrate it with other code, etc!

johnfn10• 4 years ago

Also, of course, it's entirely free. The web version is actually paid, though it's hard to tell because they're not super transparent about the fact that you're steadily eating through a quota of initial tokens.

schleck88• 4 years ago

You can finetune the model with Textual Inversion. And you don't have the safety mechanism for nudity.

Gigachad3• 4 years ago

The online ones are usually bogged down with too much traffic. Locally seems to be much easier/faster

usehackernews• 4 years ago

Magnusviri[0], the original author of the SD M1 repo credited in this article, has merged his fork into the Lstein Stable Diffusion fork.

You can now run the Lstein fork[1] with M1 as of a few hours ago.

This adds a ton of functionality - GUI, Upscaling & Facial improvements, weighted subprompts etc.

This has been a big undertaking over the last few days, and I highly recommend checking it out. See the mac m1 readme [3]

[0] https://github.com/magnusviri/stable-diffusion

[1] https://github.com/lstein/stable-diffusion

[2] https://github.com/lstein/stable-diffusion/blob/main/README-...

jw12248• 4 years ago

Brilliant, thank you! I just got OP's setup working, but this seems much more user-friendly. Giving it a try now...

EDIT: Got it working, with a couple of pre-requisite steps:

0. `rm` the existing `stable-diffusion` repo (assuming you followed OP's original setup)

1. Install `conda`, if you don't already have it:

    brew install --cask miniconda

2. Install the other build requirements referenced in OP's setup:

    brew install Cmake protobuf rust

3. Follow the main installation instructions here: https://github.com/lstein/stable-diffusion/blob/main/README-...

Then you should be good to go!

EDIT 2: After playing around with this repo, I've found:

- It offers better UX for interacting with Stable Diffusion, and seems to be a promising project.

- Running txt2img.py from lstein's repo seems to run about 30% faster than OP's. Not sure if that's a coincidence, or if they've included extra optimisations.

- I couldn't get the web UI to work. It kept throwing the "leaked semaphor objects" error someone else reported (even when rendering at 64x64).

- Sometimes it rendered images just as a black canvas, other times it worked. This is apparently a known issue and a fix is being tested.

I've reached the limits of my knowledge on this, but will following closely as new PRs are merged in over the coming days. Exciting!

pugio3• 4 years ago

Can you describe how you did (/ are doing) this? Do you now need to use conda (as opposed to OPs pip only version)?

toinewx3• 4 years ago

I only get black images.

toinewx3• 4 years ago

I was able not to have black images by using a different sampler

--sampler k_euler

full command:

"photography of a cat on the moon" -s 20 -n 3 --sampler k_euler -W 384 -H 384

johnfn10• 4 years ago

I followed all these steps, but I got this error:

User specified autocast device_type must be 'cuda' or 'cpu'
Are you sure your system has an adequate NVIDIA GPU?

I found the solution here: https://github.com/lstein/stable-diffusion/issues/293#issuec...

dannywarner• 4 years ago

Thank you with those extra steps I got it working now myself. At least I think thank you. My work productivity for the next few days might not agree.

wokwokwok2• 4 years ago

FYI: black images are not just from the safety checker.

Yes, the safety checker will zero out images but can just turn it off with an “if False:”; Mostly black images are due to a bug, especially frustrating because it turns up on high step counts and means you’ve wasted time running it.

My experience has been roughly 2-4/32 of an image batch comes back black at the default settings, regardless of the prompt.

Just stamp out images in batches and discard the black ones.

sbierwagen2• 4 years ago

Instructions don't work here, dead ends at

  FileNotFoundError: [Errno 2] No such file or directory: 'models/ldm/stable-diffusion-v1/model.ckpt'

Looks like there's a step missing or broken at downloading the actual weights.

Going up to the parent repo points at a bunch of dead links or hugginface pages.

hhjinks• 4 years ago

I had to manually install pytorch for the preload_models.py step to work, because ReduceOp wasn't found. Why even use anaconda if all the dependencies aren't included? Every time I touch an ML project, there's always a python dependency issue. How can people use a tool that's impossible to provide a consistent environment for?

philsnow• 4 years ago

To get past `pip install -r requirements` I had to muck around with CFLAGS/LDFLAGS because I guess maybe on your system /opt/homebrew/opt/openssl is a symlink to something? On mine it doesn't exist, I just have /opt/homebrew/opt/openssl@1.1 symlinked to /opt/Cellar/somewhere.

The command that finally worked for me:

  python3 -m venv venv
  . venv/bin/activate
  CFLAGS="-I /opt/homebrew/opt/openssl@1.1/include" LDFLAGS="-L /opt/homebrew/opt/openssl@1.1/lib -L/opt/homebrew/Cellar/openssl@1.1/1.1.1q/lib -lssl -lcrypto" PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig" GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 pip install -r requirements.txt

bfirsh11• 4 years ago

Nice. We'll get this guide updated for this fork. Everything's moving so fast it's hard to keep track!

We struggled to get Conda working reliably for people, which it looks like lstein's fork recommends. I'll see if we can get it working with plain pip.

jw12248• 4 years ago

Check my comment alongside yours, I got Conda to work but it did require the pre-requisite Homebrew packages you originally recommended before it would cooperate :)

pugio3• 4 years ago

I really appreciate the use of pip > conda. Looking forward to the update for the repo!

bfirsh11• 4 years ago

Running lstein's fork with these requirements[0] but seeing this output[1]. Same steps as original guide otherwise.

Anyone got any ideas?

[0] https://github.com/bfirsh/stable-diffusion/blob/392cda328a69...

[1] https://gist.github.com/bfirsh/594c50fd9b2e6b173e31de753a842...

wincy3• 4 years ago

I couldn't get the setup process working until I switched the python distro to 3.10, as the scripts were relying on typings features that were added in 3.10 even though the yml file specified 3.9. Was strange.

fragmede6• 4 years ago

Conda is recommended because it starts from a clean environment so you're not debugging 13 other experiments the user has going on.

toinewx3• 4 years ago

Everything works excepts it only generates black images,

did you run

python scripts/preload_models.py

python scripts/dream.py --full_precision ?

arthurcolle2• 4 years ago

Disable safety check

yieldcrv2• 4 years ago

are there benchmarks?

I was following the github issue and the CPU bound one was at 4-5 minutes, the MDS one was at 30 seconds, then 18 seconds, and people were still calling that slow.

What is it currently at now?

and I don't know what "fast" is, to compare

What are the Windows 10 with nice Nvidia chips w/ CUDA getting? Just curious whats comprehensive

zone4114• 4 years ago

Around 6 seconds.

squeaky-clean7• 4 years ago

What are the Windows 10 with nice Nvidia chips w/ CUDA getting?

Are you referring to single iteration step times, or whole images? Because obviously it depends on the number of iteration steps used.

Windows 10, RTX 2070 (laptop model), lstein repo. I get about 3.2 iter/sec. A 50 step 512x512 image takes me 15 seconds.

dmd2• 4 years ago

Wait, what? On my M1 imac I’m getting about 25 minutes. What am i doing wrong?

solarkraft2• 4 years ago

I ran into:

ImportError: cannot import name 'TypeAlias' from 'typing' (/opt/homebrew/Caskroom/miniconda/base/envs/ldm/lib/python3.9/typing.py)

icedchai3• 4 years ago

I ran into this. You need Python 3.10. I had to edit environment-mac.yaml and set python==3.10.6 ...

itsuka• 4 years ago

I followed the conda instruction which uses Python 3.9 and ran into the same issue. The workaround is to import TypeAlias from typing_extensions:

stable-diffusion/src/k-diffusion/k_diffusion/sampling.py

(before)

  from typing import Optional, Callable, TypeAlias

(after)

  from typing import Optional, Callable
  from typing_extensions import TypeAlias

This issue is tracked in https://github.com/lstein/stable-diffusion/issues/302

personjerry• 4 years ago

TypeAlias is only used once, you can open sampling.py and remove the import on line 10 and the usage on line 14:

  from typing import Optional, Callable

  from . import utils

  TensorOperator = Callable[[Tensor], Tensor]

sanroot992• 4 years ago

What's the performance of these models ,how much pc spec required for sane operation?

Yido2• 4 years ago

Cool

totetsu2• 4 years ago

What do I need for the in painting? is there a source for the models/ldm/inpainting_big/last.ckpt' file?

badc0ded• 4 years ago

I used this: wget -O models/ldm/inpainting_big/last.ckpt https://heibox.uni-heidelberg.de/f/4d9ac7ea40c64582b7c9/?dl=... Found it here: https://huggingface.co/spaces/multimodalart/latentdiffusion/...

This worked afterwards: python scripts/inpaint.py --indir data/inpainting_examples/ --outdir outputs/inpainting_results

dgreensp3• 4 years ago

I'm working on getting this running. Instead of "venv/bin/activate" I had to run "source venv/bin/activate". And I got an error installing the requirements, fixed by running "pip install pyyaml" as a separate command.

dgreensp3• 4 years ago

Wow, I'm getting as low as 1.2 seconds per "step" (about a minute for a 512x512 image with default settings) on my 32 GB M1 laptop (2021, 16-inch).

valley_guy_124• 4 years ago

Having to use "source" means you have an older version of conda. Python package management is kind of a mess.

bfirsh11• 4 years ago

There's a little "." before "venv/bin/activate" that's easy to miss. I'll update it to "source" to make it more obvious.

dgreensp3• 4 years ago

Great. FYI, the issue with pyyaml was caused by the "--pre" flag. The reason installing it separately fixes the problem is because it installs it without the "--pre" flag, and then it is already installed when you install the requirements.

I haven't been able to make sense of n-samples and n-iters. Changing the former caused generation to freeze at 0%. Changing the latter seems to generate multiple images, even though the SD docs for n-iters are "sample this often".

rhacker2• 4 years ago

I wonder if this is going to be a huge boon to m1 sales.

sgt1012• 4 years ago

might be easier to wait for Diffusers to merge the pull request...

sroussey2• 4 years ago

This should be put into a docket image to avoid various potential conflicts with locally installed libraries.

Anyone do this for the M1?

bfirsh11• 4 years ago

Unfortunately this can't run in Docker because Docker for Mac can't access the M1 GPU. (Several layers of virtualization and emulation!)

sroussey2• 4 years ago

Ah, thanks. Yes, that makes sense.

schleck88• 4 years ago

Conda environment

hnarayanan• 4 years ago

Do you want to add more layers to make it extra slow?

sgt1012• 4 years ago

also brew upgrade not brew update

ChildOfChaos2• 4 years ago

Is there anyway to keep up with this stuff / beginners guide? I really want to play around with it but it's kinda confusing to me.

I don't have an M1 Mac, I have an Intel one with an AMD GPU, not sure if i can run it? don't mind if it's a bit slow, or what is the best way of running it in the cloud? Anything that can product high res for free?

Karuma• 4 years ago

Yes, you can run it on your Intel CPU: https://github.com/bes-dev/stable_diffusion.openvino

And this should work on an AMD GPU (I haven't tried it, I only have NVIDIA): https://github.com/AshleyYakeley/stable-diffusion-rocm

There are also many ways to run it in the cloud (and even more coming every hour!) I think this one is the most popular: https://colab.research.google.com/github/altryne/sd-webui-co...

holoduke2• 4 years ago

follow this guide: https://github.com/lstein/stable-diffusion/blob/main/README-...

i am runnig it on my 2019 intel macbook pro. 10 minutes per picture

EddySchauHai• 4 years ago

https://beta.dreamstudio.ai/dream

It's not free but I've played with it a lot over the last two days for around $10, generating the most complex photos I can (1024x1024, 150 steps, 9 images, etc)

Daegalus3• 4 years ago

I wrote a guide for AMD.

https://yulian.kuncheff.com/stable-diffusion-fedora-amd/

it's for, but it could be adapted to Linux as.lomg as you install the right drivers and such.

yreg• 4 years ago

Have you managed to set it up? I might have the same computer as you.

ChildOfChaos2• 4 years ago

Not yet, I haven't had much time to look into it all yet.

Looks like it's going to be a lot of fun though.

code512• 4 years ago

Without k-diffusion support, I don't think this replicates Stable Diffusion experience:

https://github.com/crowsonkb/k-diffusion

Yes, running on M1/M2 (MPS device) was possible with modifications. img2img and inpainting also works.

However you'll run into problems when you want k-diffusion sampling or textual inversion support.

Birch-san4• 4 years ago

stable-diffusion supports k-diffusion just fine on M1. You just have to detach a tensor in to_d() to stop the values exploding to infinity. https://twitter.com/Birchlabs/status/1563622002581184517?s=2...

code512• 4 years ago

I've been following your MPS branch and have run it but couldn't address the issue without this explanation. Thank you!

jclardy• 4 years ago

Is there a proper term to encapsulate M1/M2 Macs now that we have the M2? IE Apple Silicon Macs works but is a bit long. MX Macs? M-Series? ARM Macs?

qayxc2• 4 years ago

M-series sounds great, IMHO.

pavlov• 4 years ago

"uname -m" returns "arm64" on these computers, so you could say macOS on arm64.

fragmede6• 4 years ago

The backend, as contributed by Apple is called MPS.

gzer0• 4 years ago

The difference between an M2 air (8gb/512gb) versus an M1 pro (16gb/1tb) is much more than I expected.

  * M1 pro (16gb/1tb) can run the model in around 3 minutes.
  * M2 air (8gb/512gb) takes ~60 minutes for the same model.

I knew there would be some throttling due to the m2 air's fanless model, but I had no idea it would be a 20x difference (albeit, the m1 pro does have double the RAM. I don't have any other macbooks to test this on).

qayxc2• 4 years ago

I suspect the lack of RAM is the issue here.

andybak2• 4 years ago

Unscientifically that puts the M1 Pro GPU at about 25% of the performance of a RTX 3080.

Not too shabby...

EDIT - this comment implies it's much faster: https://news.ycombinator.com/item?id=32679518

If that's correct then it's close to matching my 3080 (mobile).

fassssst• 4 years ago

img2img runs in 6 seconds on my GeForce 3080 12 GB. 6+ it\s depending on how much GPU memory is available. If I have any electron apps running it slows down dramatically.

valley_guy_124• 4 years ago

It's likely that a significant fraction of the perf difference between Apple' GPUs and NVIDIA GPUs is due to NVIDIA's CUDA being high optimized, and Pytorch being tuned to work with CUDA.

If Pytorch's metal support improves and Apple's Metal drivers improve (big ifs), it's likely that Apple's GPUs will perform better relatively to NVIDIA than they currently do.

valley_guy_124• 4 years ago

That's probably due to swapping due to the 8GB of RAM. People who have run Stable Diffusion on M2 airs with 16 GB of RAM seem to get performance that is in line with their GPU core count.

bfirsh11• 4 years ago

Correct. We've been seeing 8GB is super slow, >=16GB is fast. We'll add that to the prerequisites.

JimmyAustin• 4 years ago

I suspect that the M2 air is thrashing the disk pretty aggressively. Diffusion models rerun the same model once per step, so for a generation with 50 steps, you copy the entire model in and out of memory 50 times. That's going to kill performance.

bm-rf2• 4 years ago

I believe the model is copied into ram once upon calling StableDiffusionPipeline, unless the mac implementation partially loads the model due to only having 8G of ram.

schleck88• 4 years ago

It's only copied to VRAM once when implemented correctly.

bm-rf2• 4 years ago

just water cool it! https://www.youtube.com/watch?v=9DyUitTVWlw

_ph_2• 4 years ago

I would assume it is the memory. The test command from the discussed link runs in slightly over 2 minutes on my M2 Air (16gb). How long does it take for yours?

Birch-san4• 4 years ago

Neither of these should take minutes. Try Heun sampler, 8 steps, Karras noise schedule. Should be possible to get good images in 11 secs (or 10 secs if you go down to 7 steps). measurements admittedly from M1 Max. https://twitter.com/Birchlabs/status/1565029734865584143?s=2...

moneycantbuy4• 4 years ago

Anyone know the largest possible image size > 512x512? I'm getting the following error when trying 1024x1024 with 64 GB RAM on M1 MAX:

/opt/homebrew/Cellar/python@3.10/3.10.6_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

capableweb• 4 years ago

just 512x require something like 10GB VRAM on your GPU, 1024x would need even more. How much VRAM does the M1 Max GPU have? You're probably running out of memory.

tgtweak3• 4 years ago

M architecture is unified memory - so the system memory is shared as GPU memory. There is likely a cap somewhere on how much can be used by a single app (or collectively by all apps).

enduser• 4 years ago

I have the same problem with anything over 512x512 on my M1 Ultra with 128GB. VRAM must be capped.

moneycantbuy4• 4 years ago

Thanks for the Ultra data point.

I'm able to get 768x896 to run, but the output image is still white noisy at 50 ddim steps, perhaps related to the phenomena of being trained/windowed on 512x512 images as sibling squeaky-clean described.

RAM usage at various sizes: 512x512 14 GB, 768x768 26 GB, 768x896 32 GB

squeaky-clean7• 4 years ago

Don't specifically know sorry, largest I can generate with my (windows pc) vram size is 512x1024. But just wanted to comment that SD is trained on 512x512 images and runs iterations in a 512x512 window.

This means anything larger than 512x512 tends to confuse it. For example 1024x1024 will have 4 non-overlapping windows and many overlapping windows.

So if your prompt is "a cat wearing sunglasses", you may get 4 separate cats as the 512x512 windows have no knowledge of each other, and each window is trying to fulfill the goal. Even more likely you'll get some sort of eldrtich horror 16-legged cat being as the windows shuffle around.

Sometimes it just works perfectly somehow, but 90% of the time the non native resolution really screws it up. I'd suggest generating 512x512 images and using a different AI to upscale them in most cases.

However it does lead to some amazing fantasy landscape art as you get weird terrains and mountains shoved up against eachother in fantastic/magical ways.

valley_guy_124• 4 years ago

Supposedly Stable Diffusion was trained on 512 x 512 images, so it's not clear that it will work well for larger images even if you had the RAM.

To generate larger images, it is standard practice to generate 512 x 512 and then use a separate tool to upscale, and maybe a second separate tool to improve the face. The Windows versions of SD environments are starting to incorporate these additional tools, but the Apple Silicon versions of SD environments are lagging behind due to Pytorch metal limitations.... It'll hopefully sort itself out in the next few months.

deckeraa• 4 years ago

Very nice to see this available for hardware I own.

Now I can achieve my dream of a Corporate Memphis + Hieronymus Bosch mashup.

bamurphymac1• 4 years ago

Low entropy observer A38B gasps, spills his coffee and slams the panic button, triggering the 3758th reboot of the simulation.

“Sorry all. We can’t spare the hardware for THAT kind of recursive self improvement. Setting this instance back to 1970. GG”

adamj9431• 4 years ago

How is Stable Diffusion on DreamStudio.ai so much faster than the reports here? Seems to only take 5-10 seconds to generate an image with the default settings.

I.e. How are they providing access to GPU compute several orders of magnitude more powerful than an M1, for free?

tgtweak3• 4 years ago

A100 devices in the cloud on preemptive/spot instances?

schleck88• 4 years ago

1. Dreamstudio is paid, with 2 EUR worth of credit for free.

2. The M1 GPU is an iGPU. A good iGPU, sure, but it's not anywhere near the performance of a dedicated, cooled GPU with dedicated VRAM.

On a 2060 Super with 8 GB of VRAM and with tensor cores it takes 15 seconds to infer with the default settings. If Dreamstudio uses deep learning GPUs then there is your answer to why it is as fast.

fleddr2• 4 years ago

The guy behind it is an ex hedge fund manager. Using private funds he's built a massive fleet of A1000s at AWS.

So it's an enormous amount of compute created from private funds that he considers to be "for humanity". Currently, he funds it and is also the "GPU overlord", he exclusively decides which applications gets to use it.

His plan, or at least his claim, is to transform this situation in it being more diversely funded (institutions, businesses, even the UN) and for access to be decided by committee with main criteria it being useful for humanity.

Let's see if he sticks to his word, but I find it inspirational. AI was on a trajectory to be solely in the hands of a hand full of ultra rich companies that can afford to train and run it, and us poor mortals being at the whims of gatekeeper terms.

This guy is on a trajectory to put AI in the hands of the people. Not just for art, for everything. If he fully sees this through, he's destined to be a tech icon.

preommr• 4 years ago

The guy's name is Emad Mostaque btw

There's a recent video interview he did that goes into his vision: https://www.youtube.com/watch?v=YQ2QtKcK2dA

joshstrange2• 4 years ago

It's insane to me how fast this is moving. I jumped through a bunch of hoops 2-3 days ago to get this running on my M1 Mac's GPU and now it's way easier. I imagine we will have a nice GUI (I'm aware of the web-ui, I haven't set it up yet) packaged in an mac .app by the end of next week. Really cool stuff.

addandsubtract2• 4 years ago

I hope this kickstarts some kind of M1 migration. There are so many ML projects I'd like to try, but they all depend on CUDA.

joshstrange2• 4 years ago

Yep, I was just thinking the same thing. M1/M2 appears to be a huge untapped resource for ML stuff as this proves. I maxed out my MBP Max and this is probably the first time I'm actually fully using the GPU cores and it's pretty freaking cool. Creating landscapes or fictional characters (think D&D) is already super fun, I look forward to playing with img2img some more as well.

bee_rider2• 4 years ago

Do they depend on CUDA, or are they just much better tuned for NVIDIA cards? I thought the whole ML ecosystem was based on training models and then running them on frameworks, where model was sorta like data and the framework handles the hardware? (albeit with models that can be tweaked to run more efficiently on different hardware) (I don't really know the ecosystem so it is definitely possible that they are more closely tied together than I thought).

fezfight• 4 years ago

It's not really better to move from one closed eco system to another. We should collectively agree to strengthen more open platforms, shouldn't we?

inamberclad• 4 years ago

I'd rather see something more platform agnostic. I'm sad OpenCL isn't a bigger success.

danaris• 4 years ago

Gods, yes. I've been trying to get various kinds of deepfake-related projects to work on M1 (or even just on Mac—the Intel Macs haven't come with Nvidia cards for years now) for some time now, as we're trying to generate video stimuli that present different people doing the same things, or the same person doing several different things, for psych research.

It's been an exercise in frustration.

dekervin2• 4 years ago

Just yesterday I read another comment on HN saying we will have to wait another decade before being able train it in someone "basement"( https://news.ycombinator.com/item?id=32658941 ). I made a bookmark for myself ( https://datum.alwaysdata.net/?explorer_view=quest&quest_id=q... ) to look for data that help estimate when it will be feasible to run Stable Diffusion "at home". I guess it's already outdated!

zone4114• 4 years ago

Umm training is not the same as running it.

squeaky-clean7• 4 years ago

To run stable diffusion at home you have to download the model file, which took the equivalent of tens of thousands of hours spread across cloud provided GPUs.

If the model file just vanished from everyone's hard drive one day, and cloud providers installed heuristics to detect and ban image dataset training, retraining the model file would actually take decades for any consumer, even an enthusiast with a dozen powerful GPUs. The image dataset alone is 240TB.

simonebrunozzi• 4 years ago

I don't want to sound lazy, but I would be expecting a .dmg for Macs, and I don't seem to find it. Am I blind, or it simply hasn't been prepared yet?

omarelbie• 4 years ago

At the rate it's moving it doesn't seem too far off, but I think it's just a tad too early.

adrianvoica• 4 years ago

Tried "transparent dog", got rickrolled. Why is this NSFW? ...anyway, I disabled the filter and... it's pretty neat! Calling all AI Overlords, soon. :))

wenbin• 4 years ago

Thanks for the writeup! It works smoothly on my M1 Macbook Pro!

A few days ago, I tried Stable Diffusion code and was not able to get it work :( Then I gave up...

Today, following steps in this blog post, it works for the very first try. Happy!

imtemplain• 4 years ago

I'm ready to pay for a Windows + AMD GPU guide at this point, why is there no single blogpost on this, please help.

Daegalus3• 4 years ago

I'm sure you can do it by installing the ROCm drivers, install pytorch-rocm based on the download page for pytorch, and make sure to set any settings in environment variables.

you can try following my Linux AMD guide and see what you can pick and choose to get it working

https://yulian.kuncheff.com/stable-diffusion-fedora-amd/

bobthebl0b2• 4 years ago

I was like you, but man $2,000 no thanks. This saved my budget and time https://github.com/glonlas/Stable-Diffusion-Apple-Silicon-M1...

vvanirudh• 4 years ago

Running into this error `RuntimeError: expected scalar type BFloat16 but found Float` when I run `txt2img.py`

yboris4• 4 years ago

Confirming I'm stuck on the same error when running the tutorial-instructed python scripts/txt2img.py command

    RuntimeError: expected scalar type BFloat16 but found Float

ml_basics• 4 years ago

Yes, me too! Please post here if you find a solution for all the other people that come and find this by commmand-F'ing this error

yboris4• 4 years ago

SOLUTION - append the command with `--precision full`

sytse2• 4 years ago

Awesome, that works

For reference the full command:

`python scripts/txt2img.py \ --prompt "a red juicy apple floating in outer space, like a planet" \ --n_samples 1 --n_iter 1 --plms --precision full`

moneycantbuy4• 4 years ago

What's with the ~25% chance of an image being all black? Also, seeds aren't replicating.

amilios• 4 years ago

How long does it take to generate a single image? Is it in the 30 min type range or a few mins? It's hypothetically "possible" to run e.g. OPT175B on a consumer GPU via Huggingface Accelerate, but in practice it takes like 30 mins to generate a single token.

LanternLight83• 4 years ago

Runs on my 2070S at 12s/image (no batch optimization) and on my GTX1050 4GB at 90s/image

holoduke2• 4 years ago

On my late 2019 intel macbook pro with 32gb and a AMD 5550m it takes about 7-10 minutes to generate an image.

Gigachad3• 4 years ago

I'm using a 2021 Macbook Pro with the base tier M1 Pro and it generates images in about 1 minutes per image.

Birch-san4• 4 years ago

About 10 secs per image on M1 Max with the right noise schedule and sampler. https://twitter.com/Birchlabs/status/1565029734865584143?s=2...

lostmsu• 4 years ago

I was able to run YaLM 100B in about 5min per iteration, NVMe being the bottleneck.

mrkstu• 4 years ago

I consistently have items only partially in frame- horses/fish/etc- any tips on getting the algo to keep specified items fully in frame?

SirYandi• 4 years ago

Struggle with this too. One keyword which helps is 'wide angle'. Sometimes 'full body shot' works for generating humans.

mdswanson• 4 years ago

virtualenv isn't required. You can just use python -m venv venv and get the same results with one fewer dependency.

andrethegiant• 4 years ago

Yesssss I've been waiting for this!

_venkatasg2• 4 years ago

I keep running into issues, even after installing Rust in my condo environment (using conda). Specifically the issue seems to be building wheels for `tokenizers`:

  warning: build failed, waiting for other jobs to finish...
  error: build failed
  error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module -- --crate-type cdylib -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-310-darwin.so'` failed with code 101
  [end of output]
 note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
  Failed to build tokenizers
  ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Any suggestions?

benhalllondon• 4 years ago

I played around a bit and found out dropping the tokenisers version to 0.11.6 worked

`pip install tokenizers==0.11.6` first

_venkatasg2• 4 years ago

That worked thank you!

msoad• 4 years ago

Please someone package all of this and the WebUI into an Electron app so common people can also hack on it!

andrewmunsell• 4 years ago

There are several packages that provide web UIs, like this one for example: https://github.com/hlky/stable-diffusion-webui

It's not quite the ease of setup of an Electron app, but once setup it's pretty easy to use.

gregsadetsky3• 4 years ago

Bananas. Thanks so much... to everyone involved. It works.

14 seconds to generate an image on an M1 Max with the given instructions (`--n_samples 1 --n_iter 1`)

Also, interesting/curious small note: images generated with this script are "invisibly watermarked" i.e. steganographied!

See https://github.com/bfirsh/stable-diffusion/blob/main/scripts...

grishka• 4 years ago

Also, interesting/curious small note: images generated with this script are "invisibly watermarked" i.e. steganographied!

Why?

Retr0id• 4 years ago

So that future iterations of StableDiffusion (or similar models) don't end up getting trained on their own outputs.

cageface• 4 years ago

After playing around with all of these ML image generators I've found myself surprisingly disenchanted. The tech is extremely impressive but I think it's just human psychology that when you have an unlimited supply of something you tend to value each instance of it less.

Turns out I don't really want thousands of good images. I want a handful of excellent ones.

chrisfrantz• 4 years ago

Human curation will likely remain valuable into the future.

Gigachad3• 4 years ago

What's this log message about when generating an image?

Creating invisible watermark encoder (see https://github.com/ShieldMnt/invisible-watermark)...

gregsadetsky3• 4 years ago

See my sibling comment here https://news.ycombinator.com/item?id=32684003 :-)

The code that generates it is here: https://github.com/bfirsh/stable-diffusion/blob/main/scripts...

You can remove `img = put_watermark(img, wm_encoder)` which appears at lines 317 and 333 to get rid of the watermarking.

js2• 4 years ago

A few suggested changes to the instructions:

    /opt/homebrew/bin/python3 -m venv venv  # [1, 2]
    venv/bin/python -m pip install -r requirements.txt  # [3]
    venv/bin/python scripts/txt2img.py ...

1. Using /opt/homebrew/bin/python3 allows you to remove the suggestion about "You might need to reopen your console to make it work" and ensures folks are using the just installed via homebrew python3, as opposed to Apple's /usr/bin/python3 which is currently 3.8. It also works regardless of the user's PATH. We can be fairly confident /opt/homebrew/bin is correct since that's the standard homebrew location on Apple Silicon and folks who've installed it elsewhere will likely know how to modify the instructions.

2. No need to install virtualenv since Python 3.6 which ships with a built-in venv module which covers most use cases.

3. No need to source an activate script. Call the python inside the virtual environment and it will use the virtual environment's packages.

schappim2• 4 years ago

I just got rick-rolled by the model.

Using the prompt: "1990s textbook background mephis style"[sic] (yup I meant memphis)[0], I got back this: [1]. Rerunning the same prompt, I got: [2].

[0] https://files.littlebird.com.au/Shared-Image-2022-09-02-10-2...

[1] https://files.littlebird.com.au/grid-0004-2xXAGF.png

[2] https://files.littlebird.com.au/grid-0005-kcfgq7.png

fastball• 4 years ago

The rick-roll is the NSFW filter. It's not actually that great at detecting NSFW content.

schappim2• 4 years ago

Thanks! You of course are right :)

    txt2img.py line 324 update to
    x_checked_image = x_samples_ddim #, has_nsfw_concept = check_safety(x_samples_ddim)

... disables the NSFW so you don’t get the rickroll images. It seems to be over enthusiastic in terms of its NSFW detection

bschwindHN3• 4 years ago

Everyone posting their pip/build/runtime errors is everything that's wrong with tooling built on top of python and its ecosystem.

It would be nice to see the ML community move on to something that's actually easily reproducible and buildable without "oh install this version of conda", "run pip install for this package", "edit this line in this python script".

smoldesu3• 4 years ago

I found a pretty great Docker image for the SD webui. I was forced to extreme measures since NixOS isn't super friendly with Conda (and I was particularly lazy).

Worked out fine in the end, though. Highly recommended if you're on an Nvidia rig: https://github.com/AbdBarho/stable-diffusion-webui-docker

sanroot992• 4 years ago

Docker the silver bullet /s

nl4• 4 years ago

I don't think this is particularly fair. This is literally hours old, and people installing now are really debugging rather than installing a "finished" build.

The forks are weird mashups of bits of repos, and running on a M1 GPU is something that barely works itself.

Give it maybe 3 months and it will be much smoother.

bschwindHN3• 4 years ago

I think it is fair, actually. This is not unique to hours-old python projects, it's a common theme among almost all python tools I've used.

I have a suspicion that that something written in Julia, Go, Rust, or possibly even C wouldn't have nearly this many issues. I'm not talking about debugging the actual functionality of the software, but rather the environment and tooling surrounding the language and software built with it.

This project in particular should be an easy case because you know the hardware you'll be running on ahead of time.

I'm ranting a bit, but I've tried so many tools based on python and almost _none_ of them built/installed/ran correctly on the happy path laid out in each project's readme.

Anyway, sorry, rant over.

pdehaan• 4 years ago

My anecdotal experience is that it briefly gets better, then much worse. Build scripts downloading tarballs from dead URLs and dependencies on unspecified versions of libraries that have since had breaking API changes are frequent issues.

…it’s still awesome that people put in the effort to do these things at all, but the tools often have a tendency to make me feel like an archaeologist trying to piece what ancient artifacts are missing and how they were supposed to all fit together.

zmmmmm• 4 years ago

Interesting question to me is whether it's actually in part fundamental to the success of Python. There are plenty of "clean" ecosystems that avoid at least some of these issues. But in generally they fail to thrive in these type of spaces and people keep coming back to ecosystems that are messier.

Is it possible that creativity and innovation actually require a level of chaos to succeed? Or alternatively, that chaos is an inevitable byproduct of creativity and innovation, and any ecosystem where these are heavily frowned on deters the type of people who actually drive the cutting edge forward?

Just putting it out there as food for thought. It does translate back to quite often, that it acutally does make a lot of sense to do your prototyping and experimenting in one ecosystem but then leave that behind to deploy your production workloads where possible.

brundolf• 4 years ago

You can have an ecosystem that's chaotic (and vibrant) without its core pillars being chaotic (and shaky)

colordrops• 4 years ago

JavaScript is a much cleaner ecosystem but for some reason there's a long running stigma against it. It would work fine for this use case.

systemvoltage• 4 years ago

I disagree, dependencies are sort of a universal problem. Ever had to set LD_LIBRARY_PATH?

Python is pretty much innocent here.

forrestthewoods2• 4 years ago

Python is pretty much innocent here.

Could not possibly disagree more. https://xkcd.com/1987/

The problem with dependencies is that for very bad reasons people don’t ship them.

Someone needs to package SD with a full copy of the Python runtime and every dependency. This should be the default method of distribution.

#ShipYourDamnDependencies

mrtksn2• 4 years ago

It's not just the python, this is the experience practically everywhere and that's why people create containers etc. It's excruciatingly hard to setup the environment to start doing anything productive these days, you can't just start coding unless you use an IDE like Xcode or PyCharm.

konart• 4 years ago

Not sure about that. Golang or Rust have no such problem in my experience.

rattray2• 4 years ago

JS isn't perfect, but it's so much easier to deal with than Python in these regards.

sbierwagen2• 4 years ago

I ran into many of these same problems when trying to replicate ROS environments five years ago. It's not Python's fault, it's just academics being bad at releasing software. After all, why would they?

In a university, ten steps to replicate the PI's personal environment is perfectly fine. How would releasing a single binary make their life any easier? Why would they bother?

fulafel• 4 years ago

Looking through the top level comments, there are only 2 like that and they are about build errors in dependent native libraries written in other languages, which is not really the Python ecosystem's fault (as the author has chosen not to distribute prebuilt stuff).

e403• 4 years ago

For me:

    File "/Users/layer/src/stable-diffusion/venv/lib/python3.10/site-packages/torch/serialization.py", line 250, in __init__
      super(_open_file, self).__init__(open(name, mode))
    FileNotFoundError: [Errno 2] No such file or directory: 'models/ldm/stable-diffusion-v1/model.ckpt'

The directory is empty. Hmm.

I forgot to

    mv sd-v1-4.ckpt models/ldm/stable-diffusion-v1/model.ckpt

On a Mac Studio

    data: 100%|| 1/1 [00:43<00:00, 43.20s/it]
    Sampling: 100%|| 1/1 [00:43<00:00, 43.20s/it]

Yido2• 4 years ago

Interesting!

RosanaAnaDana• 4 years ago

This whole 2 month period has felt like the first few steps onto some kind of exponential.

e403• 4 years ago

I just found out that Activity Monitor doesn't show GPU activity. :(

anderiv• 4 years ago

Perhaps you just missed it. It appears as a column on the CPU tab.

e403• 4 years ago

I guess I meant the Dock icon doesn't show GPU's, only CPU history.

But, I didn't know about the feature. I just tried it and it went to 98% while running SD.

dzink2• 4 years ago

If you have a top of the line M1 MBP but the hard drive is 2TB, would it make sense to plug in an external hard drive for the 4TB model or would it render the effort futile due to performance issues?

ggerganov• 4 years ago

The model is 4GB, not 4TB

Myrmornis2• 4 years ago

The various articles/tutorials seem a bit confusing: even though they say "M1", they also worked fine for me on an Intel Mac (and does end up using GPU).

Does anyone know how to think about the --W --H and --f flags to create larger images? I have 64GB memory, but I get errors from PyTorch saying things like "Invalid buffer size: 7.54 GB" when I try to increase W and H, and I haven't managed to make the Python process use more than about 15GB by playing around so far.

wut42• 4 years ago

That must be using CUDA then, and you need a gpu with at least 8GB of VRAM, afaik (not RAM).

Myrmornis2• 4 years ago

Ah-ha, thanks. My Intel Mac (2019) has https://www.techcenturion.com/intel-uhd-graphics-630

Being an Integrated GPU, the Intel UHD Graphics 630 doesn’t have any Video/Graphics Memory of its own. Instead, it utilizes the system’s memory (RAM) dynamically for the same purpose. You can change the maximum Video Memory from the BIOS settings.

I get the impression Apple isn't going to give me much control over that (https://www.reddit.com/r/macmini/comments/e82knm/can_i_set_h...). I did have "Automatic Graphics Switching" enabled, but I'm seeing much the same with it off.

chromejs10• 4 years ago

I keep getting `No module named 'ldm'` after I run `python scripts/dream.py --full_precision`. I've confirmed 'ldm' is activate in conda. Any idea?

abc_lisper• 4 years ago

I get the same thing too!

vesterde• 4 years ago

Try to put `sys.path.insert(0, os.getcwd())` after `import sys` in dream.py.

That fixed it for me.

totetsu2• 4 years ago

Watching pythons progress bar and waiting this long for an image, I feel like we've come full circle to the days of dail-up modems

dzink2• 4 years ago

If you have an M1 MAX with 64GB of memory you can make your images bigger. 512x512 only takes 13GB :)

nicolashahn• 4 years ago

how do you do this?

shagie• 4 years ago

As a side bit, this model appears to have difficulty with the prompt "wolf with bling walking down a street" and often generates an image that I am fairly sure is not unique and is not representative of the idea that is trying to be communicated in that text.

UniverseHacker• 4 years ago

Anyone else surprised that the results just aren't very good? I followed the instructions and it works, but just seems kinda-like deep dream level results circa 2005. Lots of blurry eyes in the wrong spots, most objects seem really blurrly and cut off. Nothing like the demo examples I've seen online. Are the default options not optimal to get the best results?

bobthebl0b2• 4 years ago

Thanks for this tutorial, I had errors and spent time to fix them and I found this script that install LStein project on M1: https://github.com/glonlas/Stable-Diffusion-Apple-Silicon-M1...

On my side this helped me to make it works. I ran it and it was installed.

mikhael28• 4 years ago

Very interestingly, this is the first true use case I have noticed where new, bleeding edge technology is much better, seemingly, on M1 than Intel GPUs.

Run Stable Diffusion on Your M1 Mac’s GPU

Related Stories