Three-eyed forehead in Stable Diffusion

https://ahrm.github.io/jekyll/update/2023/01/02/three-eyed-forehead.html

by hexomancer • 4 years ago

112 51 4 years ago

ahrm.github.io

dang• 4 years ago

Recent and related:

Remaking old computer graphics with AI image generation - https://news.ycombinator.com/item?id=34212564 - Jan 2023 (73 comments)

A_D_E_P_T2• 4 years ago

This reminds me of something a few friends and I tried a couple of months ago: No matter what prompt was used, neither Midjourney nor Dream Studio could generate an image of a man wearing a red suit jacket with a blue shirt. (We were trying for red suit + blue shirt + white tie... but even just the first two proved impossible.) Presumably the combination is so unusual as to run counter to the training data of the models. Likewise for a forehead with three eyes.

kleiba• 4 years ago

Stable Diffusion, using the prompt "red suit jacket on blue shirt":

https://imgur.com/a/bwjQLxv

Further fine-tuning would certainly improve the final result.

A_D_E_P_T2• 4 years ago

Interesting. Thanks. For some reason, at the time, it didn't work at all -- but I've double checked it right now with Midjourney & in Dream Studio, and it works now using the same old prompts. Old sticking points are being swept aside pretty fast...

ETA: They still can't seem to make it with the white tie, though!

zone411• 4 years ago

Still funny how it loves six fingers per hand

Aerroon• 4 years ago

Generate an image you like.

Open it up in GIMP, paint the shirt blue and jacket red.

Feed resulting image into img2img with a low denoising strength (0.3-0.4 maybe) with the textual prompt of what you want.

This can help you get combinations of things the AI doesn't easily do itself.

version_five• 4 years ago

An astronaut riding a horse is unusual too but it can make that.

I think the reason is it doesn't understand compositionality, meaning it doesnt have any logic or "intuition" about the relationships between the colors and items (or eyes and forehead) so it fails to constrain the way they are assembled in the image.

morsch2• 4 years ago

I tried to make Dall-E 2 draw "Santa giving a massage to a woman" (to personalize a gift certificate), and the results were bizarre. Mostly I got women giving Santa a massage, but even those had worse artifacts than usual. When I adjusted the prompt to be more explicit as to who was giving the massage and who was receiving it, the results got increasingly weird.

With some finagling, I got an acceptable image, but it was much worse than I would have expected. On the other hand, the whole thing made for a funny story to tell when I was handing over the present.

https://imgur.com/a/3UWLBeV

This is line art (because I wanted to print it in black and white), but the non-line-art style had the same issue. Also it's Saint Nicholas and not Santa, but I just confirmed that the issue still persists with Santa.

londons_explore• 4 years ago

I suspect Dall-E 2 has had some kind of 'negative training' for pornography... Ie. something beyond just filtering porn from the training set. Perhaps they put pornographic images through it with a reversed loss function?

And A woman giving santa a massage is just too similar to a bunch of porn...

BeFlatXIII• 4 years ago

Likewise for a forehead with three eyes.

No Shiva images were included in the training data?

phiy• 4 years ago

this is pretty fascinating to think about the implications. Everyone assumes that AI is going to be all powerful and scary, but these little edge cases just completely stop it in its tracks. It makes me wonder if any of the self-driving car projects have run into weird edge cases like this that which has stopped them from releasing it more seriously.

I can see many types of clothing could completely throw off the cars in a crosswalk type situation. Reflections are solved with lidar, I wonder if a sudden breeze picks up a dense cloud of dust, that the car reacts very unpredictably.

Its a possibility that insurance scams will evolve in the future to target these vehicles using adversarial techniques that for humans are obvious. How could we even stop that? It'd likely be a cat and mouse game.

Llamamoe• 4 years ago

In Midjourney this could be done using miltiprompting, and Automatic's webui supports an analogous gesture with the AND keyword.

aenvoker• 4 years ago

Making a cyclops was a long-running challenge among the users of Midjourney. Turns out multiprompting “photo of a girl:: big eye” so that the AI would attempt to make both concepts individually and simultaneously in the same image would do it.

https://www.reddit.com/r/deepdream/comments/v9jr73/how_can_y...

tiborsaas6• 4 years ago

"each inpainting took about 20 seconds which was quite annoying. But I could envision a future where generation is basically real-time, imagine navigating through possible generations using mouse wheel and tweaking the parameters and seeing the effects in real-time"

This is really funny actually, considering what basic Photoshop tools are capable of out of the box :)

orbital-decay• 4 years ago

Yeah, the entire thing reads like a case of tunnel vision. Even image-gen models are capable of much more than just cherrypicking and "tweaking the parameters", not to mention of the entire range of image manipulation tools.

Model output being defined by the training data is still a valuable observation for many unaware of the limitations, regardless of how obvious it might be for others.

fortyseven• 4 years ago

There's a lot of things I try to do in Stable Diffusion that I could do in minutes in Photoshop.

But half the fun is trying to do it entirely inside the generative art tools available. If I were really trying to make something good I'd use both together.

drmaximus• 4 years ago

Is photoshop cabale of inpainting what you tell it to inpaint?

tiborsaas6• 4 years ago

With plugins, yes, but that's not the point. Inpainting is great, but using inpainting to get rid of the masking outline is funny to me. There are healing tools in PS that are designed for this exact purpose and they work in real time :)

hexomancer4• 4 years ago

Can you show me how "basic photoshop tools" can do this in 20 seconds?: https://ahrm.github.io/images/2023-01-02-three-eyes/inpainti...

tiborsaas6• 4 years ago

It's called feather edges for selection, it creates a blur around the selection.

https://lh3.googleusercontent.com/OFIpc-iFtO1AQty-IJ6beLEGe6...

He already had a good enough lantern, all he had to do is to set a tasteful setting for this.

ramblerman• 4 years ago

I think it's amusing how completely you miss the capability of what OP did, vs what photoshop can do.

1. Generate me an eye

2. now place 3 of these on a head where I choose

3. merge it in for me, in some way that fits the overall image.

The 4th step (missing due to time it takes) iterate over a few iterations till I see one I like.

3 years from now a novice will be able to do more advanced photoediting than a pro inside photoshop right now.

tiborsaas6• 4 years ago

I fully understood what he did, but what he dreams about is already possible now in another environment in real time. I guess AI performance will catch up quickly. Meanwhile combining different programs yield the best result. I used like 3-4 tools to get a final render I was happy with when I felt the need to play with them.

It's already the case that photoediting for novices are a reality, but the level of control is lacking.

andybak• 4 years ago

Dall-E has an interesting take on the problem: https://labs.openai.com/sc/JZIuAmvnELh8cMnBsLRVo5qk

thrdbndndn• 4 years ago

Pretty cool article, I'd say the final result is kinda underwhelming, though.

hexomancer4• 4 years ago

I didn't really spend any time to make the result look good (I am not an artist, I was more interested in the technological aspect rather than the art aspect).

Kaibeezy• 4 years ago

Not really on his forehead.

Six fingers, yeah, and also car wheels are a random mess of spokes and bolts.

bitwize• 4 years ago

There's a brilliant YouTube animation of a woman who applies one of those anime AI filters to herself. At first she's happy with the results, as it made her face considerably more beautiful. Then she discovers she has a few additional fingers, and her foot has been twisted into a sort of hand/foot/shoe deformity. Then she looks outside and sees one of those AI-generated hellscapes with a black hole where the sun should be.

Best horror film of last year.

https://m.youtube.com/shorts/G9D_PL61_pA

Dylan16807• 4 years ago

I would suggest you give the link and encourage people to watch the 20 second video for themselves before you list everything that happens in it.

sophrocyne• 4 years ago

This is what Invoke's Stable Diffusion canvas solves for.

https://youtu.be/RwVGDGc6-3o

naillo• 4 years ago

I only have a 3060 laptop gpu and an im2im run like this barely takes 3 seconds. It's really fun and near real time if you keep the unet loaded in vram in between runs instead of re-loading it like calling a script would likely do.

stuaxo• 4 years ago

Disco diffusion is worth a go, it's images are much more dream like.

fortran77• 4 years ago

I’ve tried to get stable diffusion to draw 3-armed pianists, or pianists with extra fingers, and failed, probably for the same reasons this was difficult

natch• 4 years ago

Just ask it to draw two armed pianists and it will happily generate scads of three armed ones.

terminal_d• 4 years ago

This is achievable without copy/pasting eyes: If you're using the Automatic111 GUI, go to img2img -> inpaint, mask the area for one eye (on the forehead), enter prompt, and set padding = 0 and denoising accordingly (0.4 - 0.6 would be acceptable). Repeat for all three eyes. You can add practically anything to an image with inpainting, provided your prompt and padding is correct.

boxed• 4 years ago

Square wheels is another fun example of how AI art is still super bad.

TheDudeMan• 4 years ago

Eyes? I thought they were rings/piercings (in the original).

greenhearth• 4 years ago

Looks like it stole Alex Ross style.

narrator• 4 years ago

On a similar note to Stable Diffusion refusing to put 3 eyes in the middle of a sci-fi character's forehead: I have been experimenting with GPT-3 rewriting some of my sci-fi stuff. It's really funny because it right away tries to steer the plot into the most cliche sci-fi storyline and characterization possible where all the characters are perfect almost superhero like action heroes capable of incredible feats of strength and agility. My characters have a lot of flaws, and aren't impressive in an action movie sort of way so GPT-3 winds up being almost totally unusable.

dwaltrip• 4 years ago

One of the linked resources in the article is a great high-level overview of how Stable Diffusion works:

https://stable-diffusion-art.com/how-stable-diffusion-work/

It’s a quick read and I found it very helpful.

Gravityloss• 4 years ago

Very good indeed

Three-eyed forehead in Stable Diffusion

Related Stories