Visual Anagrams: Generating optical illusions with diffusion models

https://dangeng.github.io/visual_anagrams/

by beefman • 3 years ago

826 71 3 years ago

dangeng.github.io

minimaxir5• 3 years ago

Note that this technique and its results are unrelated to the infamous "spiral" ControlNet images a couple months back: https://arstechnica.com/information-technology/2023/09/dream...

Per the code, the technique is based off of DeepFloyd-IF, which is not as easy to run as a Stable Diffusion variant.

SamBam• 3 years ago

I missed it, what was infamous about it?

minimaxir5• 3 years ago

It created a backlash because a) it was too popular with AI people hyping "THIS CHANGES EVERYTHING!" and people were posting low-effort transformations to the point that it got saturated and b) non-AI people were "tricked" into thinking it was a clever trick with real art since ControlNet is not ubiquitous outside the AI-sphere, and they got mad.

Der_Einzige• 3 years ago

I always thought it was weird that this idea took off with that particular controlnet model. Many other controlnet models when combined with those same images produce excellent and striking results.

The ecosystem around Stable Diffusion in general is so massive.

minimaxir5• 3 years ago

Other ControlNet adapters either preserve the high-level shape not enough or preserve it too well, IMO. Canny/Depth ControlNet generations are less of an illusion.

ShamelessC• 3 years ago

Per the code, the technique is based off of DeepFloyd-IF, which is not as easy to run as a Stable Diffusion variant.

I haven't dug in yet, but it _should_ be possible to use their ideas in other diffusion networks? It may be a non-trivial change to the code provided though. Happy to be corrected of course.

minimaxir5• 3 years ago

I suspect the trick only works because DeepFloyd-IF operates in pixel space while other diffusion models operate in the latent space.

Our method uses DeepFloyd IF, a pixel-based diffusion model. We do not use Stable Diffusion because latent diffusion models cause artifacts in illusions (see our paper for more details).

tpudlik• 3 years ago

Did you mean to say it's _related_? The original "spiral" image by Ugleh is explicitly credited in the "Related Links" section.

minimaxir5• 3 years ago

It’s a similar topic which is why they credit it but the mechanism is much different.

mg2• 3 years ago

I had a similar idea early last year and also dabbled with a checkerboard approach.

Here a cat is made from 9 paintings of cats in the style of popular painters:

https://twitter.com/marekgibney/status/1521500594577584141

You might have to squint your eyes to see it.

I made a few of them and then somehow lost interest.

hammock3• 3 years ago

That's really cool. Can you do 3x3x3? As in, 9x9 with 81 1-cell cats, 9 9-cell cats and 1 81-cell cat?

mg2• 3 years ago

That could be interesting. A recursive cat, so to say.

The problem would be this: In the picture at hand, the big cat is rather simple. Just a portrait of a smiling cat. While the 9 smaller cats are doing all kinds of poses to adjust to the form of the big cat portrait. So the subcats are more complex than the main cat.

When doing the recursive cat, it would be hard to make a subcat from 9 subsubcats because the subcat is already a complex image that is not as easy to recognise as the main cat.

rob742• 3 years ago

That looks more like a cat-aclysm to me TBH. Probably the model was overwhelmed by the conflicting requirements, so that neither the individual images nor the composite image are particularly good. But, as you wrote, maybe they will get better at this eventually...

jamilton2• 3 years ago

I really like the man/woman inversion.

I wonder how many permutations could legibly be generated in a single image with an extended version of the same technique. I don't understand the math, but would two orthogonal transformations in sequence still be an orthogonal transformation and thus work?

xanderlewis2• 3 years ago

I’m not sure whether ‘orthogonal transformations’ in this context refers to the usual orthogonal linear transformations (/matrices), but if so then yes.

mkl3• 3 years ago

The article explicitly specifies orthogonal matrices.

hombre_fatal• 3 years ago

The man/woman one stuck out to me as well. I probably watched it ten times. Probably because it seems so forlorn.

kurthr2• 3 years ago

The mosaics of a duck and a rabbit, however, was hilarious.

moritzwarhier• 3 years ago

This is wonderful.

willsmith725• 3 years ago

This colab notebook requires a high RAM and V100 GPU runtime, available through Colab Pro.

That's sad, I would've loved to try it.

andybak2• 3 years ago

well - chuck $10 at it and spend the rest of your month trying other things.

(Back in Disco Diffusion days I was happy to spend money on Colab Pro. It was fun)

nomel• 3 years ago

I completely disagree. It's fantastic that we can get access to this hardware for so cheap. A used V100 is $1300. You could pay for Colab Pro for 10 years with that, which will get you faster and faster hardware through the years. Where I am, a month is the cost of two bags of chips.

DonHopkins6• 3 years ago

Have you never put quarters into a PacMan machine?

willsmith725• 3 years ago

i take cars for test drives before buying them

hammock3• 3 years ago

Do real-life jigsaw puzzles like the ones shown here, exist for purchase?

shanedrgn• 3 years ago

You could always make it yourself! Not sure how well the method above would scale up though https://www.createjigsawpuzzles.com/

mkl3• 3 years ago

This research uses DeepFloyd IF, which forbids commercial use. They'd need to find/train another suitable image generator.

bertil4• 3 years ago

I’m curious if they could ask for permission from the original authors (who doesn’t love a fun puzzle?—and it’s not like the profit motive here is alarming): most licenses are default permission.

You can always reach out and ask for a one-off in good faith.

hammock3• 3 years ago

I’m curious how they even thought of the idea to train a jigsaw puzzle like that in the first place. My naive guess was that those types of puzzles were preexisting. If in fact it’s a novel type of puzzle, that idea in itself is as cool as the generator they created!

karencarits• 3 years ago

Maybe a naive question, but if AI generated images are not copyrightable, wouldn't it be possible for some people to use it for research and for other people to re-use the results commercially?

cloudyporpoise• 3 years ago

This may be one of the cooler things i've ever seen

adkaplan• 3 years ago

some of these style illusions I've seen drawn by hand before, but the lithopane ones are new to me. I'm sure the 3d printing lithopane community will love them

Nition• 3 years ago

The duck/rabbit that rearranges would be really cool to use on one of those sliding puzzles. Two valid solutions!

kurthr2• 3 years ago

With that many rearrangeable elements, you could make so many different "valid" solutions, indistinguishable without a photograph, that it would become art rather than a puzzle.

bertil4• 3 years ago

I’d need to check, but if one set of “ear and hole” can be swapped with another set, both sets have to be identical in shape and color. But if they split and attach to other edges rather than swap, that creates further connection.

If you think of the edges as nodes in a connected di-graph of ears and holes, possible pairs are connected: a swap is a two-pair cluster; further connection is a four-element chain with both ends open-ended. If that connection ties to more pairs, you might have a larger cluster of identical hears and holes. Given graph properties, that’s presumably most of them — see the prisoners paradox for why [0].

That would make the puzzle much more challenging to solve if most ears fit in most holes.

[0] The excellent Matt Parker https://www.youtube.com/watch?v=a1DUUnhk3uE but I recommend the following debate with Derek from Veritasium.

chrisweekly• 3 years ago

I really enjoy these. Great post.

cwkoss• 3 years ago

Would be cool to make some of these that look like different things under red/blue light.

DonHopkins6• 3 years ago

From the HN "Boustrophedon" discussion:

https://news.ycombinator.com/item?id=15539373

https://en.wikipedia.org/wiki/Boustrophedon

https://news.ycombinator.com/item?id=15547162

DonHopkins on Oct 25, 2017 | prev | next [–]

Scott Kim has a wonderful talent at designing "ambigrams". Check out his classic book "Inversions" and his gallery of more recent work!

http://www.scottkim.com.previewc40.carrierzone.com/inversion...

An inversion is a word or name written so it reads in more than one way. For instance, the word Inversions above is my name upside down. Douglas Hofstadter coined ambigram as the generic word for inversions. I drew my first inversion in 1975 in an art class, wrote a book called Inversions in 1981, and am now doing animated inversions.

A Scott Kim Ambigram for "George Hart":

https://www.georgehart.com/scott-kim.html

John Maeda's Blog: Scott Kim’s Ambigrams

https://maeda.pm/2017/12/17/scott-kims-ambigrams/

The Inversions of Scott Kim:

https://www.anopticalillusion.com/2012/04/the-inversions-of-...

Channel: An Optical Illusion » scott kim:

https://optical397.rssing.com/chan-26600952/index-latest.php

Scott Kim’s symmetrical alphabet:

https://stancarey.wordpress.com/2012/10/18/scott-kims-symmet...

Typography Two Ways: Calligraphy With a Twist

https://www.wired.com/2009/05/pl-arts-6/

mdonahoe• 3 years ago

The man/woman color inversion one was the most impressive to me. On the rotations, I can rotate in my mind and see the other view… but I find it very hard to color invert mentally

gitgud• 3 years ago

That is amazing, here's the link for anyone interested (there's a lot of images on that page)

https://dangeng.github.io/visual_anagrams/static/videos/grid...

usrusr• 3 years ago

For me it's the reverse: the color inversions feel hardly more impressive than the morph animations that were all the rage in the 1990, because while I certainly understand how straight-forward color inversion is on the level of pixel data, I still can't "see" that simplicity. It hardly looks any different than an alpha blend with no relation at all.

The rotations on the other hand, wow! It is perfectly visible how the pixels don't change. You can physically rotate the screen and the image "changes". I could not think of a better illustration of how diffusion model images are not just echoes of preexisting images (they certainly are), but solutions to the problem of "find a set of pixels that will match the description of {prompt}". Or in this case, "that will match {A} when oriented this way and {B} when oriented that way".

jamilton2• 3 years ago

I can see the woman when the man is shown if I look for it, but not vice versa, for whatever reason.

IIAOPSW2• 3 years ago

I feel like a neural network is probably overkill for this task and a suboptimal substitute for a theoretical understanding of optical illusions, but can't argue with results.

bertil4• 3 years ago

Most of them are not “illusions” where you perceive two identical segments being different lengths because of tricks of human perception, they are ambigrams. They rely on humans’ ability to think of any three dots as two eyes and a mouth.

They also “copy” the way those networks seem to do so often that they somehow get copyright strikes; they were either prompted on existing solutions or learned them whole through training:

* The penguin and giraffe one is a previously known ambigram, for example.

* The old lady turning into a dress is obviously based on a classic pencil drawing where a similar old lady hiding in her collar turns into a young lady looking behind her shoulder [0]; however, the network interpreted “young lady” and turned into a white dress because color-matching the two different body parts from the pencil outline and turning it photorealistic wouldn’t have been much harder otherwise. There are photorealistic interpretations, though [1].

I’m more impressed by the radically new ones, like the fire flipping into a face—but most of those rely on having two distinct parts of the image be meaningful in their own context, and not relevant otherwise.

The black-and-white inversion man/woman is impressive because the two interpretations are not on separate parts of the image. That’s where you can interpret the quality of the effect as the model having learned how humans perceive and pay attention to dark and light contrasts differently. That one captures an understanding of perception.

[0] https://www.reddit.com/r/RedditDayOf/comments/35cjn5/the_cla...

[1] https://www.jagranjosh.com/general-knowledge/optical-illusio...

IIAOPSW2• 3 years ago

I can do you even better. I've made an entire game [1] based off of multistable perception [2]. Sugihara out did me by finding optical illusions with triple interpretations [3]. A solid half of MC Escher's work was about the study of tiling wherein both the negative space and the object space could be interchanged [4].

These things aren't a mystery. There are principles you can work from to produce such multi-stable illusions in formulaic, computer generated ways without resorting to the technical debt of a neural net. But, as with so much in modern times, training a neural net gets results faster than distilling a true understanding and then translating your understanding into code.

[1] http://playinverse.com

[2] https://en.wikipedia.org/wiki/Multistable_perception

[3] https://www.isc.meiji.ac.jp/~kokichis/triplyambiguousobjects...

[4] https://i.pinimg.com/originals/6d/07/bd/6d07bd12d34f674ca4de...

dwighttk2• 3 years ago

Every single one of the examples is like "yeah... I mean, I guess... sorta"

the penguin/giraffe is probably the best one. The old lady/dress barely looks like either.

bertil4• 3 years ago

Those two are based on previously known ambigraphs:

* very closely https://www.pinterest.com/pin/giraffepenguin--13398215764267...

* or directly inspired by, but the “young lady” prompt triggered the model to pick a dress, and there’s no way to make an eye and an ear or a month and a chocker photo-realistically identical: https://www.reddit.com/r/RedditDayOf/comments/35cjn5/the_cla...

ItsMattyG• 3 years ago

oh hmm, the penguin/giraffe one when I first saw it I was like "that looks like an upside down penguin, where's the giraffe?" Whereas others I immediately saw what it was trying to be.

dwighttk2• 3 years ago

I probably saw it as a right side up penguin first.

(I just went and looked on my phone and they are all more effective when they are tiny squares instead of blown up on a big screen.)

aunwick• 3 years ago

So, Im grad school I had access to an sgi onyx and basically did this but didn't toot my horn about it because. 1. I didn't think it was particularly amazing 2. We didn't have social platforms yet.

Congratulations!

mkl3• 3 years ago

An SGI Onyx has a tiny fraction of the computing power needed to run text-to-image generative models like this.

DonHopkins6• 3 years ago

How do you pay for all that electricity?

kevinwang• 3 years ago

Wow, these examples are amazing

guybedo• 3 years ago

the explosion in creativity brought by generative AI truly is incredible.

rob742• 3 years ago

As usual with AI-generated artwork: looks nice at first sight, but if you look closer, you can't help but notice the flaws. E.g. the ambigrams: in the "happy"/"holiday" one, the second word is actually missing the "i", and the two "blessing"s are really hard to read. Also, the "campfire man"'s face seems to be melting in a very disconcerting way...

belugacat• 3 years ago

I'm a photographer, and for years I've been pixel peeping at photos taken on phones with "portrait mode"; many years after the first introduction of the feature, regardless of the implementation, results still look crummy to my eye.

Looking at fine elements like hairs (nevermind curly hair) is a disaster, especially when you're used to fine classic german/japanese optics that accurately reproduce every subtle detail of a subject while having extremely aesthetically pleasing sharpness falloff/bokeh.

I've had to swallow the pill though: No one (end users; pros are another story) cares about those details. People just want something that vaguely looks good in the immediate moment, and then it's on to the next thing.

I suspect it'll remain the same for AI generated visuals; a sharp eye will always be able to tell, but it won't really matter for consumption by the masses (where the money is).

yeldarb• 3 years ago

I'd love one of these on my wall. Imagining a framed version of the Einstein pop-art one where the circle in the middle rotates (either periodically or via a manual lever).

rafabulsing• 3 years ago

The color inversion ones would work well with an E-Ink display.

bottlepalm• 3 years ago

This is a 'wow' for me. Super human even. I couldn't make these if I tried.

Visual Anagrams: Generating optical illusions with diffusion models

Related Stories