Ten Years of Image Synthesis

https://zentralwerkstatt.org/blog/ten-years-of-image-synthesis

by theorize_ai • 4 years ago

120 22 4 years ago

Keyframe• 4 years ago

What's next though? To paraphrase Hemingway - It all happened gradually and then suddenly.

In the arts, music, graphic design, animation, 3D models, and eventually realistic looking video seem to be the clear next steps.

The key to AI art is that while the universe is infinitely complex, the visual form of most things humans can recognize follow a collection of natural patterns. Anything humans can do can be computed, the only issue is being able to set up the scope for the patterns we seek to replicate. For games like chess, a bit easier, for images that represent label-able things, harder but still manageable. The main problem of making things beyond this is giving a valid scope to the whole problem: what patterns, precisely, do we intend to replicate?

khazhoux4• 4 years ago

My mind is blown away by Midjourney v4, but the open problems are still plentiful (even before getting into video/motion, 3d etc).

After a few dozen prompts, you quickly learn how the image generation is simultaneously amazing and dumb. The dumb part is the next 10 years of improvements.

This is like Google 1999.

ajmurmann2• 4 years ago

Just getting the number of fingers and to a lesser degree arms and legs correct would solve 80% of the issues I encounter

ansk• 4 years ago

Image generative models have made significant progress over the last few years, but an often-overlooked confounding variable is that during this period, the predominant benchmarking task used by researchers (specifically those working on large-scale image synthesis) switched from unconditional image synthesis to text-to-image synthesis. Text-to-image synthesis is a fundamentally easier task as the text-conditioned distribution is much more predictable than the unconditional counterpart. Additionally, text-to-image plays to the strengths of diffusion models, which benefit from having a conditioning signal to guide the sampling process (via classifier(-free) guidance). As someone who is observing this progress from an adjacent field, it has been hard to tease out how much of this seemingly abrupt progress can be attributed to the fact that the preferred benchmarks and datasets have been shifting behind the scenes. I suspect that if these variables were well controlled, progress over the last few years would look more like the steady, incremental improvements that occurred in the field in the years prior.

multimodal• 4 years ago

Maybe I'm misreading the tone of your comment - but is that a bad thing? Reframing the approach to a problem, to state-of-the-art results; that strikes me as nothing if not progress. Don't mean to come across as negative here -- I just don't see why anyone would control those variables.

You list reasons why text-to-image is easier than unconditional image gen - but isn't that the point? They're both under the umbrella of image generation, which is ultimately pushed forward.

GaggiX2• 4 years ago

The good old GAN models were trained on single domain dataset or on Imagenet, and these datasets only encode a small set of the real distribution, make them easier to fit, also I don't think GANs are flexible enough to be effectively used in text-to-image generation, so I don't think it's an easier task, it was a more difficult task and it required the development of autoregressive and diffusion model to actually make it feasible.

sanxiyn• 4 years ago

BigGAN trained on JFT-300M and it worked fine. In fact, JFT-300M was easier than ImageNet. In their own words:

"Interestingly, unlike models trained on ImageNet, where training tends to collapse without heavy regularization, the models trained on JFT-300M remain stable over many hundreds of thousands of iterations. This suggests that moving beyond ImageNet to larger datasets may partially alleviate GAN stability issues."

This suggests GAN will work even better on, say, LAION-5B. It's just that nobody tried.

"GANs are not flexible enough to be effectively used in text-to-image generation" is an absurd statement. GANs have latents, there is no reason why just feeding CLIP to GANs shouldn't work. Again, nobody tried because most GAN works preceded CLIP.

We should try GANs again. It was abandoned without any evidence whatsoever that diffusion model etc is required. GANs have genuine advantages over diffusion models, like much faster sampling.

theemathas3• 4 years ago

You can give a blank prompt to stable diffusion and it will give you reasonable images https://www.reddit.com/r/StableDiffusion/comments/xu0dt6/no_...

flopsamjetsam• 4 years ago

I thought this was an excellent summary for someone like me, who has an interest in the subject but only a small amount of knowledge about it. Linking to the papers was a good touch.

ElijahLynn• 4 years ago

This title might be better as "Ten Years of AI Image Synthesis" or something like that. Because my first thought was container image synthesis.

jaggirs• 4 years ago

I think that by now the term has been hijacked

hakuseki• 4 years ago

There's some older work as well. Geoffrey Hinton's team used deep belief networks in 2006 to generate "handwitten" digits: http://www.cs.toronto.edu/~hinton/digits.html

Ten Years of Image Synthesis

Related Stories