Illustrating Gutenberg library using Stable Diffusion

https://storybooks.ai/

by Aedraie • 4 years ago

149 54 4 years ago

storybooks.ai

We are illlustrating existing books using stable diffusion and other ML models.

We are currently on our quest to illustrate the Project Gutenberg library.

This Show HN is really early in our journey and we are happy to receive your feedback!

naillo• 4 years ago

Is this really using stable diffusion? Has a really 'midjourney' type style to it.

Aedraie• 4 years ago

Good eyes! We currently pipe text input to Stable Diffusion, Dall-E and Midjourney at the same time and continue with the most pleasing result. The (very small compared to what we want to achieve) content we currently have required hundreds of renderings and Midjourney came out as most pleasing nearly all of the times. From what's on the website only the Sherlock Holmes renderings have been produced using Stable Diffusion. Nothing published so far from Dall-E, which most of the times is showing us uncanny valley. In terms of preference: We are experimenting hard to achieve better results using Stable Diffusion.

fsloth• 4 years ago

With this effort it sounds it would be faster to actually draw the frames yourself. Get exactly what you want and so on…

poisonarena• 4 years ago

If the majority of the models were produced with Midjourney, why did you title it as SD as u did?

samstave4• 4 years ago

>what we want to achieve

What is it, SPECIFICALLY, you want to achieve.

And, assuming you do achieve THING, what are the negative consequences of such achievement?

How will you be accountable for dark patterns?

What recourse would any biologically physical Human Being have against [THE BAD THING] that may ensue from said achievements...

SERIOUS QUESTION

Tenobrus• 4 years ago

I find stable diffusion requires a lot more prompt engineering than either midjourney or dalle, but has a higher ceiling of quality if you do put in that effort. I wouldn't expect just using exactly the same prompt for all three models would work very well, need at least a little tuning per-model.

cercatrova• 4 years ago

Midjourney now used SD underneath

frameset• 4 years ago

If I was a commercial artist doing illustration work for articles or the like, I'd be getting worried about now.

iamflimflam1• 4 years ago

I’ve been using midjourney to add a bunch of images to a conference talk I’m doing. It’s great for “ambience” type pictures, but hopeless if you actually want to represent anything concrete.

rahimnathwani• 4 years ago

You can use Stable Diffusion to make more specific compositions, but you usually need multiple steps and a little bit of manual editing.

reb• 4 years ago

It's definitely possible to get concrete representation out of DALL•E, although it takes some experimentation to achieve realism within the constraints of the tech today. Still, we're not far off at all from indistinguishable synth imagery.

Some generated work in a photographic style: — https://www.hearthe.art/ — https://objkt.com/profile/heartheart/created

odessacubbage• 4 years ago

i think people underestimate just how many commercial images effectively fall within the 'ambience' category. i'm really convinced that in the mid to longterm, the only viable future for human artists will be a matter of GANs being subjected to the same sort of social stigma that is attached to other forms of highly economically lucrative, but morally dubious practices like child labour. ultimately sustaining a living in art has always been a matter of convincing the layman that what we produce has value and this will be the same at scale. if society does not value human-made art, there wont be any.

egypturnash• 4 years ago

I am a professional artist and I am worried.

dehrmann2• 4 years ago

It feels like self-driving cars and the 80/20 rule; it'll be impressively good but not ready for prime time for a decade.

trention• 4 years ago

Self-driving cars have already been not ready for prime time for 15 years and they have at least 5 more years in this state.

usrusr• 4 years ago

But self-driving cars have a serious good enough threshold that isn't remotely as life-or-death for illustration

barrysteve• 4 years ago

Playin devil's avocado here. SD sucks at depth. It makes 3D images which should have well set perspective, look flat and uninspired. Tastes will change and people will want to move away from the "tiny rooms, flat images and only-faces" trend that's dominating art at the moment. SD will ride the wave.

Until tastes change and after the current youth generation driving the demand for this washes out, it'll probably suck to be an artist without a strong brand.

temp_account_322• 4 years ago

I feel like the next killer feature of these models will be when they can retain contextual visuals between different scenes being generated

As in, the same mouse in four different pictures in different situations, not four different mice styled in the same way over four pictures

karencarits2• 4 years ago

Textual inversion for stable diffusion, which is related, was discussed a few days ago: https://news.ycombinator.com/item?id=32643564

temp_account_322• 4 years ago

This is nice but still has some huge limitations, e.g. you need 3-5 images of something to train it. It becomes a chicken and egg problem if I start by generating something, since I cannot create 5 different images of it to train the textual inversion. It is also intensive and slow to train on each object.

Also I think we're running into limitations of using a single prompt to describe things, as it's often hard to separate the 'what' is in the scene and the 'how' to draw the scene, with many prompts looking like a giant word salad trying to tap in the internal secret language of the model

dr_dshiv2• 4 years ago

Yeah, this seems to be the key challenge right now. Some how an image and its elements might be used as context to condition the subsequent generations.

dr_dshiv2• 4 years ago

Oh, looks like Google solved this last week with dream booth: https://arxiv.org/pdf/2208.12242.pdf

hirundo• 4 years ago

The same tool can also be used to evolve an image by generating variations and selecting a few to reproduce. You could breed and cross the results at will. You could use multiple preference models for the fitness test and evolve them afk.

An "Evolve" button could become a standard feature on content editors across media.

lifeisstillgood• 4 years ago

There is something of the meta-humanity going on here. These illustrations are drawn from hundreds of years of human effort, and are being used to enhance our enjoyment of these stories. But it is only possible because of the prior human work.

It's a little like copying the work of hundreds of other artists at once.

Index funds bother me in the same way as in at some point the price discovery mechanism ought to break as everyone just uses one corpus.

All in all, this is an amazing idea, enabled by the work of amazing people, and it is like looking at something significant - I just don't know what :-)

hydrolox• 4 years ago

can you explain the index funds line

nuclearnice32• 4 years ago

I think the notion is the index fund is simply buying a stock because it has a certain market cap. It is not evaluating the fundamentals of the company.

The algorithmic art is similar in the sense that it is an averaging of previous effort. People made art and it was fed into the algorithm. The algorithm is some weighting of the art it was fed. Analogously, people who thought about companies bought and sold the stock. It landed at some price. The index fund just followed that.

There has been criticism of index funds [1] and defense [2] as far as price discovery.

Other people have raised the possibility that an index fund holding large amounts of competitor companies is an antitrust issue. [3]

[1] https://www.ft.com/content/87a3b117-7085-40f4-8cca-66bb8353c... [2] https://www.blackrock.com/corporate/literature/whitepaper/po... [3] https://corpgov.law.harvard.edu/2018/12/04/why-common-owners...

bheadmaster• 4 years ago

Human learning itself is a little like copying the work of hundreds of other artists at once.

whycombinetor• 4 years ago

Index funds hold underlying assets under management... You are paying for a slice of the underlying assets... All slices of the pie are accounted for and it doesn't create anything new. Not sure how that relates to machines reproducing human creative efforts.

dehrmann2• 4 years ago

The gripe with index funds is that they wouldn't work if everyone used them.

zerocrates• 4 years ago

They don't mean that it's creating anything but just that index funds depend on "real" market participants to buy and sell on good and bad news, forecasts, etc. If you imagine them expanding to where there are no (or almost no) real buyers and sellers, then they don't really work, and also the market doesn't really work. Though I think you'd have to get to a very (maybe impossibly) high share of passive investors for this to really occur.

That's their analogy to ML/AI art: imagine it takes over and puts most artists out of business... are the models just stuck, taking their own output as input and not really improving? Similarly here it's maybe not possible for this to happen to such an extreme degree that the imagined problem would come to pass.

samstave4• 4 years ago

There is a deeper question here ;;

At what point will we doubt reality and truth in favor of the AI provisioned one?

We talk about singularity being an AI that surpasses human int... but its goig to happen much sooner and via a diff vector, such as AI images.

the fucking speech by biden the other day looked like a deep fak to me - his mouth flapping was way exaggerated, and the fact that he could hold a salient sentence were the tells...

singularity isnt about AI being able to fool us that hte __*AI*__ is a human - its that we FOOL OURSELVES...

So I think that the whole AI art thing is going to make reality so elastic in our minds that it will be likea dream, and we litereally cannot trust a single fucking thing online. period.

noworriesnate• 4 years ago

Along the same lines, I was thinking this morning that we might soon have revenge-as-a-service where you can generate fake, "incriminating" news stories at the click of a button. Did Joe leave a lot of hate behind when he quit his job? An unscrupulous boss can generate fake news stories on random news sites (just plug in the locality name, e.g. "cnnbostonarchives.org"). Future potential employers will be shocked to learn that Joe raped and massacred 300 puppies, with lots of realistic pictures of Joe in line with the story. Most people won't have stories like this generated about them, only the ones who have enough haters.

spyder4• 4 years ago

Yea, same thing happens after reading the subreddit where all posts and comments are generated by GPT-2: https://www.reddit.com/r/SubSimulatorGPT2/

If you read it too much, and go back to normal human forums you will wonder if they're also generated by AI or if we are all just AI generated bots on the internet :-O

13of40• 4 years ago

I wonder how far you could go generating tons of images from human art "feedstock", having people select the best 0.1% of them, then using them as input for another round, and so on. Then replace the people with another AI whose job it is to grade aesthetics of an image somehow, for a fully closed loop.

Edit: In other words...

1. Train your stable diffusion model using a corpus of normal human images.

2. Use the model to generate thousands of images.

3. Use human analysts to train another AI how to find "good" images in that set.

4. Generate ten million stable diffusion images.

5. Use the other AI to find the "good" ones.

6. Use those to train another stable diffusion model.

7. Goto 4.

(You lose some information each iteration, though, so it would probably eventually converge to grey goo or millions of pictures of Agent Smith.)

spyder4• 4 years ago

The creators of stable diffusion and others are already doing that.

https://paperswithcode.com/dataset/simulacra-aesthetic-capti...

https://github.com/LAION-AI/laion-datasets/blob/main/laion-a...

nuclearnice32• 4 years ago

I like it. Your methodology seems analogous to a GAN [1]

https://en.wikipedia.org/wiki/Generative_adversarial_network

IAmGraydon• 4 years ago

Index funds bother me in the same way as in at some point the price discovery mechanism ought to break as everyone just uses one corpus.

I'm not sure what you mean here. Are you saying that index funds are driving individual equities? If so, why do you think that? Because of broad market correlation?

icambron• 4 years ago

The idea here is that if everyone is just holding index funds that follow the market, no one is responding to external signals to drive prices to any particular value. Index funds follow the market, but what does the market follow if all the equities are held by index funds?

It’s a great metaphor for what could happen if there’s no new art coming from outside the model; everything get stuck (though I’m not sure I agree, tbh, humans are just pattern-matching robots too and it seems to have worked ok)

spyder4• 4 years ago

Almost all modern thing is "from hundreds of years of human effort" and "only possible because of the prior human work", even humans themselves.

astrange• 4 years ago

Midjourney isn't very good for this (eg with https://storybooks.ai/sb/the-lion-and-the-mouse/) because it only has one style it forces everything into. Anything else you make it just going to look like that.

Though it's not as bad if you turn down stylization to "light" or do your own postprocessing.

ncr100• 4 years ago

Excellent steps, kind of tragic to lose humanity bit by bit.

Edit: I recognize the images are curated. And that we're moving in a direction.

The reality of one premise of the book The Diamond Age is closing in ... https://en.m.wikipedia.org/wiki/The_Diamond_Age#Plot_summary the Young Lady's Primer was an AI that manifest in this book, to help the child that reads the book to become .. mentally strong, I don't fully recall. The book would converse with the child and tell appropriate stories.

So I suppose we're really close (10 years) to this, needing to try and see if AI can predict the "next step" in a child's psychological growth, and somehow observe their status.

We kind of do part of this profiling with Machine Learning and our advertising prediction capacities.

Illustrating Gutenberg library using Stable Diffusion

Related Stories