DeepFloyd IF: open-source text-to-image model

https://github.com/deep-floyd/IF

by ea016 • 3 years ago

259 226 3 years ago

github.com

TheBlapse2• 3 years ago

"Imagen free"

TheBlapse2• 3 years ago

Currently down on hugging face

zimpenfish• 3 years ago

16GB VRAM minimum is a bit steep. Sadly excludes my 3080 which is annoying because I'd like something better than Stable Diffusion locally.

TaylorAlexander2• 3 years ago

If you don't mind the power consumption I noticed that older nvidia P6000's (24GB) are pretty cheap on ebay! My 16GB P5000 is pretty handy for this stuff.

coolspot2• 3 years ago

Looks like P6000 24Gb goes for $800-$1200 while you can get superior 3090 24Gb for $800-$1000 .

NBJack• 3 years ago

An M40 24GB is less than $200, if you don't mind the trouble to get it's drivers installed, cooled, etc. It's also important to note your motherboard must support larger VRAM addressing; many older chipsets won't be able to boot with it (i.e. some, perhaps almost all, Zen 1 supporters).

specproc• 3 years ago

There's a note which suggests you might be able to get by on lower. My 3060 struggles with SD on the defaults, but works fine with float16.

There are multiple ways to speed up the inference time and lower the memory consumption even more with diffusers. To do so, please have a look at the Diffusers docs:

        Optimizing for inference time [1]
        Optimizing for low memory during inference [2]

[1] https://huggingface.co/docs/diffusers/api/pipelines/if#optim...

[2] https://huggingface.co/docs/diffusers/api/pipelines/if#optim...

thewataccount3• 3 years ago

Once these are quantized (I assume they can be), they should be ~1/4th the size.

Can anyone explain why it needs so much ram in the first place though? 4.3B is only ~9GB at 16bit (I'm not as familiar with image models).

I'm really happy to see that fits under 24GB - that's what I consider the limit for being able to run on "consumer hardware".

GaggiX9• 3 years ago

Can anyone explain why it needs so much ram in the first place though?

The T5-XXL text encoder is really large, also we do not quantize the UNets, the UNet outputs 8-bit pixels, so quantizing the UNet to that precision will create pretty bad outputs.

SekstiNi• 3 years ago

They took down the blogpost, but from what I remember the model is composite and consists of a text encoder as well as 3 "stages":

1. (11B) T5-XXL text encoder [1]

2. (4.3B) Stage 1 UNet

3. (1.3B) Stage 2 upscaler (64x64 -> 256x256)

4. (?B) Stage 3 upscaler (256x256 -> 1024x1024)

Resolution numbers could be off though. Also the third stage can apparently use the existing stable diffusion x4, or a new upscaler that they aren't releasing yet (ever?).

Once these are quantized (I assume they can be)

Based on the success of LLaMA 4bit quantization, I believe the text encoder could be. As for the other modules, I'm not sure.

edit: the text encoder is 11B, not 4.5B as I initially wrote.

[1] https://huggingface.co/google/t5-v1_1-xxl

dragonwriter16• 3 years ago

IIRC SD’s first party code had singificantly higher requirements than is possible with some of the downstream optimization most people are using, which have lower speed and more system RAM use but reduce peak VRAM use.

The architecture here looks different, but the code is licensed in a way which still makes downstream optimization and redistribution possible, so maybe there will be something there.

lalaithion• 3 years ago

Has anyone tried the Scott Alexander AI bet prompts?

1. A stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth

2. An oil painting of a man in a factory looking at a cat wearing a top hat

3. A digital art picture of a child riding a llama with a bell on its tail through a desert

4. A 3D render of an astronaut in space holding a fox wearing lipstick

5. Pixel art of a farmer in a cathedral holding a red basketball

swyx• 3 years ago

where are these prompts from?

raddles• 3 years ago

Scott Alexander made a bet with those prompts here: https://astralcodexten.substack.com/p/a-guide-to-asking-robo...

And followed up with this article when he won the bet: https://astralcodexten.substack.com/p/i-won-my-three-year-ai...

epivosism12• 3 years ago

Yes, I tried them here on an earlier version of IF: https://twitter.com/eb_french/status/1618354180577714176

epivosism12• 3 years ago

I thought it was pretty definitive at the time, but when you look really closely (as Scott's opponent is likely to do), it didn't seem like a clear win yet. But that was 3 months ago, and hopefully DF is even better now.

55555• 3 years ago

So this one can create perfect text in images? If true, that’s insane

GaggiX9• 3 years ago

LDM-400M was already able to generate text (predecessor of Stable Diffusion), thanks to the fact that every token in the text encoder (trained from scratch) was available in the attention layer.

flangola7• 3 years ago

thanks to the fact that every token in the text encoder (trained from scratch) was available in the attention layer.
ChatGPT explain this like I'm 5

GaggiX9• 3 years ago

interesting there are different models: https://github.com/deep-floyd/IF#-model-zoo-

I'm also very happy for the release of the two upscaler, I can use them to upscale to result of my small 64x64 DDIM models (maybe with some finetuning).

orra5• 3 years ago

Neither the source code nor the weights are open source... This is actually worse than Stability AI's previous offering, in that regard.

ilaksh• 3 years ago

They are technically open source. It's just that the model license prohibits commercial use and the code license prohibits bypassing the filters. So it's kind of worse than closed source in a way because it's like a tease. With no API apparently.

Theoretically large companies or rich people might be able to make a licensing agreement.

rgbrgb3• 3 years ago

model license prohibits commercial use

I thought that at first, but I think it only prohibits commercial use that breaks regional copyright or privacy laws.

yellowapple5• 3 years ago

They are technically open source. It's just that the model license prohibits commercial use and the code license prohibits bypassing the filters.

Your second sentence contradicts the first. Prohibiting commercial use and prohibiting modification are each in and of themselves mutually exclusive being being "technically open source" (let alone both at the same time).

jrm42• 3 years ago

I am a lawyer, and as flimsy and wishy-washy as the term "open-source" already is, I can't even fathom what is meant by "open source" here?

Are people suggesting that "look at the code but don't touch" actually fits what some people think of as open source?

connerruhl2• 3 years ago

That'll change when the full non-research release occurs... https://twitter.com/EMostaque/status/1651328161148174337

orra5• 3 years ago

That tweet is vague. Besides, it says like 'like SD', so I will be pleasantly shocked if the models are open source.

kingcharles• 3 years ago

The examples on the README are extremely compelling; the state of the art has been raised yet again.

simonw4• 3 years ago

It looks like the model on Hugging Face either hasn't been published yet or was withdrawn. I got this error in their Colab notebook:

OSError: DeepFloyd/IF-I-IF-v1.0 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models' If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.

Zetobal8• 3 years ago

You need to accept the license on the HuggingFace model card.

lerchmo• 3 years ago

it doesn't seem like they have anything published https://huggingface.co/DeepFloyd

simonw4• 3 years ago

https://huggingface.co/DeepFloyd/IF-I-IF-v1.0 is a 404 currently.

hunkins• 3 years ago

New restriction in their License suggests the software can't be modified.

"2. All persons obtaining a copy or substantial portion of the Software, a modified version of the Software (or substantial portion thereof), or a derivative work based upon this Software (or substantial portion thereof) must not delete, remove, disable, diminish, or circumvent any inference filters or inference filter mechanisms in the Software, or any portion of the Software that implements any such filters or filter mechanisms."

thewataccount3• 3 years ago

New restriction in their License suggests the software can't be modified.

It can be modified. That just says it can't be modified to bypass their filters.

GaggiX9• 3 years ago

New restriction in their License suggests the software can't be modified.

To remove filters.

"Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:"

oh_sigh• 3 years ago

You can't remove the filters per the license, but the weights will be available soon and so anyone can just reimplement this code using the weights

Jackson__2• 3 years ago

There is a similar license clause for the weights[0] as well, so I'm not sure this would apply unless you write the code and train your model from scratch.

[0] https://github.com/deep-floyd/IF/blob/main/LICENSE-MODEL#L54

ronsor4• 3 years ago

That's already been done: https://github.com/lucidrains/imagen-pytorch

RobotToaster• 3 years ago

Then by definition it isn't open source, violating points 3, 4, and 6 of the open source definition. https://opensource.org/osd/

yellowapple5• 3 years ago

Yep. It's getting really exhausting seeing projects falsely advertising themselves as "open source". Either be FOSS or don't be; don't pretend to be while using some nonsense like the BSL or whatever adhocery is in play here.

kmeisthax• 3 years ago

As someone who's largely "OK" with morality clauses in otherwise liberal AI licenses, I think we should start calling these "weights-available" models to distinguish from capital-F Free Software[1] ones.

I'm starting to get irritated by all these 'non-commercial' licensed models, though, because there is no such thing as a non-commercial license. In copyright law, merely having the work in question is considered a commercial benefit. So you need to specify every single act you think is 'non-commercial', and users of the license have to read and understand that. Even Creative Commons' NC clause only specifies one; they say that filesharing is not commercial. So it's just a fancy covenant not to sue BitTorrent users.

And then there's LLaMA, whose model weights were only ever shared privately with other researchers. Everyone using LLaMA publicly is likely pirating it. Actual weights-available or Free models already exist, such as BLOOM, Dolly, StableLM[0], Pythia, GPT-J, GPT-NeoX, and CerebrasGPT.

[0] Untuned only; the instruction-tuned models are frustratingly CC-BY-NC-SA because apparently nobody made an open dataset for instruction tuning.

[1] Insamuch as an AI model trained on copyrighted data can even be considered Free.

Taek2• 3 years ago

For anyone who doesn't know, DeepFloyd is a StableDiffusion style image model that more or less replaced CLIP with a full LLM (11b params). The result is that it is much better at responding to more complex prompts.

In theory, it is also smarter at learning from its training data.

GaggiX9• 3 years ago

StableDiffusion style

Not really, it's a cascaded diffusion model conditioned on the T5 encoder, there is nothing really in common, unless you mean that using a diffusion model is "SD style".

tmabraham• 3 years ago

It isn't like Stable Diffusion, it's more like Google's Imagen model.

dragonwriter16• 3 years ago

It isn’t like Stable Diffusion, it’s more like Google’s Imagen model.

Yeah, it looks exactly (architecturally) like Imagen.

Google would be running circles around everyone in Generative AI (maybe OpenAI would still have a better core LLM, maybe, but portfolio-wise) if they simply had the ability to cross the gap between building technologies and writing up research papers on them and actually releasing products.

connerruhl2• 3 years ago

The full release will be soon!

https://twitter.com/EMostaque/status/1651328161148174337

jacob0193• 3 years ago

Any web based front ends yet? I put together a system that runs a variety of web based open source AI image generation and editing tools on Vultr GPU instances. It spins up instances on demand, mounts an NFS filesystem with local caching and a COW layer, spawns the services, proxies the requests, and then spins down idle instances when I'm done. Would love to add this, suppose I could whip something up if none exists.

ronsor4• 3 years ago

It'll probably be in the Auto1111 WebUI within a week.

jacob0193• 3 years ago

You think? Automatic1111 is still on pytorch 1.7 and SD1.5

teelelbrit3• 3 years ago

What's your app / service called?!

jacob0193• 3 years ago

Not publicly released. I'd be happy to show you if you send me an email. See my profile.

atleastoptimal• 3 years ago

Text
Hands

good god it solves the two biggest meme issues with image models in one go. Will this be the new state of the art every other model is compared to?

Taek2• 3 years ago

There are good reasons to believe that this will be the new state of the art by a comfortable margin. Hard to know until we can actually play with it.

gwern2• 3 years ago

We already knew those were going to be solved by scale like using T5 instead of the really small bad text encoder SD used, because they were solved by Imagen etc.

astrange2• 3 years ago

There's fundamental tradeoffs, as there always will be when you're compressing things into an image model.

So, this is going to have new different issues. Since it's similar to Imagen, it probably can't handle long complex prompts as well, since they developed Parti afterward.

Here's my question: are there any image models where, if you prompt "1+1", you get an image showing "3"?

dragonwriter16• 3 years ago

So, this is going to have new different issues.

Well, yeah, its a bigger set of models (particular the language model) that takes more resources (both to train and for inference.) That’s the tradeoff.

Here’s my question: are there any image models where, if you prompt “1+1”, you get an image showing “3”?

You want a t2i model that does arithmetic in the prompt, translates to it to “text displaying the number <result>”, but, also does the arithmetic wrong?

Yeah, I don’t think that combination of features is in any existing model or, really, in any of the datasets used for evaluation, or otherwise on anyone’s roadmap.

epivosism12• 3 years ago

Example of how much better it can do compared to midjourney, on a complex prompt: https://twitter.com/eb_french/status/1623823175170805760

It is able to put people on the left/right and put the correct t-shirts and facial expressions on each one. This is compared to mj which just mixes together a soup of every word you use and plops it out into the image. Huge MJ fan of course, it's amazing, but having compositional power is another step up.

Zetobal8• 3 years ago

That's not how any of this works what do you even mean by compositional power? Every model speaks a different "language" comparing prompts like this has no merit and shows only that the person who makes the claim lacks understanding of the subject matter.

epivosism12• 3 years ago

Compositional power might mean "the image more resembles the composition you want and describe"

i.e. if you say "a red cube on a green sphere" in DeepFloyd, you will get it. If you say that in MJ, you won't. That means you have more power to compose the image you want with this tool.

circuit10• 3 years ago

So it seems like Midjourney looks better and has more realistic faces but this one is better at following the prompt and generating what you actually want

dragonwriter16• 3 years ago

Midjourney follows the “facing each other” better, but DF has readable text. This may be due to DF also suffering the effect that ISTR is common to MJ and SD that specifying conflicting things it understands leads to priority the “visibility” one (the specific text, in this case) over the “compositional” one (in this case, “facing each other”.)

vitorgrs4• 3 years ago

Midjourney always look very aesthetic pleasing, I guess because of their RLHF tuning with Discord data... But it doesn't really follow prompts as well as Dall-e for example.

But in the end, people want pretty pictures. So is a complicated situation.

BoorishBears5• 3 years ago

The tweet they shared is from February and uses an outdated version of MJ, this is what I got from V5: https://i.imgur.com/0uxtZDe.png

Midjourney does much better overall. Composition is neat, but MJ is so incredibly far ahead in terms of quality of output, it honestly doesn't matter if you have to go and do composition manually (and with new AI based tools, that's easier than ever too. Do a bad cut and paste job then infill your way back to a coherent image)

wheresmyshadow2• 3 years ago

Can't wait to see how all of this is going to look like in ten years. I know we are all nitpicking right now but these results are totally mindblowing already.

petesergeant• 3 years ago

I wonder when we can start fuzzing brains. Wire you up to a machine that measures happiness or anxiety or anger or whatever, and keep re-generating results that hone in on the given emotion.

quaintdev• 3 years ago

The thing is all these things can go downhill as well. All these things are cool today just like Google was a decade ago.

orbital-decay• 3 years ago

Spatial composition can be done easily, if you stop bothering with pure text-to-image (SD has several tricks and UIs to place objects precisely, they are all janky but they do work, that's practically photobashing). Attribute separation is also easily done with tricks like token bucketing, so your Indian guy will look Indian, and your East Asian guy will look East Asian. All of that is easy if you abandon the ambiguous natural language and use higher-order guidance.

What's really required is semantic composition. Making subjects meaningfully and predictably interact, or combining them together. And also the coherence of the overall stitched picture, so you don't end up with several different perspective planes.

epivosism12• 3 years ago

Here are some play markets on manifold markets tracking its release: https://manifold.markets/markets?s=relevance&f=all&q=deepflo...

35% to full release by end of month, although it may not have adjusted.

epivosism12• 3 years ago

There's a discord with tons of sample images, where we've been waiting patiently for the release, coming SOON, for 3 months now. https://discord.gg/pxewcvSvNx

CamperBob22• 3 years ago

What these AI companies need are some good old-fashioned leakers. We should be seeing these models show up on sketchy pirate sites, complete with garish 80s-style cracking screens crediting various '1337 haX0rs with witty pseudonyms.

ronsor4• 3 years ago

Well, NovelAI was hacked and had their image generation model leaked last year.

causality03• 3 years ago

Is this intended to replace Stable Diffusion? Somebody want to give the eli5?

nickthegreek• 3 years ago

Stability AI releases DeepFloyd IF, a powerful text-to-image model

Hope not. This is a worse license.

gjsman-1000• 3 years ago

This is the dumb part about open-source models. Criminals, governments, and propaganda spreaders need not worry about the license; but legitimate users do.

Jackson__2• 3 years ago

As far as I can tell from Emad's discord and twitter discussion, the idea appears to be to make this a "research" release, and therefore the worse license.

At a later point the model will be renamed "StableIf", and released with a similar license to StableDiffusion.

thejarren• 3 years ago

Second paragraph in the link:

DeepFloyd IF is a state-of-the-art text-to-image model released on a non-commercial, research-permissible license that provides an opportunity for research labs to examine and experiment with advanced text-to-image generation approaches. In line with other Stability AI models, Stability AI intends to release a DeepFloyd IF model fully open source at a future date.

mkaic11• 3 years ago

This does outperform Stable Diffusion 2.1, but uses a different architecture and requires more memory and compute. Stable Diffusion runs its denoising process in a compressed "latent space" which is how it was able to be so compute-efficient compared to other diffusion models. It also uses the (relatively) small text encoder from OpenAI's CLIP model to encode user prompts. Both of these optimizations meant that it could run much faster compared to say, DALLE or Imagen, but it didn't follow complicated user prompts especially well and had trouble with things like counting and text-rendering.

DeepFloyd IF is based on Google's Imagen model, which has two key differences from Stable Diffusion: (1) it denoises in pixel space instead of a compressed latent space, and (2) it uses a 10x larger pretrained text encoder (T5-XXL-1.1) compared to SD's CLIP encoder. (1) allows it to better render high-frequency details and text, and (2) allows it to understand complex prompts much better. These improvements come at the cost of multiple times more memory usage and compute requirements compared to SD, though.

In terms of "will it replace SD?"—in the short term I think yes. But I still think latent diffusion models are the future. For example, Stability is gearing up to release Stable Diffusion XL right now, a larger version of the original SD that does higher fidelity and higher resolution generations. I wouldn't be surprised if it takes the crown back from DeepFloyd when it releases, but I guess we'll have to see.

lucidrains• 3 years ago

tldr: bigger text encoder is better. SD will catch up quickly, as conditioning on a new set of precomputed text embeddings is a trivial change

dragonwriter16• 3 years ago

For example, Stability is gearing up to release Stable Diffusion XL right now, a larger version of the original SD that does higher fidelity and higher resolution generations. I wouldn't be surprised if it takes the crown back from DeepFloyd when it releases, but I guess we'll have to see.

SDXL is available on StabilityAI’s hosted services already, so they can be compared head to head.

mftb• 3 years ago

This was an excellent summary, ty.

2bitencryption• 3 years ago

thanks for the explanation!

denoising in latent space certainly seems like the "correct" path. My (amateur) thinking is, the more you can do in latent space, the better.

causality03• 3 years ago

Does the increased memory footprint mean it can't be run on a normal desktop like SD?

reckless• 3 years ago

Seems to be entirely a different approach for diffusion.

DeepFloyd IF works in pixel space. The diffusion is implemented on a pixel level, unlike latent diffusion models (like Stable Diffusion), where latent representations are used.

srajabi• 3 years ago

Wow this does so well on text! The original model struggled a lot, it's impressive to see how far they've come.

mkaic11• 3 years ago

I'm quite curious how much of the improvement on text rendering is from the switch to pixel-space diffusion vs. the switch to a much larger pretrained text encoder. I'm leaning towards the latter, which then raises the question of what happens when you try training Stable Diffusion with T5-XXL-1.1 as the text encoder instead of CLIP — does it gain the ability to do text well?

FL33TW00D• 3 years ago

Check out figure 4A from the ImageGen paper: https://arxiv.org/pdf/2205.11487.pdf

From that - I would strongly suspect the answer to your question to be yes.

minimaxir4• 3 years ago

DeepFloyd IF is effectively the same architecture/text encoder as Imagen (https://imagen.research.google/), although that paper doesn't hypothesize why text works out a lot better.

radq• 3 years ago

It is most likely due to the text encoder - see "Character-Aware Models Improve Visual Text Rendering". https://arxiv.org/abs/2212.10562

GaggiX9• 3 years ago

The CLIP text encoder is trained to align with the pooled image embedding (a single vector), which is why most text embeddings are not very meaningful on their own (but still convey the overall semantics of the text). With T5 every text embedding is important.

piyh• 3 years ago

That alone is a huge milestone.

simonw4• 3 years ago

It's much better, but it's not perfect. Here's what I got for:

a photograph of raccoon in the woods holding a sign that says "I will eat your trash"

https://twitter.com/simonw/status/1651994059781832704

Semonto• 3 years ago

It actually adds quite a nice charm by wording it not correctly

indymike• 3 years ago

This is not a problem. The sign was clearly made by the raccoon.

itslennysfault• 3 years ago

This could be super cool for logos. I've tried using Stable Diffusion to generate logos and it does pretty good at helping brainstorm, but the text is always gibberish so you can use its idea, but you have to add your own text which basically means creating a logo from scratch using its designs as inspiration.

alex_sf6• 3 years ago

You can't use this to make logos for any commercial product, and it's not safe to use it for hobby projects either, based on the current model license.

krisoft3• 3 years ago

You can't use this to make logos for any commercial product

Yeah good luck figuring out that a particular logo was generated with this particular model. And if someone does good luck doing anything about it.

With this amount of fear one wouldn't dare to cross a road without three layers of bubble wrap, plus written authorisation from a lawyer plus a feasibility study from a traffic engineer.

dormento• 3 years ago

You also cannot just ingest people's media to train a for-profit AI image generation service, and well, here we are.

PS: not an AI apologist, just pointing the irony. Feels like those fan sonic characters "original content do not steal".

ZoomZoomZoom• 3 years ago

Well, surely you don't expect to just take the generated stuff and use it right away? For logos you usually need something that some entity can hold the claim on, so you probably need a human touch in any case.

robbomacrae• 3 years ago

MidJourney's v5 model did a great job producing our logo https://summer.ai/static/nearby/android-chrome-192x192.png

Admittedly it was one of several hundred I had it spit out for me. But the design was completely original and caught me by surprise.

I tried making modifications but everyone kept telling me they prefer the original exactly as MidJourney made it!

dragonwriter16• 3 years ago

Authorship is not a requirement for trademark, though having copyright as well is nice.

phonescreen_man• 3 years ago

Try controlNet with stable diffusion, I’ve been having good results for text.

minimaxir4• 3 years ago

GitHub: https://github.com/deep-floyd/IF

Colab Notebook for running the model based on the diffusers library: https://colab.research.google.com/github/huggingface/noteboo...

Hugging Face Space for testing the model: https://huggingface.co/spaces/DeepFloyd/IF

Note that the model is substantially more compute-intensive than Stable Diffusion, so it may be slower even though that space is running on an A100.

mkaic11• 3 years ago

HF also wrote a blog post on how you can mess around with the model in a python notebook using their excellent Diffusers library: https://huggingface.co/blog/if

minimaxir4• 3 years ago

I knew the model would have difficulty fitting into a 16GB VRAM GPU, but "you need to load and unload parts of the model pipeline to/from the GPU" is not a workaround I expected.

At that point it's probably better to write a guide on how to set up a VM with a A100 easily instead of trying to fit it into a Colab GPU.

bwesz• 3 years ago

I tried the HF Space and it generates images of 64x64 resolution, which are basically useless.

ollin• 3 years ago

It generates 64x64 for the first stage, but there's a button to upscale your favorite 64x64 image to usable resolution.

GaggiX9• 3 years ago

There is a big "Upscale" button.

ShamelessC• 3 years ago

Note that the model is substantially more compute-intensive than Stable Diffusion, so it may be slower even though that space is running on an A100.

Any more specifics on this? Sampling has gotten better too. My primary concern is the amount of memory necessary for a generation (batch size = 1, or I guess "2" using classifier free guidance).

Similar to OpenAI's cascaded diffusion models and GLIDE, you can presumably run the models in sequence, unloading earlier models from memory to make room for the 2nd and 3rd stage models.

Right now, I really only need 256px resolution. So, will I be able to fit the first stage (64px) model in memory on its own with my 12 GB RTX 3060? What about the 2nd stage (64px -> 256px)? 3rd stage?

dragonwriter16• 3 years ago

They’ve said the whole pipeline (using the large models in each step with a choice, IIRC) can be run sequenced with 14GB VRAM.

mkaic11• 3 years ago

I think this model will result in a massive new wave of meme culture. AI's already seen success in memes up to this point, but the ability for readable text to be incorporated into images totally changes the game. Going to be an interesting next few months on the interwebz, that's for sure. Exciting times!

marvinkennis2• 3 years ago

Seeing a lot of text-to-image out there recently. Does anyone know what the current state of the art is on image-to-text? Thinking something similar to Midjourney's /describe command that they added in v5

mkaic11• 3 years ago

While it's not publicly available yet, I have strong suspicions that multimodal GPT-4 may actually be SOTA in image-to-text. The examples shown in the Sparks of AGI paper were extremely impressive imo, though of course those are cherry-picked so it's unclear how well the model will perform on non-cherry-picked images.

jah2422• 3 years ago

This is text + image -> text but pretty cool and still might be of interest to you:

https://llava-vl.github.io

marvinkennis2• 3 years ago

Just entering "Describe this image" in the chat prompt got me exactly what I was looking for. Thanks!

alex_sf6• 3 years ago

The current license makes this largely unusable for nearly any purpose. Really disappointing release from SAI.

edkennedy• 3 years ago

I think this is just for pre-release, and they will release fully licensed for Commercial. It doesn't make sense to have a model like this that can do game changing text and logos... but then not license it for commercial. If they don't, that would be ridiculous.

danwee• 3 years ago

I would be more interested in image-to-text models. Does someone know of any decent model? I saw the GPT4 demo, and they showed that they do image-to-text... but then that was actually a fake (i.e., the model was interpreting the image filename).

jkea• 3 years ago

MidJourney has the describe function which is kind of like that. Not sure how decent it actually is

theRealMe• 3 years ago

Can you provide a source on gpt4 image model being fake? I haven’t heard that before, though I have wondered why I haven’t heard anything about the image part and don’t have access to image processing myself.

runnerup• 3 years ago

AFAIK, converting an image to a text summary isn't really a thing by itself. The related work would be "visual reasoning" which is the ability to ask things about the image in natural language and get responses back also in natural language.

I believe the current SOTA test for NLVR is VQAv2[0] or GQA[1].

0: https://visualqa.org/ 1: https://arxiv.org/pdf/1902.09506.pdf

minimaxir4• 3 years ago

For a fast-but-less-robust model, you can use a ViT encode/GPT-2 decoder model: https://huggingface.co/nlpconnect/vit-gpt2-image-captioning

For a more-robust-but-hard-to-run model, you can use BLIP2: https://huggingface.co/Salesforce/blip2-opt-2.7b

grumbel• 3 years ago

CLIP Interrogator[1], which is also build into AUTOMATIC1111, gives quite reasonable results, at least if all you need is a prompt, it can't handle complex interactions:

Image: https://i.imgur.com/husplYZ.png

Output: "a white horse with a sign that says rexel's in space, pixelperfect, inspired by Paul Kelpe, official simpsons movie artwork, alternate album cover, in style of nanospace, by Apelles, pickles, pespective, pop surrealism, ingame, in a space cadet outfit, sifi"

[1] https://huggingface.co/spaces/pharma/CLIP-Interrogator

jah2422• 3 years ago

This is text + image -> text but pretty cool and still might be of interest to you:

https://llava-vl.github.io

etaioinshrdlu• 3 years ago

Does paying Hugggingface to run it on the GPU count as commercial use?

Thoreandan• 3 years ago

"Hi! I'm B-19-7, but to everyperson I'm called Floyd." -Planetfall (1983)

My first thought on seeing "Floyd" and "IF" together. It looks like a Pink Floyd reference from the About page on https://deepfloyd.ai/ though.

dr_kiszonka• 3 years ago

Looks like music generation is on their roadmap. Fun!

https://stability.ai/careers?gh_jid=4142190101

youssefabdelm• 3 years ago

Meh, results feel hodge podge like a bunch of models were stitched together

vitorgrs4• 3 years ago

Tried using right now, and it's way better than Stable Diffusion (be it 1.5, 2.1 or SDXL).

But is harder to get a good picture. This fine tuned with a good RLHF will be amazing.

grungegun2• 3 years ago

What does this mean? Isn't the quality of a model determined by how easy it is to get a good picture?

vitorgrs4• 3 years ago

Not necessarily. IMO a good model needs to follow your prompt well, and that was my problem with Stable Diffusion.

I've been trying to get a good portrait picture with "neon lights" on Stable Diffusion and it is almost impossible. Meanwhile with the new Dall-e, that was possible. The picture specially with SDXL is good, but it doesn't really have neon lights...

I tried now similar prompt on deepfloyd and managed to get there!

marginalia_nu• 3 years ago

Gorbachev holding meatball pasta in both hands. 1980s synth futuristic max headroom aesthetic. Neon lights.
Aristotle in ancient greek clothes. Toga. New york, rain, film noir, fog, art deco, neon lights, blade runner sci fi

Seems to be holding up recently well with the first promt. Second was only OK.

dang• 3 years ago

(via https://news.ycombinator.com/item?id=35743727, but we've merged that thread into this earlier one)

jlsreleaf2• 3 years ago

Website design main page. Bright vibrant neon colors of the rainbow slimes, slime business, kid attention grabbing, splashes of bright neon colors. Professional looking Website page, high quality resolution 8k

jlsreleaf2• 3 years ago

Website design for slime. Professional looking, high-quality, 8k, brightest neon colors of the rainbow slimes, splashes of neon colors in background, kid attention grabbing, eye catching

anirbanc88• 3 years ago

https://www.kaggle.com/code/anivana/deepfloyd-if-playground/

I played with some ready prompts here

bulbosaur123• 3 years ago

What are the official and unofficial discords?

I found only this one on their subreddit: https://discord.gg/GvsvNrVkk5

bicepjai• 3 years ago

I understand, I have a decade old 2 nvidia 1080 to card, can we infer and train IF on them ?

DeepFloyd IF: open-source text-to-image model

Related Stories