Show HN: AI OmniGen – AI Image Generator with Consistent Visuals

by lcorinst • 2 years ago

169 47 2 years ago

AI OmniGen is an advanced AI image generator, offering identity preservation for consistent subject representation and seamless image editing for refined, customized visuals.

oatsandsugar• 2 years ago

I mean, I struggle even getting Dall-E to iterate on one image without changing everything, so this is pretty cool

empath75• 2 years ago

it seems like there's a lot of potential for abuse if you can get it to generate ai images of real people reliably.

hnbad• 2 years ago

We literally already had AI fake porn of Taylor Swift making the rounds a while ago. Prepare for women in public positions to face that kind of bullshit more frequently.

CamperBob24• 2 years ago

Eh, once it's ubiquitous, nobody will care.

ed• 2 years ago

Elegant architecture, trained from scratch, excels at image editing. This looks very interesting!

From https://arxiv.org/html/2409.11340v1

Unlike popular diffusion models, OmniGen features a very concise structure, comprising only two main components: a VAE and a transformer model, without any additional encoders.
OmniGen supports arbitrarily interleaved text and image inputs as conditions to guide image generation, rather than text-only or image-only conditions.
Additionally, we incorporate several classic computer vision tasks such as human pose estimation, edge detection, and image deblurring, thereby extending the model’s capability boundaries and enhancing its proficiency in complex image generation tasks.

This enables prompts for edits like: "|image_1| Put a smile face on the note." or "The canny edge of the generated picture should look like: |image_1|"

To train a robust unified model, we construct the first large-scale unified image generation dataset X2I, which unifies various tasks into one format.

nairoz• 2 years ago

trained from scratch

Not exactly. They mention starting from the VAE from Stable Diffusion XL and the Transformer from Phi3.

Looks like these LLMs can really be used for anything

yieldcrv• 2 years ago

Pretty cool, comfy ui and community is too cumbersome for me and still results in too much throwaway content

ilaksh• 2 years ago

I think this type of capability will make a lot of image generation stuff obsolete eventually. In a year or two, 75%+ of what people do with ComfyUI workflows might be built into models.

freilanzer2• 2 years ago

Well, at the moment it seems it's not working at all.

lelandfe2• 2 years ago

I left all the defaults as is, uploaded a small image, typed in "cafe," and 15 minutes later I am still waiting on this finishing.

cubefox4• 2 years ago

Same, I left running for half an hour but nothing happened.

bob_12003• 2 years ago

The author updated their code a couple of days ago, and it runs smoothly on my end, producing results in about one minute. https://github.com/VectorSpaceLab/OmniGen

phromo• 2 years ago

On an A100 running 512x512 takes roughly 20s for one image+text input (50 iterations)

YetAnotherNick• 2 years ago

This seems to work faster: https://huggingface.co/spaces/Shitao/OmniGen

anyi09881• 2 years ago

Curious what's the actual cost for each edit? Will this infra always be reliable?

CamperBob24• 2 years ago

I was able to clone the repo and run it locally, even on a Windows machine, with only minimal Python dependency grief. Takes about a minute to create or edit an image on a 4090.

It's pretty impressive so far. Image quality isn't mind-blowing, but the multi-modal aspects are almost disturbingly powerful.

Not a lot of guardrails, either.

hyuuu• 2 years ago

could you elaborate on the multi modal aspect of this model?

KerryJones• 2 years ago

Love this idea -- you have a typo in tools "Satble Diffusion"

wwwtyro• 2 years ago

With consistent representation of characters, are we now on the precipice of a Cambrian explosion of manga/graphic novels/comics?

fullstackwife• 2 years ago

not yet, still can't generate transparent images

derefr• 2 years ago

Why do you need that? For manga specifically, generate in greyscale and convert luminance to alpha; then composite; then color.

Or, if you need solid regions that overlap and mask out other regions, then generate objects over a chroma-keyable flat background.

Vt71fcAqt7• 2 years ago

From the controlnet author:

Transparent Image Layer Diffusion using Latent Transparency

https://arxiv.org/abs/2402.17113

https://github.com/lllyasviel/sd-forge-layerdiffuse

Multicomp• 2 years ago

I sure hope so - at the very least I will use it for tabletop illustrations instead of having to describe a party's scenario result - I can give them a character-accurate image showing their success (or epic lack thereof).

haccount2• 2 years ago

I would say we already had one of those. There's more hand crafted human made content available than anyone cares to read.

While this will enable a certain degree of more spam it will more importantly, on the positive side of things, democratize the creative process to those who want to tell a story in images but lack the skill and resources to churn it out traditionally.

jowday• 2 years ago

It’s not really consistent - or anymore consistent than, say, SDXL with IP adapter. Even in their example images the character they’ve input comes out wearing different clothes.

block_dagger• 2 years ago

This looks promising. I love how you can reference uploaded images with markup - this is exactly what the field needs more of. After spending the last two weeks generating thousands of album cover images using DALL-E and being generally disappointed with the results (especially with the variations feature of DALL-E 2), I'm excited to give this a try.

1010082• 2 years ago

I am working on a API to generate avatars/profile pics based on a prompt. I tried looking for train my own model bt I think it's a titanic task and impossible to do it myself. Is my best solution use an external API and then crop the face for what was generated?

ncoronges• 2 years ago

The simplest commercial product for finetuning your own model is probably Adobe firefly, although there’s no API access support yet. But there are cheap and only slightly more involved options like Replicate or Civit.ai. Replicate has solid API support.

Check out:

https://replicate.com/blog/fine-tune-flux

1010082• 2 years ago

Is it Flux 1 possible to download and deploy to my own server? (And make a simple API on top of it?) I don't need fine tuning.

haccount2• 2 years ago

You can use a few controlnet templates and then whatever model you like and consistently get the posture correct. The diffusion plugin for Krita is a great playground for exploring this.

bob_12003• 2 years ago

https://github.com/VectorSpaceLab/OmniGen

littlestymaar• 2 years ago

Cool they even released the weights![1] didn't expect that from the tone of the release post to be honest.

[1] https://huggingface.co/Shitao/OmniGen-v1

sswz• 2 years ago

Using a single model to unify all image generation tasks, including many computer vision tasks and visual language reasoning, could transform future image generation models. Although some capabilities, like text-to-image, aren't perfect, it's a significant advancement. The model's ability to integrate so many tasks with strong instruction-following skills is impressive. I'm excited about the broad impact OmniGen could have on future research.

gremlinsinc• 2 years ago

Anyone know how it handles Text? That's kind of my deal breaker, I like Ideogram for it's ability to do really cool fonts, etc.

Show HN: AI OmniGen – AI Image Generator with Consistent Visuals

Related Stories