Text2Video-Zero Code and Weights Released by Picsart AI Research (12G VRAM)

https://github.com/Picsart-AI-Research/Text2Video-Zero

by ftxbro • 3 years ago

421 121 3 years ago

ftxbro2• 3 years ago

The associated paper is "Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators" at https://arxiv.org/abs/2303.13439

meghan_rain• 3 years ago

Wow, scroll to the part where stick figures are provided as guidance https://github.com/Picsart-AI-Research/Text2Video-Zero#text-... and tell me this won't upend the VFX industry!

whitemary• 3 years ago

It won't upend the VFX industry.

If a workflow cannot support iterating on every detail in isolation with psychotic levels of control, it won't even be adopted by the VFX industry.

nine_k• 3 years ago

It will help add video illustrations to ordinary presentations or talking-head videos. A market enough for some attention from, say, Adobe, and for a bunch of less expensive offerings.

echelon• 3 years ago

Once this tech matures, they won't have any work left to do.

Kids at home will be making Star Wars, not Industrial Lights and Magic.

detrites2• 3 years ago

This is exactly what it supports, and limitlessly.

An AI model doesn't get tired of receiving direction and doing retakes.

It's a matter of time.

rvz2• 3 years ago

No it won't. At least not now when we are seeing deformed artefacts in videos like this one: [0]

[0] https://old.reddit.com/r/StableDiffusion/comments/1244h2c/wi...

SV_BubbleTime• 3 years ago

Funnier than almost any network TV I’ve seen in years.

ftxbro2• 3 years ago

some call it deformed artefacts some call it meme magic

sockaddr• 3 years ago

I think there's a boob on that camel's hip.

dragonwriter• 3 years ago

It will create an entirely new low end to the industry (which may be used in prototyping workflows by the high end), but the high end is going to want both more detailed control of the avatars (I would say models, but that means different things in the colliding domains) and already does skeletal animation with many more control points.

You want to upend the VFX industry, you probably want text-to-3d-model and text-for-script-for-existing-animation-systems, that still supports either of those being finetuned manually with existing tools.

pjonesdotca• 3 years ago

First, tweening. If you can get an AI to do the job of a tweener you've already massively impacted the VFX/Animation pipeline.

DrewADesign14• 3 years ago

OK... This won't upend the VFX industry. Maybe a subsequent generation that is vastly more precise, but this isn't even ballpark. The stick figure results are vastly less impressive than current methods made with less time and nearly as little effort. Look at Unreal Engine's talk at GDC and then consider that movie studios generally won't use those tools for anything visually important because they just aren't as good as pre-rendered 3D. Even in places like The Mandalorian it was only used for background imagery.

londons_explore• 3 years ago

Someone will see these results, and think "if only it was 10x better, I could be rich by selling to the vfx industry". They'll then start training a 10x bigger model (there is plenty of video training data out there).

And in 3 months, it will be upending the vfx industry...

reasonabl_human2• 3 years ago

ControlNet is the new development that will really allow us to guide diffusion model outputs more granularly - first I’m seeing it used against video generation

mightytravels• 3 years ago

Wow - this looks awesome! Love the video2video.

adeon2• 3 years ago

It kinda works. I tried "kermit doing push ups" and it looks like kermit. And Kermit sorta looks like he is in push up position. Other than that it looks a lot like those early image generation AIs that vaguely look like what you asked but is all mutated. Animation itself is not very good for this prompt.

But hey, still pretty good for the early days. Maybe need to figure out how to prompt engineer it and tune it. Seems like it's very heavily based on existing image models? Wondering how easy it is to adapt to other image models. I think I need to read the paper. https://imgur.com/a/h3ciJNn

spokeonawheel• 3 years ago

I tried "spongebob hugging a cactus with his butt hanging out" and it said "error".

perhaps I dont understand this tool

adeon2• 3 years ago

You are welcome: https://imgur.com/a/bjfjhQq

bunnywantspluto• 3 years ago

Can we run this on a CPU?

novaRom2• 3 years ago

Yes, you can run any inference with any model on CPU, but here some examples:

To create a single frame with Stable Diffusion 4-5B parameters, 512x512, 20 iterations takes 5-30 minutes depending on your CPU. On any modern GPU it's only 0.1-20 seconds!

Similarly with LLMs, to produce one token with a transformer of 30-50 layers 7-12B parameters you will wait several CPU minutes while it takes few seconds on a Pascal-generation GPU and tiny fraction of second on Ampere.

probablynish• 3 years ago

The *.cpp adaptations of popular models appear to have optimized them for the CPU, for example llama.cpp and alpaca.cpp let me generate several tokens in a matter of seconds.

gpderetta• 3 years ago

To create a single frame with Stable Diffusion 4-5B parameters, 512x512, 20 iterations takes 5-30 minutes depending on your CPU

Depends a lot on the cpu. Are you specifically talking about Text2Video or SD in general? IIRC, last time I tried SD on my CPU (10 core 10850k, not exactly cutting edge) it did take less than one minute for more than 20 iters. This was about 4-5 months ago, things might have gotten better.

The GPU (even a vintage 1070) was faster still of course.

tjoff• 3 years ago

That is much better than expected, depending on how easy it is to setup it is very much worthwhile if you don't have access to a decent GPU.

dcreater2• 3 years ago

Can this get the ggerganov treatment so that I can run it on Apple silicon?

crakenzak3• 3 years ago

with llama.cpp getting GPT4all inference support the same day it came out, I feel like llama.cpp might soon become a general purpose high performance inference library/toolkit. excited!

ggerganov• 3 years ago

Heh, things seem to be moving in this direction, but I think it's still a very long way to go. But who knows - the amount of contributions to the project keep growing. I guess when we have a solid foundation for LLM inference we can think about supporting SD as well

Labo333• 3 years ago

SD models are much smaller so you can probably already run it with the github code.

Image models work well at least: https://huggingface.co/docs/diffusers/optimization/mps

It is even possible to run them in the browser: https://stablediffusionweb.com/

bdcravens• 3 years ago

I entered "A dachshund doing a backflip"

It was a dachshund. Doing some sort of a dance in the air. Nothing resembling a flip.

https://imgur.com/a/eqHpuo7

leereeves• 3 years ago

The examples on GitHub are so much better than the examples here. I wonder if the authors are cherry picking or we just don't know how to get good results.

Why are people downvoting this? The examples on GitHub are clearly doing the action described (unlike the dachsund above) and don't have the wild deformations that the "Spongebob hugging a cactus" or "Will Smith eating spaghetti" animations have.

gremlinsinc• 3 years ago

it only matters to the inpatient among us, give it a few weeks and things will only get better. The pace of ai innovation is frightening but exciting too.

nullsense• 3 years ago

They're probably cherry picking. I know on ModelScope text2video I have to generate 10 clips per prompt to get 2 or 3 usable clips and out of that will come one really good one. But it's fast enough to do that I just sort of accept that as the workflow and generate in batches of 10. I assume it's likely the same for this.

Can't wait to try it though.

throw101010• 3 years ago

In my experience AI generative processes always involve a lot of cherry picking, rarely have I generated 1 image or 1 text and it was perfect or very representative of what can be attained with some refining.

Seems fair to choose good results when you try to demonstrate what the software "can" do. Maybe a mention of the process often being iterative would be the most honest, but I think anyone familiar with these tools assume this by now.

godelski• 3 years ago

This is, unfortunately, standard practice on generative papers. Best is to show cherry picked and random selection. But if you don't cherry pick you don't get published. Even having random samples can put you in a tough position.

To be fair, metrics for vision are limited and many don't know how to correctly inturpret them. You both want to know what's the best images the network can produce a well as the average. Reviewers, unfortunately, just want to reject (no incentive you accept). That said, it can still be a useful tool because you know the best (or near) that the model can produce. This is also why platforms like HuggingFace are so critical.

I do think we should ask more of the research community but just know that this is in fact normal and even standard. Most research judge by things like citations and just reading the works themselves. The more hype in a research area the more noisy the signal for good research. (This is good research imo fwiw)

cubefox3• 3 years ago

As I mentioned recently, imgur hijacks the back button, at least in my config (mobile Chrome). I recommend not using it anymore.

tinus_hn• 3 years ago

As I mentioned recently

If only people would just listen to your sage advice!

This kind of sentence makes it come across as if you are on a high horse talking down to the unwashed masses. You might want to avoid this.

Workaccount2• 3 years ago

Someone here showed me this extension yesterday:

https://github.com/libredirect/libredirect

I don't know think it will work on mobile chrome, but for others like me that can't take imgurs bloat anymore, its great.

fleischhauf• 3 years ago

maybe the frame rate is not high enough to capture the flip, could just be a super fast dachshund..

fortyseven• 3 years ago

He's tryin' his best. :{

zamnos2• 3 years ago

They're Good Dogs Brent

2c2c2c• 3 years ago

looks like the leg motions of a kickflip, but didn't include the board

voxelghost• 3 years ago

what a time to be alive!

RajSinghLA• 3 years ago

Dear fellow scholars

crakenzak3• 3 years ago

this is two minute papers, with Dr. Karoly Zsolnai-Feher

detrites2• 3 years ago

For the adventurous among us, here's that boolean you might want to change:

https://github.com/Picsart-AI-Research/Text2Video-Zero/blob/...

stevefan1999• 3 years ago

Come on, somebody please make this an environmental variable

totetsu• 3 years ago

This is exactly the kind of thing the Taoist philosophers were warning us about.

belter• 3 years ago

You are the wind beneath my wings...

novaRom2• 3 years ago

Is there any estimate how much training data someone like MindGeek has?

astrange3• 3 years ago

More important question is who has the annotation budget.

SentinelLdnma• 3 years ago

Economists will ponder for decades why LLMs never increased labor productivity.

FractalHQ• 3 years ago

What was it? That link returns a 404 now.

rottc0dd• 3 years ago

This should help:

https://web.archive.org/web/20230329124453/https://github.co...

Check line 45

nprateem• 3 years ago

So long giphy

evolveyourmind• 3 years ago

Seems very limited. I wonder if the same can be achieved with just stable diffusion and neighbor latent walks with very small steps. On the other hand the interpolation techniques with the GigaGAN txt2img produce much higher quality “videos” than this

valine• 3 years ago

Right, stable diffusion and control net depth network could give very good results if you have a source video.

mnemotronic• 3 years ago

[edit] _Almost_ everything I try except the predefined examples returns "error". "a pencil with wings" returns something nothing like a pencil that does, in fact, have wings.

There will be a learning curve.

rasz• 3 years ago

at least two of the example dogs have 5 paws

rexreed2• 3 years ago

Question about models and weights. When an organization says they release the weights, how is that different from when an organization releases a model, say Whisper from OpenAI? In what way are the model releases different in these cases?

superb-owl• 3 years ago

The model is just an untrained architecture. You'd need to spend a lot of money (a) gathering data, and (b) running GPUs to train it.

The weights are the fruition of training. They make the model actually useful.

rexreed2• 3 years ago

When you download a trained model for use by Python, I'm assuming the model contains both the architecture (the neural net or even a boosted tree) as well as the weights / tree structure that makes the model actually usable in inference. When organizations release a trained model, I'm assuming that the weights are necessary to make use of that model? If not, then are they not really releasing the model, but just the architecture and training data?

leodriesch• 3 years ago

OpenAI also released the weights for Whisper. Some model releases, like LLaMa from Meta, just contain the code for training and the data. You can train the weights yourself, but for LLaMa that takes multiple weeks on very expensive hardware.

(LLaMa also released the weights for researchers and they leaked online, but the weights are not open-source.)

When a company releases a model including the weights, you can download their pre-trained weights and run inference with them, without having to train by yourself.

lifeisstillgood• 3 years ago

Weirdly Corridor digital had a AI generated video and they suffered from what is slightly happening here - the image of the bear / panda or whatever is a different animal each time (ie it's a panda, just a different one "hallucinated" each frame.

corridor digital handled it by training their model on specific images of specific people - and so they effectively said "video of the panda called phil that we have trained you on images of"

Clearly this is not possible here - so I am missing how they got it close

EGreg• 3 years ago

I want to output an animated GIF. What is the command?

Do I run pix2pix at the end?

Also can I somehow have more frames and set the GIF speed?

jonas-w• 3 years ago

I see that there is a 12GB VRAM requirement. Can my 6GB GPU do anything to at least provide some performance advantages as instead of running it entirely on the CPU?

tehsauce• 3 years ago

I haven't seen a single example from this model that demonstrates video with any time of time continuity. It appears every frame is independent to each other.

mlboss• 3 years ago

Pose based controlnet looks like it demonstrates continuity

fauxpause_• 3 years ago

I feel like they really missed an opportunity to make their example Horse in Motion rather than Horse Galloping on a Street

Text2Video-Zero Code and Weights Released by Picsart AI Research (12G VRAM)

Related Stories