Stability.ai – Introducing Stable Video 3D

https://stability.ai/news/introducing-stable-video-3d

by ed • 2 years ago

742 131 2 years ago

stability.ai

Filligree2• 2 years ago

If the animations shown are representative, then the mesh output may very well be good enough to use in a 3d printer.

Looking forward to experimenting with this.

neom2• 2 years ago

I don't know much about 3D printing, would be very interested in learning more about this idea if you'd be so kind as to expand on it. Could I have AI spend all day auto scanning what teens are doing on instagram, auto generate toys based on it, auto generate advertisements for the toys, auto 3D print on demand?

SirSourdough• 2 years ago

Hypothetically, sure, assuming the parent comment that these meshes are sufficient for modelling is correct and that you can find any teens who want a non-digital toy.

I think a good hobbyist application for this would be something like modelling figurines for games, which is already a pretty popular 3D printing application. This would allow people with limited modelling skills to bring fantastical, unique characters to life “easily”.

CobrastanJorji• 2 years ago

I think their suggestion was more "I have a photo of a cool horse, and now I would like a 3D model of that same horse."

Another way of looking at it, 3D artists often begin projects by taking reference images of their subject from multiple angles, then very manually turning that into a 3D model. That step could potentially be greatly sped up with an algorithm like this one. The artist could (hopefully) then focus on cleanup, rigging, etc, and have a quality asset in significantly less time.

maicro• 2 years ago

OP is suggesting that this (AI model? I honestly am behind on the terminology) could replace one of the common steps of 3D printing - specifically, the step where you create a digital representation of the physical object you would want to end up with.

There are other steps to 3D printing in general, though; a super rough outline:

- Model generation

- "Slicing" - processing the 3D model into instructions that the 3D printer can handle, as well as adding any support structures or other modifications to make it printable

- Printing - the actual printing process

- Post-processing - depending on the 3D printing technology used, the desired resulting product, and the specific model/slicing settings, this can be as simple as "remove from bed and use" to "carefully snip off support structures, let cure in a UV chamber for X minutes, sand and fill, then paint"

As I said before, this AI model specifically would cover 3D model generation. If you were to use a printing technology that doesn't require support structures, and handles color directly in the printing process (I think powder bed fusion is the only real option here?), the entire process should be fairly automatable - a human might be needed to remove the part from the printer, but there might not be much post-processing to do.

The rest of your desired workflow is a bit more nebulous - I don't know how you would handle "scanning what teens are doing on instagram", at least in a way that would let you generate toys from the information; generating and posting the advertisement shouldn't be too hard - have a standardish template that you fill in with a render from the model, and the description; printing on demand again is possible, though you'll likely need a human to remove the part, check it for quality and ship it. You could automate the latter, but that would probably be more trouble than it's worth.

jsheard4• 2 years ago

With previous attempts at this problem the shaded examples could be quite misleading because details that appeared to be geometric were actually just painted over the surface as part of the texture, so when you took that texture away you just had a melted looking blob with nowhere near as much detail as you thought. I'd reserve judgement until we see some unshaded meshes.

What they show in the demo: https://i.imgur.com/9bZNTcd.jpeg

What comes out of the 3D printer: https://i.imgur.com/MZrzsfh.png

SV_BubbleTime• 2 years ago

It’s always been this. None of these ever show the untextured model.

When I see a demo where they are showing wireframes I know it’ll be good enough.

euazOn• 2 years ago

Therefore, what is the main usecase of this model? Generating cheap 3D assets for videogames?

strich• 2 years ago

There exists software to reproject texture normals back on to a high poly model. So this problem does have a solution for anyone interested.

Oioioioiio3• 2 years ago

There are AI models who can create proper meshes though.

ionwake4• 2 years ago

Im sorry for dumb lazy question. But would the input require more than one image? Is there a demo url to test this? I think it might jsut be time to buy a 3d printer.

EDIT> Does "single image inputs" mean more than one image?

kylebenzle• 2 years ago

Single image means one image.

dartos• 2 years ago

Can confirm the word single means 1

ionwake4• 2 years ago

lol cmon guys don't be too hard on me it does say "inputs"

simonw• 2 years ago

It's just a single image. It guesses the shape of the bits it can't see based on vast amounts of training data.

ionwake4• 2 years ago

Amazing! Thank you

exodust• 2 years ago

I have an even lazier question after failing to speed-read the article.

Does this output an actual 3D mesh? Or does it only output a 3d-looking rendered animation?

airstrike• 2 years ago

that demo animation is so clever and satisfying

amelius• 2 years ago

But it doesn't look very realistic, tbh.

dreadlordbone• 2 years ago

it doesn't break Euclidian space at least

itsgrimetime• 2 years ago

I can’t get them to play

ddtaylor• 2 years ago

Does anyone know what hardware inference can run on or memory requirements?

Mathnerd314• 2 years ago

In the repo the model weights file is 9.37GB, whereas sdxl turbo is 13.9GB, and I don't see any mention of huge context windows, so probably it just needs a decent graphics card.

kouteiheika3• 2 years ago

It crashes with an out-of-memory error on my 24GB 4090, so at least when it comes to their sample script the answer is "a lot". Maybe it's just an inefficient implementation though.

dragonwriter2• 2 years ago

Pretty much every initial Stability release has been inefficient and has resources drop a lot when optimized for real consumer hardware community engines appeared for running the model.

OTOH, with their shift to a less open licensing structure, community tooling probably won’t emerge with the same level of energy.

canadiantim• 2 years ago

I can't wait until we can use something like this for architectural design

whywhywhywhy2• 2 years ago

SDXL+Controlnet and then feeding it just blocked out depth maps are probably more useful for that.

andybak• 2 years ago

Something like https://dust3r.europe.naverlabs.com/ might be more appropriate?

kouteiheika3• 2 years ago

Just tried to run this using their sample script on my 4090 (which has 24GB of VRAM). It ran for a little over 1 minute and crashed with an out-of-memory error. I tried both SV3D_u and SV3D_p models.

[edit]Managed to generate by tweaking the script to generate less frames simultaneously. 19.5GB peak VRAM usage, 1 min 25 secs to generate at 225 watts.[/edit]

ganeshkrishnan3• 2 years ago

4090 is in weird spot. High speed but low RAM. Theoretically everything should run in ai but practically nothing runs

LoganDark2• 2 years ago

4090 has more VRAM than most computers have system RAM. Surprised this is considered "low RAM" in any way except for relative to datacenter cards and top-spec ASi.

jokethrowaway• 2 years ago

What can't you run? Unquantised large text models are the only thing I can't run

Stable diffusion, stable video, text models, audio models, I never had issues with anything yet

Hikikomori2• 2 years ago

Maybe dont use a gaming card for ai then? 24 is plenty as most games dont use more than half in 4k.

Sohcahtoa82• 2 years ago

They don't want to cannibalize sales of the super-expensive GPUs dedicated to ML/AI.

5090 likely won't have more than 32 GB, if even that much.

Zenst• 2 years ago

Perhaps NVIDIA or somebody could invent a RAM upgrade via NVLINK? Seems plausible and not every problem would want to add another GPU when the ability to add the extra memory alone is all they need.

chaostheory2• 2 years ago

Yeah, I’m still debating whether I go with a Mac Studio with the RAM maxed out (approx $7500 for 192 GB) or a PC with a 4090. Is there a better value path with the Nvidia A series or something else? (I’m not sure about tibygrad)

idonotknowwhy2• 2 years ago

Didn't know 24GB was considered low lol.

karolist4• 2 years ago

You can add multiple, but practically speaking you're better off with used 3090s which you get 2 for the price of one 4090.

I have 3090 Ti and I can run Q4 quant 33b models at 30t/s with 8k context. A 4090 would allow me to do the same but with ~45t/s, both inference speeds are more than fast enough for people so 3090 is the usual choice. In my tests on runpod, H100 with 80GB memory is around the same speed as 3090, so slower than a 4090.

chaxor3• 2 years ago

"Theoretically everything should run in ai"

Odd statement. I don't really know what you mean by that. Perhaps 'math _works_, code should too' ?

I would definitely agree that it _should_ work.

I'm of the belief that no one should _have to_ publish (e.g. to graduate, get promotions, etc) in academia, and that publications should only occur if they're believed to be near Novel prize worthy, and fully reproducible by code with packaging that should last and work in 10 years, from data archives that will exist in 10 years.

But it seems I have been outvoted by the administration in academia.

Hence, we get this "ai that doesn't run" phenomenon

jug• 2 years ago

Almost sounds like a GPU vendor who isn't seeing enough competition.

blackoil• 2 years ago

It is targeted to gamers, that professionals are buying. They should be buying A6000 which has 48GB.

bionhoward• 2 years ago

a ~multi thousand dollar chip with 24gb is “low ram” ? That’s more ram than most computers!

GistNoesis• 2 years ago

I managed to get it working with a 4090. You need to adjust the parameter decoding_t of the sample function in simple_video_sample.py to a lower value (decoding_t = 5 works fine for me). I also needed to install imageio==2.19.3 and imageio-ffmpeg

kouteiheika3• 2 years ago

Ah, yep! You're right! It works now!

monkeynotes• 2 years ago

Yeah, this is to be expected with early adoption. This stuff comes out of the lab and it's not perfect. The key thing to evaluate is the trajectory and pace of development. Much of what folks challenged ChatGPT with a year ago is long lost in the dust. Go look at stable diffusion this time last year. Dall-E couldn't do words and hands, it nails that 90% of the time in my experience today.

remotefonts• 2 years ago

About words, Dall-e is nor even close to nail it 90% of the time. Not even 50%. Maybe they nerf it when you request a logo from it, but that was my experience in the last few days.

whywhywhywhy2• 2 years ago

Dunno why the defaults for this stuff isn't the base performance, feel I always have to tweak the batch size down on all the base scripts even with 24gb cos everything assumes 48gb

londons_explore• 2 years ago

All the examples resemble plastic children's toys...

How would it handle other objects? (People, fabrics, buildings, plants, mountains, mechanical parts, etc)

programjames• 2 years ago

It's hard to get camera position tracking for random objects, so it looks like they used simulations. There's probably a lot more plastic children's toy models in Blender than people, fabrics, buildings, &c.

issung• 2 years ago

Stable Video 3D (SV3D) is a generative model based on Stable Video Diffusion that takes in a still image of an object as a conditioning frame, and generates an orbital video of that object.

So can it actually output a 3d model? Or just images of what it thinks the object would look like from other angles?

krebby• 2 years ago

The reference video (https://youtu.be/Zqw4-1LcfWg) says they use a NeRF / structure from motion and then create a mesh with marching cubes from the generated radiance field. This is how most soa text-to-object generators work now as well

2StepsOutOfLine• 2 years ago

I'm also struggling to find any examples of how to actually get a 3D model output. Very few references to this capability outside of the blog post.

dubin2• 2 years ago

I'd like to play around with something like this, but from my understanding my machine (Macbook, 2021 M1) isn't nearly powerful enough (right?). Are there remote/cloud environments where I can run models like this?

ilaksh• 2 years ago

I suggest just using Stability's API. You aren't allowed to use it locally for commercial use anyway.

You could set something up on RunPod or AWS, but I doubt it's worth the effort.

dubin2• 2 years ago

Awesome, thank you!

It does look like SV3D is not a part of the API currently, but only a matter of time I imagine.

thrdbndndn• 2 years ago

The emphasis here is Single Image, but can this model generate with multiple images too?

We know that a single image of an object physically can't cover all the sides of it, so it's all guesswork in AI. This is totally fine for certain scenario, but in lots of other cases, it's trivial to have multiple images of the same object, and if that offers higher fidelity, it's totally worth it.

I'm aware there are many algorithms or AI models that already do that. I'm asking about Stability's one specifically because if they have impressive Single Image result, surely their multi-image results would also be much better than state-of-the-art?

pksebben3• 2 years ago

If it's not there yet, I'm willing to bet it will be soon enough given folks hacking it apart and injecting their own solutions.

leesec• 2 years ago

I wonder when Emad will be outed as a fed or a fraud. He's certainly leaving a trail of nasty behavior in the industry.

preommr• 2 years ago

elaborate?

esafak• 2 years ago

https://en.wikipedia.org/wiki/Emad_Mostaque#Controversy_and_...

dheera• 2 years ago

They compare against Zero123-XL, but they should compare against MVDream instead. MVDream is quite good. If you fiddle with the loss you can get even better results.

abdellah123• 2 years ago

Did you write the blog post using AI ?

throwaway743• 2 years ago

Anyone know of anything that'll auto rig/add weights?

ImHereToVote2• 2 years ago

There are numerous tools that auto-rig humanoid figures. The obvious one: https://www.mixamo.com/#/

nbzso• 2 years ago

Billions purred into technology with minimal use case application. What is the direct implication of this tech? Porn on demand?

akanet• 2 years ago

There are many direct consequences of people being able to directly transform text into textured 3d models, and even vaster indirect consequences if one pauses to reflect. A tight feedback system with good cohesion could revolutionize, art, design, mechanical engineering, video games, etc, etc.

ecronant• 2 years ago

Avante-garde/experimental film making is the main benefactor of all this.

Basically, cool looking video no one watches. I say that as a huge fanboy/artist myself. It is like Christmas every other day right now. All this VC money being set on fire to make better Avante-garde film tools is just wonderful. A dream come true.

GeoAtreides• 2 years ago

Extracting object from an image, transforming the object (rotating, for example), re-blended it into the original image.

Or making 3d game assets from objects you have around. Imagine: take your phone, go around town, into shops, into churches, come back, press a button, get huge library 3d assets to populate your game.

Or, something like this, for IKEA: couple of photos of a room --> extract objects --> let user re-arrange furniture. The room could be either the user's room or an IKEA showroom.

You can do it with existing tools, but this kind of technology reduces it to pressing a couple of buttons.

Ultimatt• 2 years ago

How has no one drawn attention to this was science fiction in Enemy of the State in 1998 now a trivial reality https://www.youtube.com/watch?v=4AjLXZV46eE

Stability.ai – Introducing Stable Video 3D

Related Stories