Apple Releases Open Weights Video Model

https://starflow-v.github.io

by vessenes • 8 months ago

451 163 8 months ago

starflow-v.github.io

coolspot• 8 months ago

STARFlow-V is trained on 96 H100 GPUs using approximately 20 million videos.

They don’t say for how long.

moondev• 8 months ago

Apple intelligence: trained by Nvidia GPUs on Linux.

Do the examples in the repo run inference on Mac?

satvikpendem2• 8 months ago

Looks good. I wonder what use case Apple has in mind though, or I suppose this is just what the researchers themselves were interested in, perhaps due to the current zeitgeist. I'm not really sure how it works at big tech companies with regards to research, are there top down mandates?

ivape• 8 months ago

To add things to videos you create with your phone. TikTok and Insta will probably add this soon, but I suppose Apple is trying to provide this feature on “some level”. That means you don’t have to send your video through a social media platform first to creatively edit it (the platforms being the few tools that let you do generative video).

They should really buy Snapchat.

ozim• 8 months ago

I guess Apple is big in video production and animation with some ties via Pixar and Disney. Since Jobs started Pixar and it all got tied up in myriad of different ways.

nothrowaways• 8 months ago

Where do they get the video training data?

postalcoder• 8 months ago

From the paper:

Datasets. We construct a diverse and high-quality collection of video datasets to train STARFlow-V. Specifically, we leverage the high-quality subset of Panda (Chen et al., 2024b) mixed with an in-house stock video dataset, with a total number of 70M text-video pairs.

justinclift• 8 months ago

in-house stock video dataset

Wonder if "iCloud backups" would be counted as "stock video" there? ;)

devinprater6• 8 months ago

Apple has a video understanding model too. I can't wait to find out what accessibility stuff they'll do with the models. As a blind person, AI has changed my life.

densh2• 8 months ago

As a blind person, AI has changed my life.

Something one doesn't see in news headlines. Happy to see this comment.

badmonster• 8 months ago

What other accessibility features do you wish existed in video AI models? Real-time vs post-processing?

tippa1234• 8 months ago

+1 and I would be curious to read and learn more about it.

fguerraz• 8 months ago

Something one doesn't see in news headlines.

I hope this wasn't a terrible pun

kkylin3• 8 months ago

Like many others, I too would very much like to hear about this.

I taught our entry-level calculus course a few years ago and had two blind students in the class. The technology available for supporting them was abysmal then -- the toolchain for typesetting math for screen readers was unreliable (and anyway very slow), for braille was non-existent, and translating figures into braille involved sending material out to a vendor and waiting weeks. I would love to hear how we may better support our students in subjects like math, chemistry, physics, etc, that depend so much on visualization.

Rover222• 8 months ago

`Something one doesn't see` - no pun intended

WarcrimeActual• 8 months ago

I have to believe you used the word see twice ironically.

K0balt• 8 months ago

I must be wrong, but can’t help but harbor a mild suspicion that your use of sight metaphors is not coincidental.

phyzix5761• 8 months ago

Can you share some ways AI has changed your life?

darkwater2• 8 months ago

I guess that auto-generated audio descriptions for (almost?) any video you want is a very, very nice feature for a blind person.

gostsamo7• 8 months ago

Not the gp, but currently reading a web novel with a card game where the author didn't include alt text in the card images. I contacted them about it and they started, but in the meantime ai was a big help. all kinds of other images on the internet as well when they are significant to understanding the surrounding text. better search experience when Google, DDG, and the like make finding answers difficult. I might use smart glasses for better outdoor orientation, though a good solution might take some time. phone camera plus ai is also situationally useful.

devinprater6• 8 months ago

Image descriptions. TalkBack on Android has it built in and uses Gemini. VoiceOver still uses some older, less accurate, and far less descriptive ML model, but we can share images to Seeing AI or Be My Eyes and such and get a description.

Video descriptions, through PiccyBot, have made watching more visual videos or videos where things happen that don't make sense without visuals much easier. Of course, it'd be much better if YouTube incorporated audio description through AI the same way they do captions, but that may happen in a good 2 years or so. I'm not holding my breath. Google as a whole is hard to get accessibility out of more than the bare minimum.

Looking up information like restaurant menus. Yes it can make things up, but worst-case, the waiter says they don't have that.

javcasas• 8 months ago

Finally good news about the AI doing something good for the people.

p1esk• 8 months ago

I’m not blind and AI has been great for me too.

Workaccount23• 8 months ago

People need to understand that a lot of angst around AI comes from AI enabling people to do things that they formally needed to go through gatekeepers for. The angst is coming from the gatekeepers.

AI has been a boon for me and my non-tech job. I can pump out bespoke apps all day without having to get bent on $5000/yr/usr engineering software packages. I have a website for my side business that looks and functions professionally and was done with a $20 monthly AI subscription instead of a $2000 contractor.

GeekyBear• 8 months ago

One cool feature they added for deaf parents a few years ago was a notification when it detects a baby crying.

embedding-shape5• 8 months ago

Is that something you actually need AI for though? A device with a sound sensor and something that shines/vibrate a remote device when it detects sound above some threshold would be cheaper, faster detection, more reliable, easier to maintain, and more.

Damogran6• 8 months ago

I also got notification on my apple watch, while being away from the house, that the homepod mini heard our fire alarm going off.

A call home let us know that our son had set it off learning to reverse-sear his steak.

SatvikBeri• 8 months ago

My wife is deaf, and we had one kid in 2023 and twins in 2025. There's been a noticeable improvement baby cry detection! In 2023, the best we could find was a specialized device that cost over $1,000 and has all sorts of flakiness/issues. Today, the built-in detection on her (android) phone + watch is better than that device, and a lot more convenient.

andy_ppp• 8 months ago

I wonder if there's anything that can help blind people to navigate the world more easily - I guess in the future AR Glasses won't just be for the sighted but allow people without vision to be helped considerably. It really is both amazing and terrifying the future we're heading towards.

shagie2• 8 months ago

From a couple years ago...

https://www.microsoft.com/en-us/garage/wall-of-fame/seeing-a...

https://youtu.be/R2mC-NUAmMk

https://youtu.be/DybczED-GKE

... and that was 10 years ago. I'm curious for what it could do now.

xnx• 8 months ago

https://play.google.com/store/apps/details?id=com.google.and...

asadotzler2• 8 months ago

AURA Vision for blind and low vision people has been doing this for years. Be My Eyes has been doing this for years without AI. Meta Ray-Bans can do this. There's nothing new coming soon that hasn't already been available for a while, only refinements.

whatsupdog• 8 months ago

As a blind person, AI has changed my life.

I know this is a low quality comment, but I'm genuinely happy for you.

basilgohar• 8 months ago

I'm only commenting because I absolutely love this thread. It's an insight into something I think most of us are quite (I'm going to say it...) blind to in our normal experiences with daily life, and I find immense value in removing my ignorance about such things.

robbomacrae• 8 months ago

Hi Devin and other folks, I'm looking for software developers who are blind or hard of sight as there is a tool I'm building that I think might be of interest to them (it's free and open source). If you or anyone you know is interested in trying it please get in touch through my email.

mycall• 8 months ago

I'm in the middle of updating a new public transit app my team wrote in Flutter and want it to flow better for blind people. Would you rather use AI chat session with realtime information or go through the screens with VoiceOver/TalkBack controls? What about trust and hallucinations?

Any tips you can give?

camillomiller• 8 months ago

Hopefully this will make into some useful feature in the ecosystem and not contribute to having just more terrible slop. Apple has saved itself from the destruction of quality and taste that these model enabled, I hope it stays that way.

yegle• 8 months ago

Looking at text to video examples (https://starflow-v.github.io/#text-to-video) I'm not impressed. Those gave me the feeling of the early Will Smith noodles videos.

Did I miss anything?

M4v3R• 8 months ago

These are ~2 years behind state of the art from the looks of it. Still cool that they're releasing anything that's open for researchers to play with, but it's nothing groundbreaking.

Mashimo• 8 months ago

But 7b is rather small no? Are other open weight video models also this small? Can this run on a single consumer card?

tomthe• 8 months ago

No, it is not as good as Veo, but better than Grok, I would say. Definitely better than what was available 2 years ago. And it is only a 7B research model!

tdesilva• 8 months ago

The interesting part is they chose to go with a normalizing flow approach, rather than the industry standard diffusion model approach. Not sure why they chose this direction as I haven’t read the paper yet.

manmal• 8 months ago

I wanted to write exactly the same thing, this reminded me of the Will Smith noodles. The juice glass keeps filling up after the liquid stopped pouring in.

jfoster• 8 months ago

I think you need to go back and rewatch Will Smith eating spaghetti. These examples are far from perfect and probably not the best model right now, but they're far better than you're giving credit for.

As far as I know, this might be the most advanced text-to-video model that has been released? I'm not sure whether the license will qualify as open enough in everyone's eyes, though.

RobotToaster• 8 months ago

The license[0] seems quite restrictive, limiting it's use to non commercial research. It doesn't meet the open source definition so it's more appropriate to call it weights available.

[0]https://github.com/apple/ml-starflow/blob/main/LICENSE_MODEL

limagnolia3• 8 months ago

They haven't even released the weights yet...

As for the license, happily, Model Weights are the product of machine output and not creative works, so not copyrightable under US law. Might depend on where you are from, but I would have no problem using Model Weights however I want to and ignoring pointless licenses.

loufe• 8 months ago

The weights for the text-->image model are already on Huggingface, FWIW.

pabs32• 8 months ago

The output of a compiler is copyrightable, why aren't models similarly copyrightable?

mdrzn2• 8 months ago

"VAE: WAN2.2-VAE" so it's just a Wan2.2 edit, compressed to 7B.

BoredPositron• 8 months ago

They used the VAE of WAN like many other models do. For image models you see a lot of them using the flux VAE. Which is perfectly fine, they are released as apache2 and save you time to focus on your transformers architecture...

kouteiheika• 8 months ago

This doesn't necessarily mean that it's Wan2.2. People often don't train their own VAEs and just reuse an existing one, because a VAE isn't really what's doing the image generation part.

A little bit more background for those who don't know what a VAE is (I'm simplifying here, so bear with me): it's essentially a model which turns raw RGB images into a something called a "latent space". You can think of it as a fancy "color" space, but on steroids.

There are two main reasons for this: one is to make the model which does the actual useful work more computationally efficient. VAEs usually downscale the spatial dimensions of the images they ingest, so your model now instead of having to process a 1024x1024 image needs to work on only a 256x256 image. (However they often do increase the number of channels to compensate, but I digress.)

The other reason is that, unlike raw RGB space, the latent space is actually a higher level representation of the image.

Training a VAE isn't the most interesting part of image models, and while it is tricky, it's done entirely in an unsupervised manner. You give the VAE an RGB image, have it convert it to latent space, then have it convert it back to RGB, you take a diff between the input RGB image and the output RGB image, and that's the signal you use when training them (in reality it's a little more complex, but, again, I'm simplifying here to make the explanation more clear). So it makes sense to reuse them, and concentrate on the actually interesting parts of an image generation model.

mdrzn2• 8 months ago

Thanks for the explanation!

sroussey• 8 months ago

Since you seem to know way more than I on the subject, can you explain the importance of video generation that is not diffusion based?

dragonwriter3• 8 months ago

"VAE: WAN2.2-VAE" so it's just a Wan2.2 edit

No, using the WAN 2.2 VAE does not mean it is a WAN 2.2 edit.

compressed to 7B.

No, if it was an edit of the WAN model that uses the 2.2 VAE, it would be expanded to 7B, not compressed (the 14B models of WAN 2.2 use the WAN 2.1 VAE, the WAN 2.2 VAE is used by the 5B WAN 2.2 model.)

pulse7• 8 months ago

LoganDark• 8 months ago

Model Release Timeline: Pretrained checkpoints will be released soon. Please check back or watch this repository for updates.
The checkpoint files are not included in this repository due to size constraints.

So it's not actually open weights yet. Maybe eventually once they actually release the weights it will be. "Soon"

vessenes• 8 months ago

From the paper, this is a research model aimed at dealing with the runaway error common in diffusion video models - the latent space is (proposed to be) causal and therefore it should have better coherence.

For a 7b model the results look pretty good! If Apple gets a model out here that is competitive with wan or even veo I believe in my heart it will have been trained with images of the finest taste.

dymk• 8 months ago

Title is wrong, model isn’t released yet. Title also doesn’t appear in the link - why the editorializing?

giancarlostoro• 8 months ago

I was upset the page didnt have videos immediately available, then I realized I have to click on some of the tabs. One red flag on their github is the license looks to be their own flavor of MIT (though much closer to MS-PL).

andersa• 8 months ago

The number of video models that are worse than Wan 2.2 and can safely be ignored has increased by 1.

embedding-shape5• 8 months ago

To be fair, the sizes aren't comparable, and for the variant that is comparable, the results aren't that much worse.

dragonwriter3• 8 months ago

The samples (and this may or may not be completely fair, either set could be more cherry picked than the other, It would be interesting to see a side-by-side comparison with comparable prompts) seem significantly worse than what I’ve seen from WAN 2.1 1.3B, which is both fron the previous WAN version and is smaller, proportionally, compared to Apple’s 7B than that model itself is compared to the 28B combination of the high and low noise 14B WAN 2.2 models that are typically used together.

But also, Starflow-V is a research model with a substandard text encoder, it doesn't have to be competitive as-is to be an interesting spur for further research on the new architecture it presents. (Though it would be nice if it had some aspect where it offered a clear improvement.)

wolttam• 8 months ago

This doesn’t look like it was intended to compete. The research appears interesting

gorgoiler• 8 months ago

It’s not really relevant to this release specifically but it irks me that, in general, an “open weights model” is like an “open source machine code” version of Microsoft Windows. Yes, I guess I have open access to view the thing I am about to execute!

This Apple license is click wrap MIT with the rights, at least, to modify and redistribute the model itself. I suppose I should be grateful for that much openness, at least.

advisedwang• 8 months ago

Great analogy.

To extend the analogy, "closed source machine code" would be like conventional SaaS. There's an argument that shipping me a binary I can freely use is at least better than only providing SaaS.

limagnolia3• 8 months ago

I think you are looking at the code license, not the model license.

Aloisius• 8 months ago

No, it's the model license. There's a second license for the code.

Of course, model weights almost certainly are not copyrightable so the license isn't enforceable anyway, at least in the US.

The EU and the UK are a different matter since they have sui generis database rights which seemingly allows individuals to own /dev/random.

satvikpendem2• 8 months ago

Yes, I guess I have open access to view the thing I am about to execute!

Better to execute locally than to execute remotely where you can't change or modify any part of the model though. Open weights at least mean you can retrain or distill it, which is not analogous to a compiled executable that you can't (generally) modify.

cubefox• 8 months ago

Interesting that this is an autoregressive ("causal") model rather than a diffusion model.

Invictus02• 8 months ago

Apple's got to stop running their AI group like a university lab. Get some actual products going that we can all use--you know, with a proper fucking web UI and a backend.

Jtsummers• 8 months ago

Personally, I'm happy that Apple is spending the time and money on research. We have products that already do what this model does, the next step is to make it either more efficient or better (closer to the prompt, more realistic, higher quality output). That requires research, not more products.

Invictus02• 8 months ago

Not complaining about the research. I'm complaining that I have to do work to even try this, instead of Apple offering a demo or portal of some kind.

summerlight• 8 months ago

This looks interesting. This project has some novelty as a research and actually delivered a promising PoC but as a product it implies that its training was severely constrained by computing resources, which correlates well with the report that their CFO overruled CEO's decision on ML infra investment.

JG's recent departure and follow up massive reorg to get rid of AI, rumors on Tim's upcoming step down in early 2026... All of these signals indicate that those non-ML folks have won corporate politics to reduce the in-house AI efforts.

I suppose this was a part of serious efforts to deliver in-house models but the directional changes on AI strategy made them to give up. What a shame... At least the approach itself seem interesting and hope others to take a look and use it for building something useful.

Apple Releases Open Weights Video Model

Related Stories