MobileDiffusion: Rapid text-to-image generation on-device

https://blog.research.google/2024/01/mobilediffusion-rapid-text-to-image.html

by jasondavies • 2 years ago

261 60 2 years ago

blog.research.google

sp332• 2 years ago

I don't suppose there's any way to actually get this?

Kind of funny that they show the iphone 15 pro and the Samsung S24 in the comparison chart, but not their own phone the google pixel 8. (I know it will perform worse than both phones)

geor9e2• 2 years ago

Pixel marketing touts the NPU chip, which sounds ideal for accelerating a model like this. They might have reasons for leaving it out. Perhaps it's planned to ship with a new model - announcing a new phone in a model paper would be weird. I remember them advertising some Pixel-only photo editing features in the past.

refulgentis2• 2 years ago

It's great in theory until you have to pierce through several organizational boundaries to actually leverage it. Too sclerotic for it to happen regularly at scale.

causality0• 2 years ago

All the Pixel AI stuff runs on the cloud anyway. Just try using it in airplane mode.

_the_inflator• 2 years ago

There is some truth to it.

I don’t see it as a disadvantage, since Google markets services on both devices you mentioned. Hardly anyone will abandon its iPhone in favor of a Pixel just for a Google service.

So I think it’s ok what Google did marketing wise.

minimaxir2• 2 years ago

With superior efficiency in terms of latency and size, MobileDiffusion has the potential to be a very friendly option for mobile deployments given its capability to enable a rapid image generation experience while typing text prompts. And we will ensure any application of this technology will be in-line with Google’s responsible AI practices.

So I'm interpreting this that it won't ever get released.

Legend2440• 2 years ago

I would interpret it as "expect to see this powering some features in the next-generation Pixel".

jsheard• 2 years ago

We've already seen this progression - they debuted Magic Eraser as a cloud feature, then with the Pixel 8 they got it running locally on the device. But they also introduced Magic Editor with the Pixel 8, running on the cloud, and the Pixel 9 or 10 will probably run it on-device.

kccqzy• 2 years ago

I'm interpreting it as they will be adding a layer of safety restrictions. Understandable given the furore of the recent Taylor Swift generated image incident.

Everyone needs to do this and probably is already doing this. Search for "ChatGPT lobotomized" and you'll see plenty of complaints about the safety filters added by OpenAI.

babyshake• 2 years ago

I'm much more comfortable with the idea of AI watermarking images it creates instead of refusing to create images because of "safety", which in practice more often means not wanting to offend anyone. Imagine if word processors like Google Docs refused to write things you wanted to write because of mature themes. The important thing, in my opinion, is to make it a lot more difficult to pass off AI generated content as being authentic and to make provenance traceable if you were to do something like create revenge porn with AI, but not to make AI refuse to create explicit material at all.

cuuupid• 2 years ago

Google has fallen so far. Both Inception and Mobilenet were released openly and changed the entire AI world.

Nowadays we just get blog posts about results that were supposedly achieved, an accompanying paper that can’t be reproduced (because of Google’s magical “private datasets”), and some screencaps of a cool application of the tech that is virtually guaranteed to never make it to product.

minimaxir2• 2 years ago

No one ever filed multimillion dollar lawsuits over Inception and MobileNet.

The calculus is different now.

tussa• 2 years ago

Probably because the actual product is garbage.

Remember the Google assistant demo, where it booked a table at a restaurant?

That never materialized. Google assistant is just eating crayons today.

refulgentis2• 2 years ago

Yeah it did

But to your point its not as impressive as it was sold: glorified reverse phone tree. But it is what it was sold as, and it is available.

inquirerGeneral• 2 years ago

It did work, I used it myself. A quick search shows others who had my experience. This was late 2019 for me. Here's the first link and the Google post on rolling out in summer 2019:

"Google Assistant just made a dinner reservation for me... I knew this was coming... but mind blown!" https://www.reddit.com/r/googlehome/comments/ezv3us/google_a...

"Now, you can use it on all Pixel phones in 43 U.S. states.

All it takes is a few seconds to tell your Assistant where you'd like to go. Just ask the Assistant on your phone, “Book a table for four people at [restaurant name] tomorrow night.” The Assistant will then call the restaurant to see if it can accommodate your request. Once your reservation is successfully made, you’ll receive a notification on your phone, an email update and a calendar invite so you don’t forget."

https://blog.google/products/assistant/book-table-google-ass...

the-rc• 2 years ago

It did materialize.

https://www.reddit.com/r/googlehome/comments/ezv3us/google_a...

Or the comments under https://youtu.be/-RHG5DFAjp8

It's probably hard to trigger these days because most places support OpenTable or similar.

N3cr0ph4g1st• 2 years ago

Assistant makes reservations for me just like that demo :P but in general I agree it's gone very downhill

rysertio• 2 years ago

Original paper: https://arxiv.org/abs/2311.16567

ollin2• 2 years ago

some points that stood out to me:

1. they made a lot of careful tweaks to the unet network architecture - it seems like they ran many different ablations here ("In total, our endeavor consumes approximately 512 TPUs spanning 30 days").

2. the model distillation is based on previous UFOGen work from the same team https://arxiv.org/abs/2311.09257 (hence the UFO graphic in the diffusion-gan diagram)

3. they train their own 8-channel latent encoder / decoder ("VAE") from scratch (similar to Meta's Emu paper) instead of using the SD VAEs like many other papers do

4. they use an internal dataset of 150m image/text pairs (roughly the size of laion-highres)

5. they also reran SD training from scratch on this dataset to get their baseline performance

asimpleusecase• 2 years ago

Google may very well be first to create AGI but it will be wrapped in so many “safety” layers that it would effectively be lobotomised. Let’s just hope that a Google AGI never gets to watch A clockwork orange.

spupy• 2 years ago

So what could be some use cases for this apart from as a toy or for faking photos/art?

simultsop• 2 years ago

Generating memes to friends sparing search time. Describing sketches have them on the fly sharing with teams?

nonrandomstring• 2 years ago

People search and remember things visually. Even if they're not consciously aware. So on the Cybershow [0] we decided to jump-in and use AI images as a quick way to visually tag episodes with something meaningful and fun.

We did that despite some moral ambivalence/uneasiness around AI "art".

For example, give me a "young and exciting Dana Meadows in front of a board of systems theory"

I'm not awful at photoshopping things, and sometimes that's the only way to get a specific image one has in mind. But it saves time and lets us concentrate on writing and researching instead.

TBH if an artist/illustrator came along and said "Let me do the episode icons even though you can't pay me yet" I'd feel inclined to ask the AI to step aside.

[0] https://cybershow.uk/episodes.php

malka• 2 years ago

At first it seems nice. Then I realise it is by Google.

This will never see the light of day.

thyrox• 2 years ago

I never upvote any Google's A.I. research articles as most of the time it is: look what we have done, but we will never release anything.

OpenAi gets a lot of criticism for being closed, but at least I can play with their api most of the time.

What's the point of this if we will never be able to use this?

rvnx• 2 years ago

Corporate AND personal marketing.

AI researchers can make any claim, the risk of getting busted is close to none.

Didn't work ? Well dataset was different

Didn't work ? Well code was different

Can I try your work ? Well it's proprietary / I don't have access / We shutdowned the cluster

But the result is guaranteed increase in salary and job opportunities.

Since these companies are publicly listed, they are by definition encouraged and encouraging to make grandiose claims in order to make themselves more attractive to investors, and they can blame the individuals if it becomes discovered.

My favorite being that Bard (PaLM version) is sentient, but it was too big this time.

Imagine a large pharmaceutical company claiming they can cure very important diseases, but the results cannot be independently verified, nor audited.

It’s ok in the short-term, but not when you make that claim during few years.

nuz• 2 years ago

The point is probably the implication that it'll be pushed to android as a native feature (in the photos app or similar), thus it making sense for investors reading this to put money into google rather than e.g. stability or openai etc. The people writing the article are likely shareholders etc.

htsh• 2 years ago

Dreambooth was kinda great?

That said, I agree that I wish there were more done post-research towards products with some of this stuff.

djoldman• 2 years ago

https://arxiv.org/pdf/2311.16567.pdf

whywhywhywhy2• 2 years ago

Are like people at Google Research not embarrassed that none of this stuff ever makes it to real life?

Google AI internally needs a huge culture change, stop acting like academics making things for academics and start working like developers making products for customers.

I'd say in 10 years we'll be looking back and seeing the wasted potential but actually you can look back around 10 years and already see the wasted potential of all the things Google demoed or papered and never shipped.

unclad5968• 2 years ago

That's what research is. It's somebody else's job to determine whether or not it would be useful as a product.

whywhywhywhy2• 2 years ago

That attitude is why they're now behind in the AI race despite a 10 year head start...

Stop larping as an academic and actually start working as an employee, you know what they actually are.

itsTyrion• 2 years ago

I'm tired of this crap

MobileDiffusion: Rapid text-to-image generation on-device

Related Stories