Generative Models: What do they know? Do they know things? Let's find out

https://intrinsic-lora.github.io/

by corysama • 2 years ago

381 122 2 years ago

intrinsic-lora.github.io

goryramsy• 2 years ago

SSL error? Just me?

corysama3• 2 years ago

Works on my machine. Alternative link: https://github.com/duxiaodan/intrinsic-lora

01HNNWZ0MV43FF• 2 years ago

Works here. "Verified by DigiCert". SHA-256 fingerprint of 38:2C:D4:2D:33:C0:2B:C6:67:8E:65:7C:E1:7B:84:6D:04:73:A7:E7:91:CD:B3:5B:8E:AD:90:1A:F1:E1:1A:08

dkarras• 2 years ago

This is pretty remarkable. So these really do learn humanly interpretable representations and not only doing some magic in the billion dimensional hyperplane that we can't hope of deciphering.

corysama3• 2 years ago

As an old 3D graphics engineer, the fact that albedo is in there is as just striking as it should be expected.

The core components of physically based rendering are position (derivable from image XY and depth), surface normal, incoming light, and at least albedo + one of a few variations on surface material properties such as specularity and roughness.

That the AI is modeling depth is pretty expected. Modeling surface normal is a nice local convolution of depth. But, modeling albedo separated from incoming light is great. I wonder if specularity is hiding in there too.

hackerlight• 2 years ago

It's a good depth map, too. Better than other tools I've seen which require fiddling with lots of knobs to get a good result. Might be useful for textures for parallax mapping.

viraptor2• 2 years ago

I'm mostly ok with the idea that the text-to-image models do this. What blows me away is that the autoregressive model does it too. We're not even asking it to produce a specific object which has a meaning or context assigned. The network learned to produce those representations as just "this is a useful decomposition if I want to recreate the same thing again". The jump from image compression to figuring out how to represent objects in space is insane. And makes me want to feed some optical illusions through that network - what's the depth map of the infinite staircase?

int_19h7• 2 years ago

I find it amazing that, for all the evidence we have of generative models having some fairly complex internal model of the world, people still insist that they are mere "stochastic parrots" who "don't really understand anything".

mewpmewp22• 2 years ago

From the other side however if you really think about it, our understanding of everything must be stochastic as well. So perhaps this sort of thing yields in many complexities that we are not aware of. How would I know I am not a stochaistic parrot of some sort. I am just neural nets traine on current envrionment while the base model that is dependent on DNA, through evolution and natural selection of the fittest. Same as currently competing LLMs where the best one will win out.

wredue• 2 years ago

We have zero evidence that these models have any “understanding” of anything.

“Understanding” would mean that they be able to train themselves, which they are as yet unable to do.

We are tuning weights and biases to statistically regurgitate training data. That is all they are.

asadotzler2• 2 years ago

Well, when the cherry picked videos from OpenAI have a cat springing a fifth leg for no reason, people are right to be skeptical.

bbor2• 2 years ago

I have no idea what Toyota or adobe are up to and why they’re funding research with a name like this, but I fucking love it. It’s science, let’s get some whimsy back in here!!

More materially:

  Optimized with a small set of labeled images, our model-agnostic approach adapts to various generative architectures, including Diffusion models, GANs, and Autoregressive models.

Am I correct in understanding that this is purely a visuospatial tool, and the examples aren’t just visual by coincidence? Like, there’s no way to stretch this to text models? Very new to this interpretability approach, very impressive.

raincole• 2 years ago

Bojack Horseman reference that we didn't know we need.

seattle_spring• 2 years ago

What is this, a crossover episode?

sp332• 2 years ago

There's research into editing facts in language models. https://rome.baulab.info/

frri• 2 years ago

Si hzux

blovescoffee• 2 years ago

You have no idea why Toyota or Adobe are funding computer vision research?

bbor2• 2 years ago

lol yeah a little. A) it said something very odd sounding like “Toyota university of Chicago”, which wtf why does Toyota have a university, and B) most labs would be hesitant to publish a paper with an extraneous clause in the title just to reference an absurd cartoon

dougmwne4• 2 years ago

This is awesome!

So one of the big reasons there was hype about Sora is that it felt very likely from watching a few videos that there was an internal physical simulation of the world happening and the video was more like a camera recording that physical and 3D scene simulation. It was just a sort of naïve sense that there HAD to be more going on behind the scenes than gluing bits of other videos together.

This is evidence, and it’s appearing even in still image generators. The models essentially learn how to render a 3D scene and take a picture of it. That’s incredible considering that we weren’t trying to create a 3D engine, we just threw a bunch of images at some linear algebra and optimized. Out popped a world simulator.

ClumsyPilot• 2 years ago

Well we do have game engines for a while now. Maybe a game engine with a chat interface would do a better job

TheDudeMan• 2 years ago

Depends on whether your goal is to render a scene or push the envelope in generative AI.

3abiton• 2 years ago

I think is the key, most likely it was a big part of their training dataset.

patcon• 2 years ago

we weren’t trying to create a 3D engine, we just threw a bunch of images at some linear algebra and optimized. Out popped a world simulator.

Heh. Sounds like what a personified evolution might say about a mind ;)

whimsicalism2• 2 years ago

just a sort of naïve sense that there HAD to be more going on behind the scenes than gluing bits of other videos together

People really still think that's all thats happening?

0_____0• 2 years ago

Who is people? Most folks only vaguely know about generative AI if they know about it at all. I'm technical but not in software specifically and usually ignore most AI news, so GP's comment is in fact news to me!

sigmoid10• 2 years ago

There's a lot of nonsensical babble out there from people who have minimal technical insight - both on the sceptic and on the enthusiast side of AI.

It really wasn't clear for the longest time how these models generate things so well. Articles like this one are still rare and comparatively new. And they certainly haven't been around when less informed enthusiasts where already heralding AGI.

On the other hand, we have accrued quite a bit of evidence by now that these models do far more than glue together training data. But there are still sceptics out there who spread this sort misinformation.

stormfather• 2 years ago

Making Sora's results by glueing videos together would be far more impressive IMO. That would take ASI

asadotzler2• 2 years ago

Like the cat springing a 5th leg and then losing it just as quick, in a cherry-picked video from the software makers? How does that fit your wishful narrative?

alpaca128• 2 years ago

Look at how anyone who's not an artist (especially children) draws pretty much anything, whether it's people or animals or bicycles. You'll see a lot of asymmetric faces, wrong shading, completely unrealistic hairs etc and yet all those people are intelligent and excellent at understanding how the world works. Intelligence can't be expressed in a directly measurable way.

Those videos may be cherry-picked but they're almost good enough that you have to pick problems to point out, problems that will likely be gone a couple iterations later.

Less than a decade ago we read articles about Google's weird AI model producing super trippy images of dogs, now we're at deepfakes that successfully scam companies out of millions and AI models that can't yet reliably preserve consistency across an entire image. In another ten years every new desktop, laptop and smartphone will have an integrated AI accelerator and most of those issues will be fixed.

ozgung• 2 years ago

We humans live in a 3D world and our training set is a continuous stereo stream of a constant scene from different angles. Sora, on the other hand, learned the world by watching TV. It needs to play more video games, in order to learn 3D scenes (implicit representation of a world) and taking their pictures (rendering). Maybe that was the case I don’t know.

DinaCoder992• 2 years ago

It needs to play more video games, in order to learn 3D scenes (implicit representation of a world) and taking their pictures (rendering).

This can be extrapolated from TV, too.

SomeoneFromCA• 2 years ago

NNs are _not_linear algebra. The genius of NN is that it is a half-linear algebra (assuming most of them these days use ReLU activation.) And thst half linearity gives power.

GuB-42• 2 years ago

Turns out that 3D graphics also involves a lot of linear algebra.

That's also why GPU are so effective at neural networks. There is a lot of the same maths.

ukuina• 2 years ago

The name is a reference to a fictional gameshow in Bojack Horseman called "Hollywoo Stars and Celebrities: What Do They Know? Do They Know Things?? Let's Find Out!"

https://bojackhorseman.fandom.com/wiki/Hollywoo_Stars_and_Ce...!

echelon• 2 years ago

I adore that show so much. I have this sticker on my laptop.

If you haven't seen Bojack Horseman, it's funny and heartfelt. Palpably existential. If that kind of thing speaks to you, you owe yourself a watch.

In terms of a complete animation package, I think it easy outdoes Futurama. There's so much relatable depth to it. It hits hard, but it stays lighthearted enough to make you feel good about it.

As it turns out, I'm now working on filmtech, so that "Hollywoo" sticker fits me even better now.

the_af2• 2 years ago

I want to like Bojack and I know it has drama and heartfelt moments (watched all of season 1 I think), but in my opinion it's undone by its moments of "wacky" humor. I don't even mean that there are humanoid animals and humans living together, no explanation -- I can embrace that. I mean the Simpsons/Family Guy kind of humor... I could do without it and I think it would make the show better.

Or maybe it does get better after season 1? Since everyone seems to love it.

jeffparsons• 2 years ago

It hits hard, but it stays lighthearted enough to make you feel good about it.

That wasn't my experience of it. It's a brilliant show, to be sure, but at some point I felt the bleakness, particularly around Bojack's inability to help himself, became too much for me. Maybe different parts of the show resonate in different people.

khedoros1• 2 years ago

but it stays lighthearted enough to make you feel good about it.

There were plenty of parts that were dark enough to make it difficult to watch. There were lighthearted parts, but that's not a description I would've applied to the show as a whole.

saberience2• 2 years ago

Just a warning to anyone who read this comment, the show (in later seasons) gets incredibly dark, existential, and depressing, to the point where I couldn't watch it anymore as it made me both anxious and depressed.

Anyone prone to anxiety or depression or sad thoughts in general should probably avoid, it's really that depressing, and I can imagine it making suicidal people even more suicidal.

reso• 2 years ago

Not gonna lie I upvoted this post based only on the title.

_____-___• 2 years ago

I reference this specific game show title quite a bit, but I don't think that many people get it sadly, so I just seem weird haha

jncfhnb2• 2 years ago

It’s for the best. We wouldn’t want this getting too commercial.

lysecret• 2 years ago

I have found my people. I watched this show like 6 times haha

maximus-decimus• 2 years ago

Or as they keep calling it in the show, HSaCWDTKDTKTLFO.

Them spelling the whole acronym as if it was short might be my favorite running gag in the show.

mindcrime• 2 years ago

Damn, now I'm suddenly finding myself wanting to go back and re-watch Bojack Horseman. Not that that would be a bad thing.

vslira• 2 years ago

Not to be a skeptic or anything, but how do we know normal maps etc weren’t enriched into the datasets by the image gen companies?

I understand this paper links to open source model where that can be verified, but maybe this is one secret sauce of these more advanced models?

dougmwne4• 2 years ago

You would need to train on pairings of normal map images to source images. To my knowledge, that’s not a common training technique and this ability seems to bridge across several open models.

coffeebeqn• 2 years ago

It’s also kind of cool if it “understands” a normal map and that it maps to specific 3D geometry. People generally use 3D tools like Zbrush to sculpt the details to normals. Probably some people can draw them too

auggierose2• 2 years ago

So how do they get the normals, for example? Are they generated by the AI before the image is generated in order to generate the image, and they are just reading them off some internal state?

corysama3• 2 years ago

The training data is just billions of {RGB image, text description} pairs. So, it appears the model figured out how to make the normals as part of the process of making the images.

Or, are you asking how the researchers extracted it?

I-LoRA modulates key feature maps to extract intrinsic scene properties such as normals, depth, albedo, and shading, using the models' existing decoders without additional layers, revealing their deep understanding of scene intrinsics.

auggierose2• 2 years ago

Yes, that sentence "modulates key feature maps" doesn't tell me anything. What do they mean when saying they extract the normals?

dougmwne4• 2 years ago

Yes, there’s rudimentary evidence that there’s essentially a 3D engine within the model that participates in generating the image. If we could inspect and interpret the whole process it would likely be bizarre and byzantine, like a sort of convergent evolution that independently recreates a tangled spaghetti mess of unreal engine, adobe lightroom, and a physical simulation of a Canon D5.

qingcharles• 2 years ago

Essentially similar perhaps to the 3D engine that a human brain runs that generates a single "3D" image from two 2D cameras (eyes) and fills in missing objects in blind spots, etc.

LarsDu882• 2 years ago

Reminds me of when I tried to extract G-buffers from my Unity High Definition Rendering Pipeline test project: https://www.youtube.com/watch?v=Fwtc694qNUM

I'm not sure if this paper is really proving anything though. There's a giant-ass UNET Lora model that's being trained here, so is it really "extracting" something from an existing model, or simply creating a new model that can create channels that look like something you'd get out of a deferred rendering pipeline.

After all, taking normals, albedo, and depth and combining them (deferred rendering) is just one of several techniques to create a 3d scene. Wasn't even used in videogames until the early 2000s (in a Shrek videogame for the Xbox! (https://sites.google.com/site/richgel99/the-early-history-of...)

What would really be awesome is to get a LORA model that can extract the "camera" rotation and translation matrix for these image generation models. That would really demonstrate something (and be quite useful at the same time).

joefourier• 2 years ago

If you look at the supplementary material, they do a test where they train the Lora with a randomly initialised Unet and it is largely incapable of extracting any surface normals as opposed to using the pretrained Stable Diffusion Unet - clearly showing the features of the model are relevant to its performance.

sfink• 2 years ago

I don't really know what I'm talking about, but doesn't this address that?

with newly learned parameters that make up less than 0.6% of the total parameters in the generative model

0.6% sounds like a small number. Is it measuring the right thing?

Certainly, I wouldn't expect the model to necessarily be encoding exactly the set of things that they're extracting, but it still seems very significant to me even if it is "just" encoding some set of things that can be cheaply (in terms of model size) and reliably mapped to normals, albedo, and depth.

(I don't care what basis vectors it's using, as long as I know how to map them to mine.)

LarsDu882• 2 years ago

0.6 percent of a model with 890 million parameters is still 5.34 million parameters.

That's still pretty big. Maybe big enough to fake normals, learn some albedo smoothing functions, and learn a depth estimator perhaps??

hn_throwaway_99• 2 years ago

I skimmed the paper, but a lot of it was over my head. As someone not very versed in image generation AI, can anyone help me understand? This sentence (which another commenter highlighted) appears to be the key part:

I-LoRA modulates key feature maps to extract intrinsic scene properties such as normals, depth, albedo, and shading, using the models' existing decoders without additional layers, revealing their deep understanding of scene intrinsics.

What exactly does "modulates key feature maps to extract intrinsic scene properties" mean? How were these scene property images generated if no additional decoding layers were added?

visarga3• 2 years ago

Say you have a neural network with 1B parameters, you add 5M more parameters spread around (LoRA), and continue training for a while but only the newly added parameters, not the base network. The result is a "modulated" network that would predict scene properties.

The interesting thing is that it only takes a few more parameters so it must be that the original network was pretty close already.

kenjackson2• 2 years ago

Great explanation! Wish I could upvote more as it took me from head scratching to “that makes total sense”.

educaysean• 2 years ago

This is good news for VR (or spatial computing). If the models understand the physical world as well as the paper shows, generating two projections of a scene does not sound like a difficult ask. Really excited for what's to come.

modeless2• 2 years ago

Ooh, so this can take real images and predict albedo and lighting! Please, someone use this to make relightable gaussian splatting scenes. Dynamic lighting would really expand the usefulness of 3D scans made from photos, and I haven't seen anyone get anything close to what I would call "good" results in that area yet.

IshKebab• 2 years ago

Can it definitely use real images? I imagine extracting the depth map from real images would be the most useful application if it can.

modeless2• 2 years ago

"Fortunately, diffusion models are not only powerful image generators, their structure as image-to-image models makes it straightforward to apply to real images."

It looks like they actually only tried depth/normal maps for real images, not albedo/lighting maps, but it certainly seems possible. Honestly depth maps often look impressive at first glance but typically if you actually try to use them for anything (e.g. reprojection, DoF blur) their hidden flaws instantly become super apparent, and they aren't as useful as you might think. Gaussian splatting reconstructions already do reprojection better anyway. OTOH albedo maps look weird, but relighting is extremely useful and important. I'm not excited about yet another way to generate approximate depth maps, but I am excited about relighting.

dvh• 2 years ago

I asked ai that was posted here yesterday to draw common european chub and virtually every single one had adipose fin.

nottorp• 2 years ago

Hmm. Facebook has had an option to create "3d photos" for a while. I think they're annoying but now I wonder how they're made, and if you really need a neural network to figure out the 3d part in a 2d photo.

anjc• 2 years ago

Would be interesting to see if the perceptive abilities of generative models are superior to human perception, when tested on optical illusions that humans are fooled by. E.g., do they correctly assess depth in a Ponzo illusion scenario

renonce• 2 years ago

So is this GPT for images? They take a generative model and apply finetuning via LoRA on some downstream task such as surface normal and conclude that these models intrinsically learn these representations, and find that they do better than supervised approaches?

I think this is awesome but maybe it’s not really that surprising given how this “generate and then finetune” approach has already worked so well?

leobg• 2 years ago

Yet again. Anyone who placed their bets on LiDAR for self driving is proven wrong. Cameras and neural nets will rule the day.

daxfohl2• 2 years ago

I wonder if you could do this on the NN-generating NN from https://news.ycombinator.com/item?id=39458363.

Then you could have it start to create a vocabulary and an understanding around what all these parameters and weights do, since to us they're just a bunch of seemingly random floating point numbers.

At which point we may be able to start creating NNs from first principles, without even requiring training.

HellDunkel• 2 years ago

Do we know the unet was never trained on gbuffers?

Generative Models: What do they know? Do they know things? Let's find out

Related Stories