StyleNeRF: A Style-Based 3D-Aware Generator for High-Resolution Image Synthesis

http://jiataogu.me/style_nerf/

by lnyan • 4 years ago

109 26 4 years ago

jiataogu.me

kingcharles• 4 years ago

GANs are getting creepier by the day.

nicoco2• 4 years ago

Do you mean their potential (mis)use is creepy, or are you genuinely scared by their "abilities"?

nlehuen• 4 years ago

This whole research domain is confusing to me. I do get that it's a hard and cool problem to solve and I do see why researcher could work on it on those grounds.

But I'm concerned about what solving this is useful for, especially with the focus on generating human faces. Obligatory Jurassic Park quote: "Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.".

I can imagine this technology to be used in the entertainment industry (movies, video games) as a way to save money on art. But it's quite obvious that this technology can be weaponized and used in the propaganda industry (if it isn't already).

So, this could save money in fun applications on one end, and provide means for mass opinion manipulation weapon on the other end. I am certainly biased but I'm not sure the recreative use case is worth the risk of the weaponized one (a bit like using tactical nukes for a firework show).

What am I missing then? Are there any other use cases that would make this technology actually desirable for the greater good or humanity, or is it just a weapon with some potential recreational use cases?

fxtentacle• 4 years ago

Cool idea :)

1. They generate a low-resolution position map using NeRF => Good camera and 3D control

2. They then upsample that with StyleGAN2 => Good features in the final 2D image.

So in effect, this is a StyleGAN2 network where the latent space has the 3D-related components split off into a different generator network.

egnehots• 4 years ago

that's weird that they mostly ignore the comparison to stylegan3: https://nvlabs.github.io/stylegan3/

it's not a nerf based solution but since the goal is high resolution image syntesis..

petesergeant• 4 years ago

subtitle: an amazing trip through the uncanny valley

harry-wood• 4 years ago

The twinkling children's TV style background music in that video is not making it any less creepy.

birdyrooster• 4 years ago

Interesting that the reflections in each eye vary so much between each other.

Mandelmus• 4 years ago

This might fix that: https://dorverbin.github.io/refnerf/

Allstar• 4 years ago

The fact that this was in part developed with the help of Facebook AI Research is kind of unsettling.

That said, the results are actually very impressive.

throwoutway• 4 years ago

Didn’t Facebook just get sued for illegally harvesting faces & tags of people? If derivatives of that data make it into research like this, where does that stand from a legal perspective?

blurbleblurble• 4 years ago

Kinda puts a spin on the "face" in "facebook"

thedorkknight• 4 years ago

It literally puts a spin on it, actually!

axg112• 4 years ago

It's only unsettling if you are completely unfamiliar with the machine learning field. FAIR publishes research in many ML/AI fields. They're responsible for plenty of advances and contributions.

truemotive• 4 years ago

I’m not sure that’s what was meant by unsettling…

motohagiography• 4 years ago

While watching it, I wondered whether the "this x does not exist," GAN sites also implies a subset of "this x must exist" and if looking at it that way could yield discoveries in other areas. e.g. This X must exist, where x was a solar system, protein, wave function, chemical substance, cell, transcendental number, etc. Naive view, but when you're extapolating from a field of arbitrary 2D data, it seems there are a lot of ladders you could climb with that.

axg112• 4 years ago

I see there are a lot of questions/doubts in the comments about the usefulness of such research. This type of research is the precursor to gaining the ability to generative photorealistic 3D worlds. What is that useful for? Anywhere we currently generate 3D environments, including movies, games, VR/AR.

StyleNeRF: A Style-Based 3D-Aware Generator for High-Resolution Image Synthesis

Related Stories