New AI Imaging Technique Reconstructs Photos with Realistic Results

https://news.developer.nvidia.com/new-ai-imaging-technique-reconstructs-photos-with-realistic-results/

by dsr12 • 8 years ago

219 70 8 years ago

news.developer.nvidia.com

subcosmos3• 8 years ago

They're at it a-GAN!

merpnderp• 8 years ago

GAN is just a brilliant concept. I look forward to all the new tools to come out of this.

p1esk3• 8 years ago

They didn't use GAN for this.

SlowRobotAhead• 8 years ago

The example with the eyes being removed works nicely on the attractive girl, it adds sexy eyes back in.

Works less nicely on the old man, where it adds the same sexy eyes.

subcosmos3• 8 years ago

It's trained on celebrities, hence the bias :p

azinman2• 8 years ago

The left eye (her right) doesn’t look realistic to me

apendleton• 8 years ago

It looks weird when they only do one, but when they do both it becomes symmetrical again and looks less weird.

glandium• 8 years ago

Note it's not any old man, it's Ernest Borgnine, who played Dominic Santini in Airwolf.

exhilaration• 8 years ago

In case you only read the HN comments, don't miss the amazing video from the article: https://www.youtube.com/watch?v=gg0F5JjKmhA

acmecorps• 8 years ago

I thought I was the only one. Came for the news, stayed for the comments.

khuss2• 8 years ago

Nice breakthrough. What could be some potential applications?

jackweirdy• 8 years ago

The first thing that comes to mind is restoring the images that were recovered (damaged) at the end of this process (follow through to part 3 for full examples of images):

http://www.bbc.co.uk/rd/blog/2017-12-morecambe-wise-video-fi...

https://www.bbc.co.uk/rd/blog/2017-12-morecambe-wise-video-x...

http://www.bbc.co.uk/rd/blog/2017-12-morecambe-wise-film-rec...

coolspot• 8 years ago

Deep fakes

halfjoking• 8 years ago

Hate watermarks? Remove them with this service and no one will know you got your image from 9gag...

khuss2• 8 years ago

Clever idea

gabblegabble• 8 years ago

Demosaic JAV!

JoeCoder_2• 8 years ago

How long until I can use this as a Photoshop plugin?

ttoinou2• 8 years ago

Years

gefh• 8 years ago

There's been content aware fill for a long time: https://youtu.be/Ge9jsJZ3lA0?t=245 It's not the same backing tech but for practical purposes it's as good.

ttoinou2• 8 years ago

Current inpainting algorithm is good for removing wires or adding textures. The new nVidia technique should be able to synthesize new information not in the original image

JoeCoder_2• 8 years ago

I doubt Adobe's content aware fill could replace someone's eye or a few of the other examples in the video.

Jach• 8 years ago

About -10 years ago (at least that's about when a gimp script doing a version of this was put out, Smart Remove / Heal Selection -- https://www.youtube.com/watch?v=3h1gZJsjKxs -- I recall Adobe demoing similar stuff at various points though I suppose this is the first time someone's used a generative neural net to fill things in rather than just nearby pixels)

dnautics• 8 years ago

They keyframe in the YouTube link is creepy because when they obliterate the eye section there's maybe weak linkage? They don't talk to each other and fill in different eyes on each side.

daveguy• 8 years ago

Keyframe is only half creepy. The left eye was the one filled in by the algorithm. The right eye is filled in similarly a few seconds later. For human faces it seems to substitute very generic replacements of features (and take no more cue from the surrounding photo than: eye shaped thing should go here, chin shaped thing should go here, brown hair should go here, etc. The video is definitely worth watching. For human faces it seems to take less cue from surroundings than inanimate scenes. Although it probably just seems that way because we are so sensitive to peculiarities in images of the face.

Edit: it's somewhere between a surrounding texture fill and a semantic / context based reconstruction. Texture fill would produce blank skin for an eye. Ideal reconstruction would take into account appropriate wrinkles, symmetry, expected bone structure. It works better for still life / scenes than for faces.

potlee• 8 years ago

Can someone who understands this tell us what is the catch here?

taeric2• 8 years ago

The catch is it is more like an artists interpretation. Anyone expecting that it will truly fix old damaged pictures to be like they were is going to be disappointed. Anyone just wanting to fill in some gaps will be excited. (Same for just procedural generation of some images, I suppose. Would be neat to see what it can do from a very low fidelity outline of a house/forest/etc.)

taneq• 8 years ago

As opposed to it being truly magical, I guess?

johndough• 8 years ago

You'll probably hit memory limits if you go much beyond 512x512-sized holes.

Additionally, computation times grow quickly with higher resolution and you already need a high end GPU for this resolution to get a reasonably interactive response time.

You'll also need a favorably licensed pretrained model or a few 10000 training images and masks.

So all in all, I can't see any deal breakers, but I'd probably still use PatchMatch instead.

rhcom22• 8 years ago

For reference the GPU they're using for this paper is the NVIDIA V100 GPU, a datacenter GPU costing $8,000.

hanrelan• 8 years ago

Since they programmatically generate the masks you wouldn't need those, just the set of training images. So it wouldn't be too hard to find since you're not looking for paired images, just a bunch of images of faces/landscapes/whatever you're trying to inpaint.

devit• 8 years ago

Any actual software?

Seems amazing for face shot retouching.

lepouet• 8 years ago

Adobe often show things like that, maybe a little less advanced... This is nice with a small picture in a video, but for now you always end up disapointed when you try on high res pictures where quality is important...

fudged71• 8 years ago

Having used photoshop content-aware-fill quite a bit, what was very interesting in the video with the eyes and the hairline was that it wasn't just filling in with what was already in the image, but also what it thought should be in the image. It's not just locally aware but also... world aware (if given the proper training data).

state_less3• 8 years ago

This is pretty cool. I'm looking forward to the day that not only can we reconstruct what was removed, but reconstruct what's not in the photo (image sensor), but could be inferred. Can we reconstruct the 3D scene and fill in the parts the camera can't see? For example, if it's a front view of a person, make a 3d model of a person and texture the backside clothing.

It seems like if we put together all the advancements these deep inference engines are producing, we may be able to reconstruct a reality, and allow people to walk around in it using VR glasses, or less advanced 3D shooter style controls.

It could be like a live earth view, any photo with a timestamp can contribute to this view and we'd create a sweet 4D model of earth across time and space.

vincnetas• 8 years ago

This is not reconstruction. It's guessing from what 'i' have seen before. So this would never reconstruct anything unseen. In a broad sense.

state_less3• 8 years ago

I think I follow what you're saying, we aren't going to be able to construct what aliens look like on a distant planet (or some other unknown). We're essentially looking at the information we have, which is why I expect continued interest in exploring for new information. Even as we continue to improve our reconstructions of 4D.

nartz• 8 years ago

Partially correct, but not necessarily true. Speaking extremely high level, the model is currently trained to fill in the blank space with something like the most probable options per pixels, based on the training set data. However, it is conceivable that it could be trained to also, say, insert an object into a scene, based on other characteristics found in the scene. Inotherwords, it could be trained to maximize a joint goal, where the second goal involved generating an object.

robotresearcher• 8 years ago

Like so:

http://vision.princeton.edu/projects/2016/SSCNet/

There's similar work at (at least) Berkeley and Stanford.

state_less3• 8 years ago

That's cool. I liked the use of scene rendering to supply training data to the network.

It'd be nice to see texture prediction on some of the voxels, so painting the occluded voxels in the scene as well as texturing those in the image.

Texture accuracy could be measured by rendering the other side of the bed and see how close the texture predictions were.

Now this would be quite a challenge, but if you could train a network to give D, given RGB, you'd have RGBD and could maybe use internet video to create some structure. Use something like a SLAM algorithm to get camera position, then detect when a model is viewed from the occluded side and get a lot of texture prediction data using real world internet video.

dschuetz• 8 years ago

Whenever Nvidia issues a research paper on how well their products work for AI applications like that in their paper I feel like instantly buying some of their more expensive scientific products.

agumonkey7• 8 years ago

personal opinion: I feel we're reaching the existential uncanny valley. computation can "do" everything and .. I feel dead inside thinking about it.

ttflee• 8 years ago

I feel that I would be waiting in the queue to buy a GPU to expand my own brain someday.

agumonkey7• 8 years ago

I'm super not sold on this. I've learned so much without any help, even when crippled by illness. I believe the superhuman thing is just a marketing trick to pursue the tech growth.

randyrand• 8 years ago

you're not alone.

agumonkey7• 8 years ago

Now it might be highly dependant on my age and life events. It shapes my perception of past, present and future. But I'm really not excited by the future tech sale. All I see is waste.

matte_black• 8 years ago

The thing to remember about uncanny valley is that it is a valley, and once you cross through you will start rising up again to the next peak. That, is where peaceful bliss awaits, where technology works so perfectly it is no longer unsettling or scary. That is the goal, and that is where we are heading more and more. The uncanny valley will soon be behind us.

arbie• 8 years ago

Unless the edge of the valley we crossed was actually The Cliff of Insanity and there _is_ no end to it.

SubiculumCode• 8 years ago

I don' know so much about valleys and such. I just think about how people rely on visual and audio evidence and how we might reach a place where it can be hard to guarantee that we are not being manipulated with doctored materials. This may ultimately be the utility of any kind of technology that documents all steps in processing from camera capture on. (erm blockchain?)

ashraymalhotra2• 8 years ago

Is the code for this open source?

bob_theslob646• 8 years ago

I'm guessing a whole line of business will be created from software like this existing? Some sort of validation engine which would verify if the photo is realistic or shopped.

I understand that you can tell in some of the examples in the video, but as it gets better, it maybe really hard to distinguish real from fake.

acdha• 8 years ago

This industry already exists (search for “photo forensics”) and I'm certain you're right to suspect it'll be booming soon. ML should also be good at faking some of the characteristics which current tools use so we must be looking at an arms race for years.

One of the big challenges I'm expecting will be the categories of attack: it seems plausible that we'd be able to limit abuse in the case where the original image is made by someone who isn't malicious by making it easy for viewers to find the originals (perceptual hashing with some sort of distributed ledger or signature system, and getting major services like Facebook to use it) but I haven't seen a convincing suggestion for how to deal with the case where the original is created by the attacker and thus any validation system would only show what they submitted it with. It seems like that'd fall back on much more failure-prone techniques — e.g. you could rely on public information to convince most people that, for example, Barack Obama didn't pledge allegiance to ISIS at a public rally but most people aren't going to have enough rock-solid documentation to prove a negative. If an attacker said that politician X was having an affair in hotel room it'd probably seem convincing to many people unless they screwed up and left proof of e.g. using stock footage, landmarks from a different city, wrong time of year, etc.

ashraymalhotra2• 8 years ago

How would it work if the mask was covering the mouth region? Afaik teeth have always been difficult to reproduce...

Viliam1234• 8 years ago

Here is some prior art: https://en.wikipedia.org/wiki/Censorship_of_images_in_the_So...

hogehoge• 8 years ago

Try it http://blog-imgs-80.fc2.com/o/p/t/opticalillusion48/shinoda_...

New AI Imaging Technique Reconstructs Photos with Realistic Results

Related Stories