Deep image prior 'learns' on just one image

https://dmitryulyanov.github.io/deep_image_prior

by singularity2001 • 9 years ago

782 224 9 years ago

dmitryulyanov.github.io

scentoni• 9 years ago

Enhance. http://knowyourmeme.com/memes/zoom-and-enhance

trhway2• 9 years ago

i find it impressive how it placed a lamp over the library window. After that i was expecting a vase with flowers to be placed on the table in the next, palace shot :)

nametube2• 9 years ago

could you highlight which part this is ?. I think it can only fill in patterns that already exist in the image. I see a big artifact over the window but no lamp.

trhway2• 9 years ago

well, probably it is that deep (i hope) neural network inside my skull that interpreted the artifact as the lamp.

bitL3• 9 years ago

Wow, so just the weight sharing architecture does so much already? I am wondering if the same could be done with LSTMs on sequences or CNNs on voice...

cs7027• 9 years ago

I'm wondering the same thing too.

Note also that this finding strongly suggests that neural net architecture actually is quite important, possibly even more important than having more data -- which contradicts the conventional wisdom!

jacquesm6• 9 years ago

There is some pretty strong evidence for this: all the toddlers in the world. You only need to show them something once and they'll immediately be able to recognize more examples of the same thing from different angles and even when it is partially hidden. All they have to guide them is the structure of their brains, not the quantity of data they have been exposed.

scotty793• 9 years ago

It's not about neural network architecture. CNNs are taught by presenting them overlapping pieces of image. To speed things up and keep things orgsnized this is not done sequentially but in parralel making multiple neurons share weights but this is just a trick.

So what makes this result possible is not the architecture of NN in CNN but rather architecture of C. That allows us to get multiple samples from single image. The rest is just that actual content of the image is easier to learn then the noise.

Brain is almost nothing like CNN.

Jokeris• 9 years ago

Do not Echo State Networks already do that?

cdumler• 9 years ago

Red Dwarf - "Uncrop" - http://www.dailymotion.com/video/x2qlmuy

sarreph2• 9 years ago

Actually... How cool would it be to have an NN that could extend an image's background with plausible scenery? Not just photoshop 'smart' fill, but for example if it detected a building on the right side of an image, it could draw the rest of it? :)

nl• 9 years ago

This exists.

This approach takes other photos of the same scene to extend a cropped pic (see also MS PhotoSynth): http://grail.cs.washington.edu/projects/sq_photo_uncrop/

This uses a GAN to fill in missing parts of a pic. Those parts could be on the edge of the picture (although the paper doesn't explore that): http://hi.cs.waseda.ac.jp/~iizuka/projects/completion/data/c...

giobox• 9 years ago

Some of these image restoration techniques are extremely impressive, but its odd to think how much visual history will be faked in tiny ways if these "fix" techniques become widespread in commonly used photo software.

I suppose this is no different really than the extent to which images are edited anyway now, in that many images are never really a fastidiously accurate representation of the original scene, but interesting to consider none the less.

justonepost• 9 years ago

Cool but I think a learned method is more interesting, especially how it can be used for image compression.

foota2• 9 years ago

This is amazing.

loser777• 9 years ago

An interesting question this raises is--what determines when the training has overfitted to the noise/error in the input image? If that's another NN that provides a perceptual model of quality, how was it trained? ;)

BatFastard2• 9 years ago

How can it possibly know what was in the white areas of the library? Is there a residual image?

Seems impossible that it guesses correctly.

azeirah• 9 years ago

It doesn't guess "correctly" at all. Zoom in on the image, and focus on the filled-in areas, they look really blurry. It just doesn't look very bad from a birds-eye view.

eutectic2• 9 years ago

Probably this is a consequence of least-squares training; the 'average' solution is extremely unlikely.

knolan• 9 years ago

Somewhat similar to content aware fill in Photoshop [0]. The untrained network can latch onto frequent patterns and match them to holes in the data.

Why doesn’t it paint everything white? Are these actually transparent images or are they somehow tagged?

[0]https://helpx.adobe.com/photoshop/using/content-aware-patch-...

theoh2• 9 years ago

It seems like it should be similar in capability to a wavelet-based approach to e.g. image inpainting. In other words the neural network architecture essentially establishes a basis of features onto which the image is projected. Gabor wavelets are known to occur in the human visual system as the 'image elements' in a similar setup -- Gabor wavelets are kind of optimal, but clearly the features that this nn architecture uses are pretty effective too.

alexlarsson• 9 years ago

It cannot know, but it can extend what’s around it to something that “looks believable”. I don’t fundamentally think it is different from asking an artist to complete the image. You’ll get an image, but no actual information about what was under the white.

nametube2• 9 years ago

It doesn't it repeats/generates pixels that minimize E, so the behaviour heavily depends on the choice of E. for inpainting E is the difference between their generated x and the true x but they mask out the error in the white areas. The model gets good results by just copying nearby pixels.

Zedmor• 9 years ago

because your brain runs on same CNN software i.e. doing pattern matching, not image matching.

eutectic2• 9 years ago

This is as close to magic as humanly possible.

sarreph2• 9 years ago

Safe to say that Arthur C Clarke would count this as 'sufficiently advanced technology' then I guess :)

cs7027• 9 years ago

Wow:

"In this work, we show that, contrary to expectations, a great deal of image statistics are captured by the structure of a convolutional image generator rather than by any learned capability. This is particularly true for the statistics required to solve various image restoration problems, where the image prior is required to integrate information lost in the degradation processes.

To show this, we apply untrained ConvNets to the solution of several such problems. Instead of following the common paradigm of training a ConvNet on a large dataset of example images, we fit a generator network to a single degraded image. In this scheme, the network weights serve as a parametrization of the restored image. The weights are randomly initialized and fitted to maximize their likelihood given a specific degraded image and a task-dependent observation model.

We show that this very simple formulation is very competitive for standard image processing problems such as denoising, inpainting and super-resolution. This is particularly remarkable because no aspect of the network is learned from data; instead, the weights of the network are always randomly initialized, so that the only prior information is in the structure of the network itself. To the best of our knowledge, this is the first study that directly investigates the prior captured by deep convolutional generative networks independently of learning the network parameters from images."

PS. This makes me wonder whether and to what degree the structure of the brain's connectome might be a necessary prior for AGI.

jmmcd3• 9 years ago

PS. This makes me wonder whether and to what degree the structure of the brain's connectome is a necessary prior for AGI.

Well, I wouldn't mix up AGI and AGI by deep learning, and more important I would emphasise that this is a good prior for images. The fundamental insight in CNNs and eventually in this work is that there is a correlation between pairs of nearby pixels. We have something similar for video and audio processing, but nothing remotely similar for other more abstract intelligence tasks.

cs7027• 9 years ago

Thanks. I'm not mixing them up! I'm just wondering whether and to what degree architecture, i.e., network structure, will prove important for other, more advanced AI tasks, including up to AGI.

otakucode2• 9 years ago

Well, there is correlation between 'nearby' things of all types. An apple is an 'apple' because we encountered the word and the thing in close proximity to one another temporally and perceptively many times during our life. All knowledge is this. Tremendously overlapping and filled with mistakes (spurious correlations induced by coincidence and happenstance), but fundamentally just that same 'insight'. Our brains are association machines. Once sufficient quantity of associations exist in a brain in a body living in a world we recognize, we call the property that results 'consciousness'.

Unfortunately there is no reason to think that you can substract the 'in a body living in a world we recognize' part and end up with anything like the same property. All those associations come from perceptive inputs, and result in motive outputs, with the biofeedback that occurs being of profoundly fundamental importance (heck, put a pen in your mouth which causes your mouth to 'smile' and you will 'feel happier'). While of course there is definitely a possibility that a machine-based intelligence could be conscious, it would doubtless have to be very different from anything we could recognize, derived as it would be from its own input, output, and the feedback between the two.

felippee• 9 years ago

This is great work. It shows that some of the "amazing" results of deep learning are not as deep and don't even require learning!

In my view this also sheds some light on the GAN's and their ability to generate "real looking images". Perhaps there is much less to generating "real looking images" then everyone attributes. E.g. in this work, the network clearly knows exactly nothing about the world and generates good looking in-paint.

mcny• 9 years ago

The results are amazing, at least to me!

xaedes• 9 years ago

"the network clearly knows exactly nothing about the world"

Well, except the network structure itself.

narrator• 9 years ago

The brain probably has a lot of hard wired networks to recognize faces and to not like being cold for example. I think even feral children have these capabilities.

eanzenberg5• 9 years ago

The “task dependent observation model” also comes out of thin air, or is it trained with lots of data?

sanxiyn5• 9 years ago

Observation model is trivial and data-free. For example, just mean squared error for denoising, mean squared error of downscaled image for super-resolution, etc.

goialoq• 9 years ago

I don't understand their editorializing

contrary to expectations,

ingore the weasel words

a great deal of image statistics are captured by the structure of a convolutional image generator rather than by any learned capability.
To show this, we apply untrained ConvNets to the solution of several such problems. Instead of following the common paradigm of training a ConvNet on a large dataset of example images, we fit a generator network to a single degraded image.

How is that _untrained_, if they are training it on an image? Is a generator network different from a ConvNet?

This is particularly remarkable because no aspect of the network is learned from data;

but they just said "we fit a generator network to a single degraded image"

As others have commented, they appear to be training on all the parts of a single complex image, and using that training to repair local deviations from the global average. This may be better than classical denoising algorithms because the NN can model the image structure better than other approaches, but this doesn't seem like a novel use of NNs.

lern_too_spel• 9 years ago

The point is that a fully connected network would not get you the same results. I.e. the convolutional structure allows it to learn interesting things from a single example because by choosing a convolutional architecture, you are implicitly imposing a prior.

kalal• 9 years ago

Agreed. The network is trained in a sense "online" on the target image and directly applied to it. I don't see much impact of this work outside image denouncing or tasks already presented in the paper.

jpster• 9 years ago

Does this mean huge datasets are no longer a prerequisite for this type of computing? Leveling the playing field for smaller teams who may no longer have to rely on Google- or FB-sized datasets?

cdancette2• 9 years ago

I don't think so. They are basically loosing entropy by going from the corrupted image to the original one that's more predictable.

I don't think you can do other things like labeling by using the same method

stanfordkid• 9 years ago

So are they saying that the topology of a deep-net is intrinsic to "reality" ... somewhat analogous to something like the Fibonacci ratio for organic forms?

adrianN• 9 years ago

The Fibonacci thing is mostly a myth perpetuated by confirmation bias tough.

jon_richards• 9 years ago

Maybe more like a fourier transform.

vog• 9 years ago

Does this also explain - on a higher level - why AlphaGo Zero was possible and so successful?

jeeceebees• 9 years ago

Alpha Go Zero is actually more similar to a GAN: https://arxiv.org/abs/1711.09091

amelius5• 9 years ago

But I see one problem. Say the image is a photo of a driveway. The gravel on the driveway will look like noise. Will the algorithm now smooth out the driveway, effectively removing the gravel?

At least a "learned", context-aware approach can prevent this from happening in theory.

pasta• 9 years ago

I think you are right.

But as a brush in a photo editor program it would be great. Then a used can 'mark' what areas need to be smoothed and the result will be great.

rsp19842• 9 years ago

In the examples they show an image of a woman next to a table. The table is covered with a table cloth. In the corrupted image the table cloth looks like all noise to the human eye, however in the reconstructed image you see that the method is able to recover most of the pattern in the cloth.

rullelito• 9 years ago

Can someone explain this in a way so that an ordinary mortal computer scientist can understand it?

vbarrielle• 9 years ago

Can someone explain this in a way so that an ordinary mortal computer scientist can understand it?

I'll try.

Instead of the common approach that tries to search for image pixels to minimize e.g. a denoising objective, they realize that they can instead search for the weights of an image generator network such that the generated image matches the objective.

They argue that the structure of the network then constitutes some prior knowledge over what a natural image should look like.

My (probably wrong) interpretation: since a convolutional neural network essentially works by looking for some spatial patterns at different resolutions, their optimization process boils down to finding the high and mid-resolution patterns that best match the input image, and then re-using that information to fill in the missing information (or replacing "noise" that does not match the extracted pattern).

fny• 9 years ago

There exist neural network structures that can solve the image processing problems shown when weights are randomly applied.

This means that for these problems the solution is encoded entirely in the network structure and not the weights.

We could use linear regression as an analogy: y = b f(x).

This is akin to saying that for some class of problems, it's the case that you can get as good of a solution from the function you pick alone (x^2, log(x)) as you could from picking a function and then computing the best coefficient. Note this isn't true for linear regression for any meaningful problems, but I feel like it's a decent analogy.

denzil_correa2• 9 years ago

There's a great explanation of the idea here in the comments : https://news.ycombinator.com/item?id=15820994

notaboutdave• 9 years ago

Here's a psuedocode rundown of exactly what it does:

  network = new NeuralNetwork()
  
  targetOutput = readFile('./images/zebra-500x500-corrupted.jpg')
  input = generateNoise(500, 500)

  while(iterationCount < overfitThreshold) {
    networkOutput = network.getOutput(input)
    loss = getLoss(targetOutput, networkOutput)
    network.backPropogate(loss)

    iterationCount++
  }

  writeFile('./images/zebra-500x500-denoised.jpg', networkOutput)

amelius5• 9 years ago

Since a convolutional network applies parameters (filters) to different parts of an image evenly, this means that this approach still learns, except not from a corpus of images, but from the image itself (i.e. one part of the network learns from different parts of the image).

This makes me wonder whether and to what degree the structure of the brain's connectome might be a necessary prior for AGI.

Convnets have been invented to speed up learning (reusing parameters). The human brain (afaik) doesn't contain such speedup. Instead it relies on massive parallellism of neurons, and thus trains all "filters" separately instead of reusing them. Therefore, I suppose this approach would not apply to humans.

amelius5• 9 years ago

Another way to think of it:

Convnets are a method to speed up learning by sharing filter-coefficients. In other words, one could just as well train the network by replacing the convnets by fully-connected layers, except that of course training would take a lot longer.

If one views convnets as purely a computational optimization, then the approach of the paper can be considered a "hack" that only works because of this optimization.

kkylin• 9 years ago

Probably more than structure (which I took to mean connectivity) -- there are a lot of things that affect (biological) neuronal computation in addition to connectivity, such as synaptic weights, distribution & types of receptors for different neurotransmitters, and the (sometimes very different) timescales associated with different neurotransmitters. Not easy things to measure.

jonmc12• 9 years ago

PS. This makes me wonder whether and to what degree the structure of the brain's connectome might be a necessary prior for AGI.

I've seen an analogy in hardware that is potentially interesting to explore.

There is a technique in A/D converters (ADC) where input signal noise can be reduced by RMS averaging the sequence of digital outputs. This increases the effective resolution of the ADC, while lowering the effective sampling frequency of the input signal.

In the case that the input noise of a precision ADC is very low, the digital averaging has no effect on effective resolution. Surprisingly, in this case you can actually inject a "hand-crafted" noise source into the input, and still realize the resolution improvements from digital averaging. The key is to find the right amount of noise to randomize the quantization noise.

In the case of the ADC, the structure of the quantizing hardware is known, as is the characteristics of quantization noise. This allows for a guassian noise to be injected with the input source (might be analogous to a "prior"). From there, one can balance the input injection noise with the output digital averaging to get an optimal increase in output resolution.

This technique is explained well in this paper: http://www.analog.com/en/analog-dialogue/articles/adc-input-...

otakucode2• 9 years ago

I have been reading about fractal image compression (don't sue me patent trolls, I haven't implemented anything I promise!) lately and so this doesn't surprise me very much. I also wouldn't be surprised if it turned out that what is happening inside the network is essentially similar in many respects to how fractal image compression works.

Fractal image compression is inherently pretty simple in concept: Find portions of the image that are similar to transformed other portions of the image. Store the things needed to transform one portion into the other, and patch together enough of those to cover the whole thing. Then you can throw out the image itself. You only need the ways in which the parts can be transformed into the others (affine transformations along with brightness/contrast shifts usually). Once you have those, you can literally start with _any_ source image and iterating the application of the transformations are guaranteed to result in something very close to the original image.

A CNN 'rediscovering' this technique feels intuitively like a very natural thing to occur, and the iterative images presented there smack of early iterations of a fractal image 'decoding' from a blank source image. The connection, of course, could be utterly specious and I am just guessing. I am intrigued, however, as I've been wanting to investigate using deep learning to perform VHS video capture cleanup as a side project for awhile now.

JumpCrisscross2• 9 years ago

Can someone break this down for this layman?

oh_sigh• 9 years ago

Neural networks are powerful without even training them. By merely designing the structure of them you are creating something.

theoh2• 9 years ago

Truly, an occasion for this koan:

Sussman attains enlightenment

In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.

“What are you doing?”, asked Minsky.

“I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied.

“Why is the net wired randomly?”, asked Minsky.

“I do not want it to have any preconceptions of how to play”, Sussman said.

Minsky then shut his eyes.

“Why do you close your eyes?”, Sussman asked his teacher.

“So that the room will be empty.”

At that moment, Sussman was enlightened.

http://www.catb.org/jargon/html/koans.html

ska5• 9 years ago

Creating is perhaps better states as “asserting”

joshmarlow• 9 years ago

This kind of reminds me of what I took away from Gerald Edelman's work ([0]).

The gist as I remember it was that he theorized that the cortex is composed of assemblies of neurons formed in development (in the womb, potentially with epi-genetic factors). Each assembly, just by chance of it's structure, is likely good at something. Over time, assemblies take up the tasks they happen to be good at and further specialize with experience.

[0] https://en.wikipedia.org/wiki/Neural_Darwinism

cs7027• 9 years ago

The structure of convolutional neural nets specifies much of the prior knowledge necessary for learning. In other words, the design of these neural nets makes a lot of correct assumptions about the nature of images (stationarity of pixel statistics, locality of pixel dependencies, and so on).

nerfhammer2• 9 years ago

is the structure of these CNNs learned or designed?

Do they run some kind of optimizer to learn the optimal CNN structure or does some person sit down and pick structures to include in it?

ska5• 9 years ago

Which in turn relies on older work.

I wonder if anyone has looked at what Mumford-Shaw implications would look like projected onto a CNN?

dwaltrip• 9 years ago

By structure, we are simply referring to the number of layers, the number of neurons in each layer, and the specific connections between neurons in each pair of neighboring layers, right?

So in this paper, they carefully chose a certain structure, set the weights randomly, and then what happened after that? I understand that they did not then train it with a training data set, but I'm not quite getting what they did with the single distorted image.

fizixer• 9 years ago

By structure, do you mean the architecture of the NN?

Meaning number of hidden layers, nodes per layer, and their connectivity with each other and with the input and output layers?

flyingspork2• 9 years ago

Not an expert so take this with a grain of salt; I could be misinterpreting the paper.

It seems that the current accepted method is to train a network with distorted images as the input and the correct undistorted images as the targets. Then after training you can feed a new distorted image into the trained network and get the estimated "fixed" image.

However this team actually uses the distorted image as both the input and the target to the net. So if they were to let the training go on for too long the network will produce an exact copy of the distorted input image. But for some reason, the structure of the network means that the estimated output learns realistic features first, and then overfits to the noise afterwords. So if you stop the training early, you get an image that incorporates realistic features from the distorted image, but hasn't had time to "learn" the noisy features.

jboggan• 9 years ago

This is fascinating because I've been running into something similar with sequence to sequence models translating natural language into Python code. I got better results stopping "early" when the perplexity was still quite high, I thought it was a little crazy.

Mahn• 9 years ago

for some reason, the structure of the network means that the estimated output learns realistic features first, and then overfits to the noise afterwords.

Why does this happen? What characteristics does the network structure have that cause this effect?

syrak• 9 years ago

So if they were to let the training go on for too long the network will produce an exact copy of the distorted input image.

It won't. The objective (min (E(x, x0) + R(x)) they are trying to optimize (over output images x) amounts to a combination of:

- The output image x should "look like" the input image x0, this is the error term E(x, x0);

- The output image x should be "regular", this is the regularization term R(x).

The latter term prevents overfitting: R should be chosen such that noisy images for example are considered irregular (high R(x)).

toblender• 9 years ago

Wow thanks for the explanation, I would they would just use what you wrote for the abstract.

At first, I thought the machine learning algorithm was able to just bridge the gap in images from nothing...

CSI time: Enhance ... enhance... enhance!

cutcss• 9 years ago

Instead of looking for patterns from itself on others images (training data) it starts with noise and deforms such noise based on patterns found on itself and favors deformations closer to the input image; eventually reaching something close to the input image without the noise (cause the noise it's pattern-less or at least weak enough to die over stronger patterns)

JumpCrisscross2• 9 years ago

Does this say more about CNNS fundamentally or our design process for their structure? What sort of assumptions might one make when "structuring" a CNN?

urgoroger2• 9 years ago

Could be wrong too, ML is not my field.

Basically, a generator neural network has two things which affect its output: an input and some parameters (weights).

Let's use the setting of the task for denoising images. They use a network who's output is to is the denoised image, and compare it to the noisy image to get a score for how good the denoised output was.

Now the strange thing is, for the input of the network, they just use random garbage. The only thing they move around to try get a good denoising score are the parameters of the network (not the input).

They find that by only adjusting the parameters, even with fixed random crap inputs, if they find the parameter setting which minimizes the noise, they actually still get great looking results.

This suggests that the networks they tested this method with (other researchers work who do well on this task) are based much more in the inherent structure of the network, rather than the model refinement from training on thousands of images, since even given random crap, they generates good results, as long as the parameters are tuned well to the noisy image.

raz32dust• 9 years ago

This is incredible! Can't help but wonder if the brain does something similar to fill up "gaps" in reality. e.g how we fill up our perception (not only vision, but general mental intuition) based on just context.

sjg007• 9 years ago

Yes it does.

randcraw• 9 years ago

That's exactly what happens when the retina is damaged. The brain fills in the void imperceptibly so you aren't distracted by the deficit. But I think biology doesn't fill in the void synthetically. It's more likely that the brain "turns a blind eye" toward the missing 'pixels' and directs your attention elsewhere, perhaps to those remaining regions with high saliency (useful detail).

rasz• 9 years ago

Not exactly. Brain has prior knowledge (deep learning). Lets take a picture of a car as an example. License plate has letters/numbers, so even if it cuts lower half you will try to guess by fitting the most likely symbol combinations. Approach from the article has no prior knowledge, wouldnt be able to inpaint meaningful letters, nor guess most cars have wheels, bumpers, drive on the road etc.

chestervonwinch• 9 years ago

This is unbelievable that it works so well.

0x53-61-6C-74• 9 years ago

CSI:Miami...I'm sorry, I take it all back.

vbuwivbiu• 9 years ago

Remember fractal image compression ?

https://youtu.be/AjdogjBxfco?t=260

cztomsik• 9 years ago

Thx for that!

eitland• 9 years ago

Upvoted you both. I normally really don't like videos but this was worth the few minutes it took. (The link points to the middle of a 9 min video.)

ska5• 9 years ago

This can also be looked at as the original source of patch based denoising, etc. In the end it's about capturing the scaling properties and self similarity of natural images. This is also, for example, why wavelets were so effective as a basis.

David Mumford particularly did some great work on this sort of thing a couple of decades ago, along with many others. I hope when people are rushing around trying to apply convolutional nets to everything they aren't losing these insights.

fjsolwmv3• 9 years ago

The benefit of CNNs is like the benefits of SVMs -- they generalize all the great old techniques so you don't have to understand them all, you just throw more CPU at the optimization problem.

eximius• 9 years ago

Huh. This seems to boil down to 'noise is higher information entropy than realistic content; partial learning will learn realistic content before learning noise' or something like that.

eref3• 9 years ago

I think that can be mainly attributed to the fact that the last few deconvolutional features are overfitted to features in the image and are somewhat robust to noise. The network does not even learn features to produce e.g. white noise as output. This is probably much less magical than the paper makes it seem to be.

magoghm• 9 years ago

Interesting. Then you might achieve similar results by compressing the image as a JPEG with low quality settings.

killjoywashere• 9 years ago

The intel and espionage communities are going to be all over this. This makes the Soviet photo retouching look like child's play.

Let's say you want to start a war, and need some evidence of chemical weapons. Now, you can drop in some images of chemical weapons and claim a GAN found them. Sample press releases:

"We believe this photo was retouched to hide the chemical weapons. Using a GAN, we recovered clear photographic proof of the chemical weapons."

"We believe they are using this tin-roofed building to hide the chemical weapons. Using a GAN on our own satellite images, we recovered clear photographic proof of the chemical weapons."

ac2u4• 9 years ago

Can't see "we used a GAN" as an explanation that would fly. To counter it you can simply say "well, the GAN could be wrong."

The technique outlined above is useful because we as humans can eyeball the results and say "the improvement on that blurry zebra photo is great!", but we know that it's an improvement based on a networks hueristics that's good enough for us, not an exact replica of the information lost through noise and compression.

fenwick67• 9 years ago

I'll grant you that it makes a good scapegoat, but frankly this doesn't really matter... the US went to war with Iraq with bad intel before the machine learning boom.

m3kw9• 9 years ago

Am I missing something here? How does this apply to DNNs that tried to recognize images? Does it mean we can train a Net with just image now?

albertTJames• 9 years ago

ok.. this is revolutionary. Using the architecture as a way to capture an image prior hints at how network structure and captured invariance are related. By analogy it leads to thinking of brain areas as both hard coded prior knowledge through their natural arrangement and plastic learning structure. Turning the problem on its head shines a new light to how we could conceive network architectures.

andbberger2• 9 years ago

This shouldn't really be surprising. Machine learning is specifically not magic. The reason CNNs have seen so much success is precisely because they build in translation-invariance, which massively cuts down on parameters while forcing the final function to have the desired structure regardless of wherever gradient descent takes the weights.

Also why most papers in deep learning are network architecture innovation.

andbberger2• 9 years ago

One more relevant note - (Olshausen and Field, 1997) showed that the filter employed by V1 simple cells could be learned using some simple assumptions about sparse coding and a single image. Translation invariance built in by way of the sampling scheme of the image, small patches.

The filters learned by the first layer of CNNs is usually of the same type, Gabor filters. Not a coincidence.

That was twenty years ago. What's old is new?

eref3• 9 years ago

What do Gabor filters have to do with this?

krona• 9 years ago

Deep learning researchers rediscover Compressed Sensing. News at 11.

foota2• 9 years ago

I wonder if you could do something similar with audio

JoeCoder_• 9 years ago

And turn crappy music into good music.

zo7• 9 years ago

I feel like this is related to the information bottleneck idea that's been floating around for some time [1]. I only really understand both of these at a superficial level, but from what I think I understand one thing that they observed is that there are two phases when training a deep learning model: a phase which maximizes the mutual information (?) between the input and the output, and a compression phase which compresses the learned representation. In that light this work makes sense, since artifacts are essentially noise that the network would filter out in the fitting process.

Very cool work though.

[1] https://youtu.be/bLqJHjXihK8

freeflight• 9 years ago

That Zebra one looks like a "before and after" image with really good MSAA. I realize the techniques are quite different, it's still interesting how the result seems so similar on a superficial level.

gbrown• 9 years ago

So what kind of generative stuff does it "learn" from really really sparse data, or from random noise? Trained generative models can produce some crazy things, I wonder if something similar could work here.

Recursive geometric patterns perhaps?

zellyn• 9 years ago

Would a plausible explanation of this be that one part of an image tells you a hell of a lot about another part of it? And that ConvNet structure captures that really well?

psergeant• 9 years ago

Yes

ac2u4• 9 years ago

Question to anyone that knows this area in depth.

I'm assuming that this impressive feat has a disadvantage (over the traditional example intensive technique of training a CNN with something like ImageNet) in the form of taking a long time to generate the corrected image.

With that assumption in mind, could this new technique be reversed? As in feeding in sharp images, getting back out corrupted ones, for the purposes of generating data sets where there isn't much data to begin with?

You could then take that data and use it to train a more traditional CNN to sort of amortize the results of the technique in the paper and have the process happening faster.

wonderous2• 9 years ago

Do you mean generative adversarial networks?

https://en.m.wikipedia.org/wiki/Generative_adversarial_netwo...

ac2u4• 9 years ago

No not really, this isn't generation for the sake of classification. It's generation in order to generate a dataset to train a new network that exhibits the characteristics of the network in the paper that can be used quickly and in a more general fashion.

tgb2• 9 years ago

I don't see how any choice of a function g(theta) could have the property they desire, ie could eliminate R(g(theta)). Can anyone explain?

sanxiyn5• 9 years ago

It's expressed somewhat awkwardly, but what's going on is that R(x) is zero if x is in range of g, and infinite otherwise. Choice of g is such that natural images are in range of g, and non-natural images aren't.

tgb2• 9 years ago

Sorry I still don't understand. They require g to be surjective. Edit: in their paper they call it f_theta and it's explicitly not surjective. Dunno why their writeup is so confused.

ReverseCold• 9 years ago

I tried to run their ipynb but it wants something called skip. I don't think it's the "skip" on PyPi (that doesn't work anyway) - so what could they be using?

sanxiyn5• 9 years ago

skip.py is in the repository under models directory.

frihani• 9 years ago

Maybe it's the presentation of the restoration process, but I'm particularly impressed with the inpainting sample.

The idea of not running the simulation `past` the realistic interpretation and using that result makes sense but the results are way beyond what I would have expected!

Great work on the write up.

fpgaminer• 9 years ago

I'm finding it hard to put into words what I find wrong with this paper, but ... here goes nothing.

So, the novel thing here is that an encoder-decoder network applied to an image can learn enough from a source image to be useful. In some ways that's obvious, but the effectiveness of it on reconstruction tasks is certainly surprising.

I have two problems, though. One is that I would take the reconstruction results with a grain of salt. The examples are clearly lab queens, where the occluded regions are not particularly interesting/challenging.

Two is the conclusion the authors reach. Somehow the authors go from the novel discovery I describe above, to saying that somehow the architecture of the network is a prior.

Well ... I mean, yeah a network's architecture _is_ a prior. But it's not actually significant.

See, in the dark ages of machine learning we had only fully connected networks. They sucked. They'd always overfit and underperform or were impossible to train. Then we finally got convolutional networks, and suddenly a whole slew of machine learning problems became easier and that hurdled us into the current renaissance.

But, you see, convolutional networks weren't the _only_ reason for the dawn of this new age. Rather it was three major things: 1) Convolutional layers, 2) more data, 3) more computing power.

Some time after the "discovery" of convolutional layers we found out that, hey, our old fully connected networks actually _do_ work. If you give them enough data and enough computational power, you can get them to perform as well as state of the art convolution networks. The great thing about fully connected networks is that they assume nothing. That means A) you can theoretically get better results and B) you don't have to spend time designing an architecture.

So we already know that architecture isn't ultimately important. You can have a giant, fully connected network, and it _will_ work, if you feed it enough data and have the computational power necessary to train such a beast.

Convolutional layers are just simplifications which make training easier. They are priors in the sense that we know a fully connected layer in image applications would just devolve into a convolutional layer anyway, so we might as well start with a convolution layer. That "design" is the prior. But it's not mandatory; the network would still function without that "prior".

So ... I'm not sure how the authors are taking their research and using it to come to the conclusion that their results are because of some magical property imbued into the network by the "priors" of the architecture.

They apparently tried other architectures and got poor results, and so they use that to claim that architecture is the only reason their technique works.

That's like if you started with ResNet for a classification problem, tried other architectures, saw that they performed worse, and then published a paper saying that Residual Networks somehow embody the fundamental forces of natural images in their architecture, and that's why they work. When the truth is that ResNets aren't special, they are just easier to train.

Another example from the annals of machine learning history: time and time again when there is a breakthrough in architectures, it's usually followed a few years later by a simplification of the architecture. For example we started with networks like VGG which are these big, hand crafted architectures. Slowly over time architectures have become _less_ exotic, instead opting to simply define a basic building block repeated N times.

The reason for this is because in the intervening years we gather more training data, better training techniques, and more computational power. So we can instead use a more homogeneous architecture which has _less_ assumptions (less priors) and at the end of the day we get _better_ results.

I'll repeat that. We put _less_ priors into our networks and we get better results.

So on the one hand we have _all_ of machine learning history telling us that priors in architectures are _bad_. On the other hand we have this paper which makes some really weird logical leap from "we tried a few architectures, they were worse, so architecture is _key_ to machine learning and it's important because we need good priors built into the architecture."

Anyone remember hand crafted feature vectors? I do. Those were priors. Guess what happened when we got rid of them and used generic networks feeding directly from the raw data? Oh right, all of modern machine learning...

aqsalose• 9 years ago

Another example from the annals of machine learning history: time and time again when there is a breakthrough in architectures, it's usually followed a few years later by a simplification of the architecture. For example we started with networks like VGG which are these big, hand crafted architectures. Slowly over time architectures have become _less_ exotic, instead opting to simply define a basic building block repeated N times.
The reason for this is because in the intervening years we gather more training data, better training techniques, and more computational power. So we can instead use a more homogeneous architecture which has _less_ assumptions (less priors) and at the end of the day we get _better_ results.
I'll repeat that. We put _less_ priors into our networks and we get better results.

Is move from VGG-like architectures a question of moving to using less priors or just move to different priors that work better?

In addition, I'd claim that a choice of training method also imbues the whole ML system with prior knowledge. If we add a dropout layer, we wish to enforce a particular regularization scheme on the model because we know that we need to regularize to model to avoid overfitting. If we choose to use a particular optimizer instead of another, we use it because we know it has desirable properties on particular objective objection.

So we already know that architecture isn't ultimately important. You can have a giant, fully connected network, and it _will_ work, if you feed it enough data and have the computational power necessary to train such a beast.

I'm slightly unclear about your argument here. What is the point of training the fully connected network for a long time and with massive datasets to get equivalent results and structure as convnets if we already do know that convolutional structure with weight sharing is a good fit for natural images? Deep learning has been casted as a magical black box that produces impressive results "yet the scientists don't know why!" by the press. On the contrary, these results suggest there might be less magic in neural net models as the recent hype would suggest.

The examples are clearly lab queens, where the occluded regions are not particularly interesting/challenging.

To me they look quite like the standard examples used in inpainting and denoising articles.

warsheep• 9 years ago

Convolutional layers are just simplifications which make training easier. They are priors in the sense that we know a fully connected layer in image applications would just devolve into a convolutional layer anyway, so we might as well start with a convolution layer. That "design" is the prior. But it's not mandatory; the network would still function without that "prior".

As far as I know this is incorrect. Can you point to a paper that shows this? If by "easier to train" you mean that the models do not overfit training data, then that's the whole point of using correct priors / hypothesis classes.

I'm not sure what bugs you in this paper, but the point is that they decouple the prior architecture from the training/optimization mechanism, and that seems interesting.

blauditore• 9 years ago

As I'm not an expert in the field; what exactly does the term

  min_x E(x; x0) + R(x)

mean?

I thought that E(x; x0) would denote the error/difference between original and corrupted images, and R(x) be the (searched-for) correction. But this doesn't seem to make sense with the next parts of their explanation.

urgoroger2• 9 years ago

x is actually the generated image they are testing against x0. A lower E(x; x0) means an image which fits well towards the objective based on the original image (depends on the task). The paper gives some examples. For example, for the task of image denoising, E(x; x0) is just the squared distance of the generated (denoised candidate) image x to the pixels to the original image x0. Obviously you would want this to be low in the generated version since it should still look close to x0.

R(x) is a regularization term to avoid overfitting. For example, in the denoising example, it could be a measure of the variation in color of x. Clearly, just taking x = x0, the squared error (E(x; x0)) is 0, but it will have high R(x) because of all the noise. That's why they try to minimize both quantities combined, so we get min_x E(x; x0) + R(x), to get close to the objective but also not overfit.

https://en.wikipedia.org/wiki/Regularization_(mathematics)

sigterm• 9 years ago

I think x_0 refers to the corrupted image and x is the (denoised/upscaled/inpainted) image the optimization algorithm tries to come up. Without R(x) the optimal solution is simply x=x0. So we need a good R(x) that can measure how good or natural the solution image is. In this work the R(x) is constraining the valid x to those that can be represented by a CNN.

cdancette2• 9 years ago

You're right, and here, they're including the prior R(x) in the model architecture. So R(x) = 0 here,

and they pose x = f_theta(z), where f_theta is a neural net.

So they can avoid overfitting by stopping the training at the right moment.

So they're not really trying to minimize E(x, x0), but rather to get a small enough value, so that they obtain good results. A too small value means overfitting.

CamperBob2• 9 years ago

Is there a straightforward explanation of how to run this anywhere, for those not steeped in the arcane ways of Jupyter?

zerostar07• 9 years ago

This is interesting news to neuroscientists as well. Even without plasticity cortical networks might be performing useful functions. Plasticity may then perform some different function.

artur_makly• 9 years ago

can someone please summarize this for us luddites? also wasnt there an HN summary project out there?

therealmarv• 9 years ago

how do I get the code running easily? I'm familiar with python but not with ipynb at all. Maybe somebody can give me some easy to follow instructions, thx.

Quanttek• 9 years ago

Can someone put their results and explanation in laymen's terms please?

machiaweliczny4• 9 years ago

Isn't it simply hardcore overfitting of network?

I haven't read the paper yet but it seems to me that it shows you can use overfitting for output interpolation with great results.

Same should be possibile for example for interpolation of video images instead od pixels.

alexasmyths• 9 years ago

Truly congratulations! Amazing.

Deep image prior 'learns' on just one image

Related Stories