Decoupled Neural Interfaces Using Synthetic Gradients

https://deepmind.com/blog#decoupled-neural-interfaces-using-synthetic-gradients

by yigitdemirag • 10 years ago

130 26 10 years ago

m1ck• 10 years ago

Is this a big deal?

alphonse232• 10 years ago

ANN are not that big of a deal, IMHO, when you compare them to other machine learning techniques i.e. Support Vector Machines. Also, see https://en.wikipedia.org/wiki/Artificial_neural_network#Theo...

Though, this article is so well presented, it deserves an awards for how pretty it is.

Anm2• 10 years ago

My gut thinks this sort of training alternative to back propagation has a lot of uses where SVM have no applicability. The article talks a lot about RNNs (neural nets for sequence prediction), but I would guess it would have uses in online learning as well. Learning twice as fast in those situations seems pretty significant to me.

edmack• 10 years ago

This area is a big deal - ML networks need to be much deeper and denser to provide human-level understanding, and training networks is currently a considerable bottleneck.

visarga2• 10 years ago

Does this method make it easier to spread a neural network over multiple GPUs/machines? I mean, does it reduce the amount of data being communicated between compute nodes or just decouples the updates from the need to wait for the rest of the net to finish?

imh• 10 years ago

Do most of the images not load for you guys too? (nevermind, works now!)

jjn2009• 10 years ago

looks fine for me

Anm2• 10 years ago

Refreshed the page and it worked.

Houshalter2• 10 years ago

None of the images load for me, even after many refreshes.

nicklo• 10 years ago

Super cool stuff in this paper.

At its heart, this is a new training architecture that allows parameter weights to be updated faster in a distributed setting.

The speed-up happens like so: instead of waiting for the full error gradient to propagate through the entire model, nodes can calculate the local gradient immediately and estimate the rest of it.

The full gradient does eventually get propagated, and it is used to fine-tune the estimator, which is a mini-neural net in itself.

Its amazing that this works, and the implications that full back-prop may not always be needed shakes up a lot of assumptions about training deep nets. This paper also continues the trend of this year of using neural nets as estimators/tools to improve the training of other neural nets. (I'm looking at you GANs).

Overall, excited to see where this goes as other researchers explore the possibilities when you throw the back-prop assumption out.

Houshalter2• 10 years ago

This is a big deal. One of the weaknesses of NNs is their time/memory complexity. Having to store every previous state and iterate backwards through them to the beginning of time, to learn. And if you want to learn "online" then you need to do that at every single step. It's unlikely the human brain works that way, and that has been one of the big arguments for why the brain can't use something similar to backprop.

visarga2• 10 years ago

It has been proven that sparse backwards connections, even with tied weights, can substitute backprop. In biological systems the feedback routes can't be the same and share connections with the feed forward routes.

taliesinb• 10 years ago

Are you referring to http://arxiv.org/pdf/1411.0247v1.pdf ? That doesn't mention sparse backward connections, but it does show that feedback weights that aren't computing the actual derivative dloss/din can till support learning. The network 'learns to learn', so to speak.

tlarkworthy• 10 years ago

wait... the model can be linear. So then it's just the second order terms of the error gradient? Or the Jacobian or something?

hacker422• 10 years ago

It will be an approximation of the Jacobian because it is just backprop. It also seems they only use the linear model when they also condition on the correct class which makes things easier. It is also only MNIST (10 classes).

mdda• 10 years ago

Hmmm - I'm also thinking that this is one of those things that probably has a much better explanation - and the science/maths will (hopefully) backfill why it works so well.

I half-remember from somewhere that as long as the gradient descent direction has the correct sign 'in expectation', then the SGD will ~work. So there's a whole lot of flexibility in there for having a good idea that at least doesn't fail horribly.

For instance, in other DeepMind work, they do lots of asynchronous weight updates - and the accuracy decrease from ignoring any kind of 'locking' is dwarfed by the speed increase of being able to run more stuff in parallel.

Another image I can't shrug off is that of Q-learning in a game, where the updates implicitly pass back from 'the future' (which also works ~better than it should). In this case, the linear model would just be an estimator of where the update values are going to land...

ebalit• 10 years ago

HOGWILD [1] lock-free weight updates are a great example of how forgiving SGD can be.

1 : https://arxiv.org/abs/1106.5730

lqdc13• 10 years ago

Anyone knows how they generated the illustration images?

Seems much better than relying on the client to render cpu intensive JS.

maxjaderberg• 10 years ago

Keynote for the static ones, After Effects for the gifs

Decoupled Neural Interfaces Using Synthetic Gradients

Related Stories