Understanding LSTM networks

http://colah.github.io/posts/2015-08-Understanding-LSTMs/

by michael_nielsen • 11 years ago

99 17 11 years ago

colah.github.io

mistermaster• 11 years ago

great explanation. many thanks! hochreiter is genius!

colah34• 11 years ago

hochreiter is genius!

It's really impressive to me how farsighted his work on LSTMs was.

In any case, I'm glad you liked the post.

p1esk• 11 years ago

Michael, this post nicely completes your book about neural networks. I was a little surprised you didn't write it yourself.

ambicapter• 11 years ago

Anybody know what he uses for his diagrams?

colah34• 11 years ago

All of the diagrams in this post were made in Inkscape, with the LaTeX plugin for equations.

reader50002• 11 years ago

Whats amazing to me is that if I understand correctly backprop still works. It is very odd that SGD on the error function for some training data is conceptually equivalent to teaching all the gates for each hidden feature when to open/close given the next input in a sequence.

colah34• 11 years ago

Whats amazing to me is that if I understand correctly backprop still works.

Yep! One computes the gradient with backpropagation and trains LSTMs on that.

It is very odd that SGD on the error function for some training data is conceptually equivalent to teaching all the gates for each hidden feature when to open/close given the next input in a sequence.

Agreed, it's pretty remarkable the things one can learn with gradient descent. I'd like to understand this better.

hyperbovine2• 11 years ago

I don't understand why this is remarkable, could you elaborate? You have a smooth map from inputs to outputs, you differentiate it via automated applications of the chain rule and use this to make the error function smaller. Why should this be remarkable?

ot• 11 years ago

colah, your posts combine a deep level of understanding and an exceptional clarity. These are both rare, especially in the cargo-cult-driven world of neural networks and deep learning.

I hope you keep writing as much as you can. Thanks!

d136o• 11 years ago

Thanks colah, that was a very readable walk-through. I've been making my way through Bishop's PRML ch 5 to get as much of a handle as possible on NNs, but your intro here to LSTM's makes me want to jump ahead and skip to the new stuff :)

Understanding LSTM networks

Related Stories