Neural Turing Machines (2014)

by rfreytag • 10 years ago

55 27 10 years ago

Interesting. Reminds me of old attempts on using NN's or ML techniques to derive programs from data kind of merged with fuzzy-logic controllers. Here's two links from the analog, Turing-complete NN's they cited in the paper:

http://binds.cs.umass.edu/anna_cp.html

http://research.cs.queensu.ca/home/akl/cisc879/papers/PAPERS...

It would be interesting to see designs like theirs and others we see posted trained until they got really good at application area, then synthesized into digital or analog NN implementations with plenty of resiliency. The components are so simple they can run with few gates or other components. Anyone studying NN's know what current state of the art is for synthesizing efficient HW from a given NN model? With or without ability to continue to learn/improve.

petra• 10 years ago

There was work done by Doug Burger from microsoft, spliting an algorithm to between a digital Cpu and a neural compute unit(simulated in analog). In algorithms that we're not bottlenecked by the digital cpu, they've even seen gains of 50x in perf/w.

But most algorithms they tested we're bottlenecked by the digital part, so the average benefit was ~3x.

But maybe if you start the design using this "turing neural network", you could see consistently large gains.

pron5• 10 years ago

Is it just me or is anyone else bothered by the lack of theory here? The results are great, but seem so heuristical and ad-hoc, like somebody tried lots of things and one seemed to work, but there's no clear idea as to why, and certainly no general theory tying things together.

visarga• 10 years ago

It's very experimental. The great thing about them is that they open the possibility of building new types of interfaces like memory, stack, tape, database or search engine that work with neural networks in inference.

See https://arxiv.org/pdf/1505.00521v3.pdf

zodiac7• 10 years ago

I can't speak to the RL methods used here, but wrt to the architecture they designed, I think it might seem less ad-hoc if you know that generally speaking new neural net architectures work well when they

1) can do some useful computation

2) with a good loss function

3) with a small number of parameters

4) in a differentiable way

5) where the gradients can flow

So for instance LSTMs are better than RNNs due to (5), convnets are good due to (3), etc.

argonaut7• 10 years ago

You've just numbered 5 different rules of thumb... That hardly supports your idea that things are not adhoc.

pron5• 10 years ago

Sure, but that's engineering (inspired by what theory of basic NN we do have), not theory.

Houshalter• 10 years ago

It's not really ad hoc. Neural nets are well studied, but they have very limited memory. They attempted to solve this problem by giving a neural net an unlimited external tape memory, inspired by Turing machines.

In order to train a neural net with backpropagation, every step has to be completely continuous and differentiable. So they made a continuous valued tape.

argonaut7• 10 years ago

inspired by Turing machines

This is the very spirit of ad-hoc engineering and trial and error: trying some architecture out because it resembles / has high-level connections to some other unrelated concept.

argonaut7• 10 years ago

Deep learning is very adhoc and experimental right now. Anyone reading through the literature would see that. That's why I'm pessimistic about AGI, but very optimistic about short-mid-term industry applications.

kikill• 10 years ago

funny, i was just reading this article on medium, about NTM https://medium.com/snips-ai/ntm-lasagne-a-library-for-neural...

albi_lander• 10 years ago

Thanks for sharing, very nice article indeed. Looks like the guy works here --> https://snips.ai/. Their product looks promising and branded as an "intelligent memory", would be interesting to know if they're using an NTM implementation in their product.

Neural Turing Machines (2014)

Related Stories