Deep Learning – Review by LeCun, Bengio, and Hinton

http://www.nature.com/nature/journal/v521/n7553/full/nature14539.html

by chriskanan • 11 years ago

128 44 11 years ago

nature.com

cowpig2• 11 years ago

Hurray for science behind paywalls!

IanCal4• 11 years ago

Here's a link where you can read it: http://www.nature.com/articles/nature14539.epdf?referrer_acc...

rfrey• 11 years ago

$32 seems steep for one article. Does anyone know if Nature allows authors to publish on their websites? Nothing so far on Hinton's, LeCun's, or Bengio's page.

cowpig2• 11 years ago

Having to pay for this at all is problematic.

robotresearcher• 11 years ago

why? Because the authors are not paid for their work, or some other reason?

chriskanan2• 11 years ago

This link allows you to access the entire document using ReadCube: http://www.nature.com/articles/nature14539.epdf?referrer_acc...

It doesn't allow you to print or to save the article.

gwern2• 11 years ago

https://www.dropbox.com/s/fmc3e4ackcf74lo/2015-lecun.pdf / http://sci-hub.org/downloads/d397/lecun2015.pdf

dmazin• 11 years ago

gwern saves the day once again.

nabla9• 11 years ago

http://libgen.in/scimag/?s=10.1038/nature14539

foxbarrington• 11 years ago

But think of how much all the research costs Nature.

IanCal4• 11 years ago

Their editor-in-chief estimates their costs to be ~$30-40,000 per article. Much higher than other publishers, it seems, probably in part due to their high rejection rate (92% in 2011).

http://www.nature.com/news/open-access-the-true-cost-of-scie...

thisisdave2• 11 years ago

Has LeCun changed his position on open access? He'd previously "pledged to no longer do any volunteer work, including reviewing, for non-open access publications." [1] A major part of his reasoning was, "Why should people pay $31 to read individual papers when they could get them for free if they were published by JMLR?"

I had assumed that meant he wouldn't write for them either (and thus wouldn't enlist other people to volunteer as reviewers when the final product would cost $32 to read).

[1] https://plus.google.com/+YannLeCunPhD/posts/WStiQ38Hioy

letitgo12345• 11 years ago

To be fair, it's a review article in nature -- so meant to advertise deep learning to scientists in other fields. His actual research is still publicly accessible.

grayclhn3• 11 years ago

"Review" in the context of your link means "referee." I don't think that most researchers would view writing a review article for Nature as "volunteer work" because the career benefits of the publication are pretty high.

So, going by the link, I don't think this is a change in his position on open access, but I also don't think that his position involved as much self-sacrifice as you'd assumed.

edit: I don't know the field well, but these review articles usually recycle a lot of material from review articles the author wrote a year or two before. The contents of the article might be basically available for free already.

thisisdave2• 11 years ago

I don't think that most researchers would view writing a review article for Nature as "volunteer work"

I was referring more specifically to the fact that Nature had to enlist volunteers to referee the paper he wrote.

I was curious whether he was okay with that even though he wouldn't volunteer himself, if his position on non-open journals had changed, or if there was some other explanation.

paulsutter4• 11 years ago

The publishers aren't the problem, and the authors aren't the problem. We the readers are the problem. Seriously. Let me explain.

I have to admit, when I saw "LeCun, Hinton, in Nature" I thought "That must be an important article, I need to read it". I haven't read every single paper by LeCun or Hinton. The Nature name affected me. That's why it's rational to publish there.

There's still no effective alternative to the journal system to identify what papers are important to read. There have been attempts, Google Scholar and Citeseer for example.

A voting system like HackerNews wouldn't work, because Geoff Hinton's vote should count for a lot more than my vote. Pagerank solved that problem for web pages (a link from Yahoo indicates more value than a link from my blog). How can scientific publication move to such a system?

hackuser• 11 years ago

How can scientific publication move to such a system?

Complete amatuer speculation: Scientists' professional societies could create elite, free, online journals, with a limited number of articles per month (to ensure only the best are published there), openly stating that they intend these to be the new elite journals in their respective fields.

dcre• 11 years ago

Karma could be used to weight votes.

Teodolfo2• 11 years ago

He won't do things for the non-open journals without getting something in return. In this case, he gets a Nature paper in return. He presumably would be willing to edit one for sufficient pay as well, he just won't do it for free.

yzh2• 11 years ago

There is a whole session on machine intelligence in this issue. Just curious, is there more CS-related research on Nature now than before?

chriskanan2• 11 years ago

Nature and Science both generally have only a few CS-focused papers per year. This issue has a special theme (Machine learning and robots), so it breaks that general pattern. I regularly read Nature and Science for general science news, commentary, etc.

yzh2• 11 years ago

Thanks.

evc123• 11 years ago

Nature should just use targeted advertising to make their journals free, similar to the way google/facebook make their services free using targeted ads.

grayclhn3• 11 years ago

I think you're overestimating the number of people that read Nature.

itistoday2• 11 years ago

Why do these articles on RNNs and "Deep Learning" never mention Hierarchical Temporal Memory?

https://en.wikipedia.org/wiki/Hierarchical_temporal_memory#D...

akyu• 11 years ago

It seems like theres a kind of taboo against HTM in the ML community. I guess it stems from their lack of impressive benchmarks, but I think that's kind of missing the point when it comes to HTM. Maybe HTM isn't the right solution, but I think there is a lot to be learned by using neocortex inspired models, and HTM is at least a solid step in that direction. And the work Numenta has contributed on Sparse Codings shouldn't be overlooked.

davmre• 11 years ago

I don't think most ML people are actively hostile to HTM, just indifferent until they show some results. For example, Yann Lecun in his Reddit AMA: http://www.reddit.com/r/MachineLearning/comments/25lnbt/ama_...

ajays• 11 years ago

I've yet to find one significant benchmark dataset where HTM beats other methods. One.

dwf• 11 years ago

Extraordinary claims demand extraordinary evidence. Numenta has plenty of the former and nil as regards the latter.

Even if they had invented some believably interesting task and done fair comparisons with other methods and shown that HTM succeeds where others fail, it would be considered worth a look by the wider machine learning community.

Teodolfo2• 11 years ago

Because it doesn't meet academic standards for publishing in NIPS and ICML, the most prestigious machine learning conferences.

Edit: To clarify, research papers generally cite other peer-reviewed research papers in similar venues preferentially. ML papers should mostly be citing ML papers in high-quality, peer-reviewed venues. HTM doesn't have papers like this to cite.

bra-ket• 11 years ago

These communities have different goals from the very beginning. Pattern recognition vs. real intelligence, which is what HTM is about. Hawkins describes this gap well in his book.

But there is some cross-pollination, see the recent projects by Stan Franklin's lab on Sparse distributed memory and composite representations, it's a step towards integration with deep learning: http://ccrg.cs.memphis.edu/assets/papers/theses-dissertation...

On the other hand check out the work by Volodymyr Mnih from DeepMind https://www.cs.toronto.edu/~vmnih/, reinforcement learning with "visual attention" is a step towards consciousness models of the HTM/SDM/LIDA camp.

paulsutter4• 11 years ago

I was also under the misapprehension that deep learning is just about classification, but that isn't true.

Yes, reinforcement learning is the path to general intelligence, and the deep learning community is showing impressive progress on that front as well. The Deepmind demo [1] and the recent robotics work at Berkeley[2] are good examples.

Thanks for the link to Stan Franklin's work. I'm glad to hear there is work to integrate the two approaches.

[1] https://www.youtube.com/watch?v=EfGD2qveGdQ

[2] http://newscenter.berkeley.edu/2015/05/21/deep-learning-robo...

paulsutter4• 11 years ago

Jeff Hawkins takes the position that only Numenta sees the human brain as a temporal prediction engine based on sparse, hierarchical memory.

But actually, RNNs are great for recognizing and predicting temporal sequences (as we saw in the Karpathy post), RNNs use a sparse representation, and RNNs can be extended with hierarchical memory [1]

The big difference is that the neural network crowd are getting some spectacular results, and Numenta, well, maybe they'll be show more progress in the future.

Jeff Hawkins is super smart and a good guy, and he might get more done if he acknowledged the commonalities in the approaches rather than having to invent it all separately at Numenta. I really dont mean to be critical. Jeff inspired my own interest in machine intelligence.

[1] page 442, "Over the past year, several authors have made different proposals to augment RNNs with a memory module. Proposals include the Neural Turing Machine in which the network is augmented by a ‘tape-like’ memory that the RNN can choose to read from or write to, and memory networks, in which a regular network is augmented by a kind of associative memory. Memory networks have yielded excellent performance on standard question-answering benchmarks. The memory is used to remember the story about which the network is later asked to answer questions."

paulsutter4• 11 years ago

If you only have time to read one paper on Deep Learning, read this paper.

A few quotes:

"This rather naive way of performing machine translation has quickly become competitive with the state-of-the-art, and raises serious doubts about whether understanding a sentence requires anything like the internal symbolic expressions that are manipulated by using inference rules. It is more compatible with the view that everyday reasoning involves many simultaneous analogies that each contribute plausibility to a conclusion"

"The issue of representation lies at the heart of the debate between the logic-inspired and the neural-network-inspired paradigms for cognition. In the logic-inspired paradigm, an instance of a symbol is something for which the only property is that it is either identical or non-identical to other symbol instances. It has no internal structure that is relevant to its use; and to reason with symbols, they must be bound to the variables in judiciously chosen rules of inference. By contrast, neural networks just use big activity vectors, big weight matrices and scalar non-linearities to perform the type of fast ‘intuitive’ inference that underpins effortless commonsense reasoning."

"Problems such as image and speech recognition require the input–output function to be insensitive to irrelevant variations of the input, such as variations in position, orientation or illumination of an object, or variations in the pitch or accent of speech, while being very sensitive to particular minute variations (for example, the difference between a white wolf and a breed of wolf-like white dog called a Samoyed). At the pixel level, images of two Samoyeds in different poses and in different environments may be very different from each other, whereas two images of a Samoyed and a wolf in the same position and on similar backgrounds may be very similar to each other. A linear classifier, or any other ‘shallow’ classifier operating on raw pixels could not possibly distinguish the latter two, while putting the former two in the same category.... The conventional option is to hand design good feature extractors, which requires a considerable amount of engineering skill and domain expertise. But this can all be avoided if good features can be learned automatically using a general-purpose learning procedure. This is the key advantage of deep learning."

"Deep neural networks exploit the property that many natural signals are compositional hierarchies, in which higher-level features are obtained by composing lower-level ones. In images, local combinations of edges form motifs, motifs assemble into parts, and parts form objects. Similar hierarchies exist in speech and text from sounds to phones, phonemes, syllables, words and sentences. The pooling allows representations to vary very little when elements in the previous layer vary in position and appearance"

deepnet• 11 years ago

Don't miss C Olah's blog where Figure.1 was copied from

http://colah.github.io/

Very very visually insightful on the nature of Neural Nets, Convnets, Deep Nets...

jphilip147• 11 years ago

Very Helpful review.

Deep Learning – Review by LeCun, Bengio, and Hinton

Related Stories