Large Scale Distributed Deep Networks (by Jeff Dean et al)

http://research.google.com/archive/large_deep_networks_nips2012.html?

80 39 14 years ago

research.google.com

Also submitted yesterday, by someone else, but it got buried: http://news.ycombinator.com/item?id=4775644

I resubmitted since I'm pretty sure this is of interest to many HN readers. Examples of why it's interesting include:

1. Google's deep learning work is now being used to power Android voice search (http://googleresearch.blogspot.ca/2012/08/speech-recognition... ); and

2. Dean claims that "We are seeing better than human-level performance in some visual tasks," in particular, for the problem of extracting house numbers in photos taken by Google's Street View cars, a job that used to be done by a large team of people (http://www.technologyreview.com/news/429442/google-puts-its-... ).

dumitrue6• 14 years ago

It's not obvious to me what it means to have "better than human-level performance" since most of the time the ground-truth itself is defined by humans :)

dsl• 14 years ago

One example I can think of is a computer could read a house number or a street sign from 100 feet away, where a human with good vision might be able to make out the same text at 20 feet.

jasonwatkinspdx• 14 years ago

In production you may use a process that's more economized than what you'd use to establish a ground truth in research. So the human team in production will have some error rate that is tolerated or won't include as much redundancy. In this sense it's easy to imagine better than human performance from the software, in the sense that the human performance you get isn't a single max value, but rather a function of budget.

robrenaud• 14 years ago

You could also ask many humans and average their response to obtain a golden label, and see how well any particular human agrees with the average. If there is a lot of variance in the human answers, then it's possible for a machine to have better than (individual) human performance, even on a human labelled data set.

fudged712• 14 years ago

CATCHAS, ironically, often seem to require "better than human-level performance" for recognition

ArbitraryLimits4• 14 years ago

> "We are seeing better than human-level performance in some visual tasks," in particular, for the problem of extracting house numbers in photos taken by Google's Street View cars

Can someone explain to me how this is news, given that the handwritten addresses on snail mail envelopes in the US have been OCR'd by neural networks for more than twenty years now?

confluence3• 14 years ago

The order of complexity difference between recognising figures on plain paper perpendicular to a scanning device in a controlled environment, and doing the same thing on huge amounts of non-standard chaotic data is why.

tgflynn2• 14 years ago

The USPS address recognition technology (at least as of 2 years ago when I was working on it) is not human-level. A fraction of the images cannot be resolved and are still sent to human keyers at rec sites. This fraction has been decreasing steadily over the years but it has not yet fallen to zero.

It's important to remember that performance is critical when talking about machine perception. OCR, handwriting recognition, face recognition, etc. can all be done but at what level of accuracy ? At least until very recently machine performance on these tasks has fallen well short of human level abilities.

ajays2• 14 years ago

Handwritten addresses aren't fully OCRed for the system to work. I worked on the first systems that were released (in the 90s), and the basic algorithm was as follows: first, try to read all the numbers in the address, and identify the ZIP and the street number. Now, given the ZIP and the street number, the number of possible street names is very small (on average, 4 or 5); this is done via the USPS's address database. Now the problem becomes one of matching the handwritten street name with one of these 4-5 names. (Of course, there's more to it, but this is the gist of it).

Last I heard, the percentage of handwritten mail successfully sorted by a machine had reached in the low 80s. The Russian company Parasoft (who also worked on Newton's online handwriting recognition) has been the leader in this field.

bravura• 14 years ago

I did my postdoctoral research on deep learning, and got into it back when Geoff Hinton's work was just an unpublished tech report.

So if anyone has any questions on it, I will try to answer.

e98cuenc3• 14 years ago

Is there any open source ML framework that includes support for DNN?

Do you know of any tutorial that may guide the beginner using DNN, I have no idea how to choose the number of hidden layers and activation functions.

Thanks!

dumitrue6• 14 years ago

http://deeplearning.net/software/theano/ is a good place to start. It's open source, in Python, has a few tutorials that lead you towards some rather state-of-the-art methods.

There's no secret sauce for how to choose the number of layers and the activation functions (and anyone that tells you otherwise is lying). It's all application-dependent and the best way to do it is by cross-validation.

Can answer questions about this topic in conjunction with mr. bravura (I'm working with the code referenced in that paper by Dean et al., here at Google, and did grad school at the same place as bravura did his postdoc).

seiji2• 14 years ago

Ask me again in two weeks when my online course in advanced machine learning for extreme beginners kicks off. We won't start with deep learning immediately, but it's in the pipeline not too soon after launch.

ArbitraryLimits4• 14 years ago

Please tell me whether the term "deep learning" has any meaning at all, and if so, what is it? It sounds like a buzzword to me, like "data scientist."

dumitrue6• 14 years ago

To some extent it's true that "deep learning" is a buzzword, though it is less arbitrary that "data scientist" (which makes no sense whatsoever). In a nutshell, what people mean by "deep learning" is the collection of tips and tricks that make it possible to efficiently train multiple layers of non-linear hidden units (or feature extractors).

Sometimes these are multi-layer neural nets, but they don't have to be.

krickle• 14 years ago

Something I have been curious about for a bit; how do you include textual data as input into a NN?

dumitrue6• 14 years ago

Here's a paper describing the main idea behind doing this: http://ronan.collobert.com/pub/matos/2011_nlp_jmlr.pdf

In a nutshell, you learn a vector of real-valued parameters for each word in your vocabulary. To train a network on sequences of words, you represent said sequence as a concatenation of the vectors of these words, and feed it as an input to the network.

To learn these vectors, you define the problem of "language modeling" as that of discriminating between two sets of sequences: S1 and S2. S1 is the set of sequences that occur in Wikipedia (of which there are many) and S2 is the same as S1, but where you replace a word in each sequence with a randomly chosen word from your vocabulary (which makes it, with very high probability, an invalid sequence of words).

Basically, by learning to discriminative between "good" and "bad" English word sequences, you can learn a language model of sorts. The model is represented by those vectors for each word.

You can then project those vectors into 2D, as bravura did a while ago, and look at what is close to each other: http://www.cs.toronto.edu/~hinton/turian.png

junto2• 14 years ago

With regards to distributed processing of neural networks, how do you split up the neural network onto the various nodes? Do you cut it up into sections and then compute the combined weights from each nodes outputs?

ajays2• 14 years ago

My NN knowledge is minimal, but from what I recall, the claim always was that you don't need more than 1 hidden layer; and too many hidden neurons results in over-fitting.

What changed?

tgflynn2• 14 years ago

The claim that you don't need more than 1 hidden layer is based on a mathematical theorem that says roughly that a sufficiently large 3 layer net exists that can fit any sufficiently smooth function. One has to look in detail at things like "sufficiently smooth" and "sufficiently large" when applying mathematical theorems to real problems - and that's a step practitioners often seem to neglect when looking for rules of thumb. Also just because a net exists doesn't mean that any given training algorithm is likely to find it.

As for overfitting the best way to reduce it is to use more training data and I believe the nets discussed in these papers have been trained on some of the largest training sets ever used.

The other problem with deep networks is that they have been considered very difficult to train with backpropagation due to vanishing or exploding gradients. I think the recent major algorithmic developments have mainly involved methods to mitigate these problems.

feniv• 14 years ago

This is written by THE Jeff Dean, who is essentially the programming equivalent of Chuck Norris at Google ( http://www.quora.com/Jeff-Dean/What-are-all-the-Jeff-Dean-fa... )

samg_2• 14 years ago

A year ago, I would have no idea what this was about. I'm very thankful that I live in a world where I can take high quality classes from Coursera that have given me the foundation to at least understand the abstract. :)

junto2• 14 years ago

Which courses did you take out of interest?

I struggle with the mathematics used in neural networks. I can understand code but as soon as I start to see calculus my brain freezes over. Does anyone know of a good online course that can give me a crash course in the mathematics required for neural networks?

My CS bachelors covered this, but I was lazy and drank too much beer. Now 20 years later I want to understand it properly.

dave_sullivan• 14 years ago

Do the theano deep learning tutorials. And keep hacking away at the math--you need it, but it eventually sinks in and becomes reasonably intuitive. Starting with code helped me grasp the math (I'm also much more comfortable reading code than math).

samg_2• 14 years ago

I've taken Andrew Ng's Machine Learning class, Daphne Koller's Probabilistic Graphical Models class, Dan Jurafsky and Christopher Manning's Natural Language Processing class, and currently Geoff Hinton's Neural Networks class.

I have spent a lot of time on Khan Academy to learn the calculus. In my experience you can get by with a surprisingly small amount of calculus, but it happens to be a small amount from a high level.

For example, backpropagation is just repeated application of the chain rule. Did take a while to get a handle on the derivatives, but it's worth it.

Large Scale Distributed Deep Networks (by Jeff Dean et al)

Related Stories