Large Scale Deep Learning – Jeff Dean [pdf]

http://static.googleusercontent.com/media/research.google.com/en/us/people/jeff/CIKM-keynote-Nov2014.pdf

by coderush • 12 years ago

260 43 12 years ago

static.googleusercontent.com

andrewcamel2• 12 years ago

Reading through documents like this really pains me because I it seems like such interesting work and I immediately want to understand it better, but then realize the time required to acquire the knowledge and experience necessary to understand and apply this technology is so great, that it almost seems like a waste of time. After all, think of all the things one could build in the 2 full-time years it would take to fully comprehend all of this to the point where it's useful in any practical application.

31reasons• 12 years ago

Its not as complicated as it seems. 2 full-time year might be required if you want to start in the PhD program for it but not for its applications.

andrewcamel2• 12 years ago

That's pretty comforting to hear. Do you know of any resources off the top of your head to learn this type of thing?

frozenport• 12 years ago

You should be complaining about the computing resources to train 24 hidden layers.

tonydiv• 12 years ago

GPUs make a lot of this tractable. Nvidia is actually offering some free compute time for researchers:

http://www.nvidia.com/object/gpu-test-drive.html

sytelus• 12 years ago

Here's another way to see: What have you been doing for past two years? Now imagine you had started learning this about 2 years ago. You would have been done by now and ready to tackle some of the most interesting problems instead of continue to do same boring stuff you had been doing for past 2 years. In 2016, come back here and look at this comment again :).

PS: For people who are saying you can "apply" DNNs in a day or learn it by a coursera course in 6 weeks - they are only very superficially right. Yeah, anyone can build ML model for a sample training data using tool in the same sense that anyone can compile sample code and have a working app. The problem is that most models don't work the first time as expected. The challenge lies in debugging the model and fix many of N possibilities to make it work. This is what working in ML is all about. It's like usual programming where it takes years of experience to debug the code and make it work for your purpose. The added twist in ML is that debugging is almost entirely statistical. When your model doesn't work, it doesn't work only in statistical sense. Your problem would be essentially that the model doesn't give expected answer this 12% of the time. For this 12% of the time, it doesn't work not because of some wrong "if" condition or misplaced subroutine call. The debugging is almost always statistical debugging - there are no breakpoints to put or no watch to set or not even exceptions. So it takes pretty solid background in statistics and probability to effectively work in ML. And yes, most likely it would take much more than 2 years.

kastnerkyle3• 12 years ago

Truly understanding this to the point where you are "caught up" with the field may take 2 years, but one of the big blessings of deep learning is abstraction. You can go from very high level "black box" approaches simply using and following example code from Torch, Theano, or Caffe, all the way to nitty gritty study of the details of various architectures, how to optimize them, and how to apply them.

Watching videos of presentations and reading slides is often much easier than comprehending papers, though ultimately the paper should have much richer detail.

Personal anecdote: Two years ago I just started learning about these things, coming from an undergraduate degree in electrical engineering. Now I am in graduate school for deep learning and AI working to push things forward, one small step at a time. It is totally possible to learn this stuff in a reasonable amount of study, and there are more free resources than ever. Note that I had a full time engineering job until 6 months ago... doing something totally different!

chestervonwinch• 12 years ago

There's not a lot of theoretical background beyond the basics of pattern recognition to get started understanding this stuff. Of course, pattern recognition requires some knowledge of probability, statistics, linear algebra, and vector calculus. There are books on pattern recognition that are rather friendly towards this prerequisite knowledge, though.

jskonhovd• 12 years ago

I disagree. I believe that the foundation you will learn in statistics, linear algebra, and other mathematics would be worth the effort to any computer scientist. You would enjoy the introduction Machine Learning courses on Udacity or Coursera.

cr4zy• 12 years ago

The second half of this video has an earlier talk that goes along with the slides: http://youtu.be/S9twUcX1Zp0?t=22m49s

nl4• 12 years ago

This whole slide deck is worth reading. A couple of highlights:

Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true.

Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rate of around 16%. The 2014 system had 24 layers and an error rate of 6.66%. Note that trained humans have an error rate of around 5%[1].

Page 50-57 talk about the miracle that is Word2Vec, and what is possible with that.

Page 60-70 talks about paragraph embedding. I haven't seen this published before.

Page 70-73 extends word/paragraph embedding for translation. I've seen a slide deck showing this works before, but I need to read the new paper cited there.

Page 74+ talks about cross-modal embeddings, especially the caption generation stuff. HN has had a few things on that over the past month or so.

[1] http://karpathy.github.io/2014/09/02/what-i-learned-from-com...

waterlesscloud• 12 years ago

"Anything humans can do in 0.1 sec, the right big 10-layer network can do too"

That's an interesting direction from which to view things.

Then AI progress can be measured by increasing that timeframe.

Though I suspect there are some pretty gigantic discontinuities in there. The things humans can do in 3-4 seconds are qualitatively different from what they can do in less than 1 second, for example.

Still, it's a useful perspective to keep in mind.

crazypyro• 12 years ago

There's also the issue that we don't even fully understand what the brain even does, so how can we claim that anything in this incomplete set of operations is possible with a computer...

sanxiyn• 12 years ago

Page 60-70 talks about paragraph embedding. I haven't seen this published before.

Paragraph embedding was published this year as "Distributed Representations of Sentences and Documents". http://arxiv.org/abs/1405.4053

nl4• 12 years ago

Thanks.

It's interesting how the state of the art is outpacing publishing.

From a quick scan that appears quite similar to the approach in papers like "Parsing Natural Scenes and Natural Language with Recursive Neural Networks" (2011)[1]. Edit: I see they cite this paper too.

[1] http://nlp.stanford.edu/pubs/SocherLinNgManning_ICML2011.pdf

jberryman• 12 years ago

Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding.

Actually, I'm not sure we can do those things in quite 0.1 sec. All I know about this is from around minute 9 from http://www.radiolab.org/story/267176-never-quite-now/ . One guest on the show even estimates thinking the simplest thought to be on the order of 0.25-0.5 sec.

rasz_pl• 12 years ago

maybe concious thought, most of the stuff in the brain is on autopilot and happens behind our backs, conciousness gets the results and just builds narrative (often rationalizing the results to make us feel better)

xfs• 12 years ago

"Anything humans can do in 0.1 sec, the right big 10-layer network can do too"

Actually this was argued by Connectionists in 1980s. It is called Feldman's 100-step rule:

The critical resource that is most obvious is time. Neurons whose basic computational speed is a few milliseconds must be made to account for complex behaviors which are carried out in a few hundred milliseconds (Posner, 1978). This means that entire complex behaviors are carried out in less than a hundred time steps. Current AI and simulation programs require millions of time steps.

Feldman, J. A., & Ballard, D. H. (1982). Connectionist models and their properties. Cognitive Science, 6, p. 206.

Strilanc• 12 years ago

Whelp, Page 54 blew my mind right out the window.

E(hotter) - E(hot) + E(big) ≈ E(bigger)
E(Rome) - E(Italy) + E(Germany) ≈ E(Berlin)

These things are linearly separable?!

abrichr• 12 years ago

Deep neural networks essentially transform input data into a vector space where the data is "easier" to model. So while the input vectors may not be linearly separable in the input space, the network learns how to transform the input vectors into a space where they are.

nl4• 12 years ago

Yes. Hence 'miraculous' Word2vec.

fuelfive• 12 years ago

These are great highlights, although your second point from [1] is a bit misleading. Humans have a 5% error rate because the average person can't differentiate between 150 different breeds of dogs and other similar nonsense.

brandonb2• 12 years ago

For those of you who want to learn the nuts and bolts of deep neural networks, Andrew Ng's tutorial on Unsupervised Feature Learning and Deep Learning is getting older but still great: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial

The best research results from 2014 and 2013 make less use of the unsupervised techniques than initially expected, so I would start by focusing on the below sections, which focus more on supervised learning with deep neural networks:

Sparse Autoencoder: Neural Networks, Backpropagation Algorithm

Building Deep Networks for Classification: Deep Networks: Overview, Fine-tuning Stacked AEs

Working with Large Images: Feature extraction using convolution

You'll need some background in matrix algebra, calculus, and probability to understand this. Having taken a previous machine learning course, although not strictly necessary, is probably extremely helpful--I'd recommend taking any standard course on ML on Coursera or Udacity, or going through any standard textbook.

EDIT: I almost forgot that Michael Nielsen (who wrote the standard textbook on quantum computation) is also writing a free online textbook on Neural Networks and Deep Learning. Chapters 1-4 are currently available and would get you pretty far: http://neuralnetworksanddeeplearning.com/

kastnerkyle3• 12 years ago

Bookwise, Yoshua Bengio, Aaron Courville, and Ian Goodfellow are nearly finished with their MIT Press book on deep learning: http://www.iro.umontreal.ca/~bengioy/dlbook/ . It is pretty strong on the true theory of what is going on in deep networks, and has fairly good intuition for how and why things work. Paired with the deep learning tutorials http://www.deeplearning.net/tutorial/, as well as the content from UFLDL it is a pretty strong foundation for advanced study.

Michael's book seems to target a more introductory level - a beginner might be better off to start with that, follow with Andrew Ng's ML course, which has a section on neural nets including an assignment implementing backpropagation, then continue with the deep learning book and the {deep learning, UFLDL} tutorials. This should be solid enough to at least read most of the cutting edge work and papers, if that is the aim.

Hugo Larochelle's youtube course https://www.youtube.com/playlist?list=PL6Xpj9I5qXYEcOhn7Tqgh... and Hinton's coursera course https://www.coursera.org/course/neuralnets are also great references.

brandonb2• 12 years ago

Didn't realize Yoshua & co had a book coming! That would definitely be the one to read.

BTW, for anybody who wants to learn machine learning in general, Kyle's blog also seems to be packed full of clear explanations with working demo code: http://kastnerkyle.github.io/

Very nice!

ehurrell• 12 years ago

Thanks for the resources, my current hobby project is attempting to recreate deep learning in Clojure, have been burning through material, cheers!

agibsonccc4• 12 years ago

I know distributed neural nets are 10000 miles out there for most, but I just want to add a few nuggets for those considering it.

I work with distributed deep nets quite a bit. It's a different animal than training on a GPU.

I am working on benchmarks with my framework deeplearning4j now.

That aside, a few neat references/projects that will be digestible for people.

For those of you already in neural net land, there's a few key takeaways when doing distributed neural nets:

parameter averaging across mini batches

(depending on the algorithm) adagrad

momentum

[1] Project Adam: http://www.wired.com/2014/07/microsoft-adam/ [2]: Associated Paper: https://www.usenix.org/system/files/conference/osdi14/osdi14...

[3]: Hogwild algorithm: http://www.eecs.berkeley.edu/~brecht/papers/hogwildTR.pdf

[4]: A variation of this I use called Iterative Reduce done by my partner Josh Patterson: https://github.com/jpatanooga/KnittingBoar/wiki/Iterative-re...

[5]: Sandblaster LBFGS by Dean and Co. http://research.google.com/archive/large_deep_networks_nips2...

pacala• 12 years ago

This video uses the same slide deck: https://www.youtube.com/watch?v=vvK-XOiKXOs

forgotAgain• 12 years ago

I wonder how many machines are chomping on raw data generated by chrome, android and search. What is Google trying to learn and what does it already know?

mr_overalls• 12 years ago

Does Deep Learning on this scale offer any obvious benefits for genomic analysis? Does it make sense to use data from the 1000 Genomes Project (or similar large-scale sequencing effort) and perform association studies?

ivan_ah• 12 years ago

HN readers in the Montreal area will have a chance to listen to the talk in person at the McGill colloquium:

Scaling Deep Learning, Wednesday, December 10th, 2:00PM-3:00 PM at the McGill University M1 amphitheater of the Strathcona building at 3640 University Street.

Large Scale Deep Learning – Jeff Dean [pdf]

Related Stories