Deep Learning: An MIT Press book in preparation

http://www.iro.umontreal.ca/~bengioy/dlbook/

by mutor • 12 years ago

224 39 12 years ago

iro.umontreal.ca

jbarrow• 12 years ago

This looks like it has the potential to be a great resource! Not to mention it's coming out of one of the big name schools of deep learning (along with UToronto and Stanford).

I imagine, though, that anyone not well versed in college mathematics may have issues with the explanations. If you want a good introductory resource, but either haven't covered or have forgotten some of the math in this book, I would recommend one of two resources:

[1] MetaCademy: http://metacademy.org [2] Neural Networks and Deep Learning (In Progress): http://neuralnetworksanddeeplearning.com

The first will take you through all the math first through some online courses and textbooks, and the second is a good general purpose introduction that I recommend to anybody interested in neural nets.

justin66• 12 years ago

Pages look ok in Firefox, borderline unreadable in Chrome. I wonder what that's about.

thomasfoster96• 12 years ago

They've converted a PDF to be shown in HTML. Possibly using PDF.js or something.

felixr• 12 years ago

Looks like they used pdf2htmlEX (https://github.com/coolwanglu/pdf2htmlEX)

plg• 12 years ago

this is great, I hope the authors keep a .pdf version available for dl once its done

nfoz• 12 years ago

Why is "deep learning" so hype nowadays? They seem like just another AI tool that will be prone to over-fitting datasets and provide an analysis that is difficult to mathematically characterize and understand in a reliable way.

Not here to be cynical/negative -- they might be of great value, this is not my expertise. Can someone explain why deep learning articles are receiving attention rather than, say, Support Vector Machines / kernel-based methods of pattern analysis? Or other nonlinear analysis? Are they related?

marcodena• 12 years ago

http://www.slideshare.net/hammawan/deep-neural-networks :)

shock-value• 12 years ago

I think you are right in your sentiment that deep learning is becoming over-hyped. I've written before that it's mostly a method of brute forcing the problem of AI that happens to work very efficiently on modern GPUs and other parallel processing units. That being said, for the tasks it does well on (mostly facial and other types of simple image recognition), it does extremely well compared to everything else out there. (And it may very well end up as a part of a future architecture that better simulates general intelligence.)

However, I don't think it is reasonable to assume that AI tools can only be valuable if they can be rigorously mathematically characterized. The "holy grail" of intelligence -- that is, actual human intelligence -- certainly can't or at least hasn't been mathematically characterized, and I don't think anyone will ever offer any "proof" (mathematical or otherwise) that the biological brain is inherently wired to arrive at or at least tend to converge to correct solutions to intelligence problems. And of course no one will argue that human intelligence isn't valuable!

eli_gottlieb2• 12 years ago

If my machine learning algorithms went wrong as regularly as the term "human error" denotes, I wouldn't be able to use them to run much of anything. Identifying faces, for instance, is just a matter of statistical accuracy, of pattern-matching, but a more complex task like automated asset-trading or world domination requires a program that will make fewer mistakes than even many trained humans.

halfcat• 12 years ago

The "holy grail" of intelligence -- that is, actual human intelligence...

When people talk about human-level AI, are they referring to specific beneficial subsets of human intelligence, like holding a conversation or interacting with the world around them, or does it literally include all of human cognition? In other words, would the "holy grail" include AI that acts extremely irrational and makes poor decisions based on temporary chemical imbalances in the body? Does the perfect human-level AI get depressed and commit suicide some of the time? Are we trying to replicate all of the parts, or only the good parts?

aheilbut• 12 years ago

The hype is ultimately due to the very good performance recently (ie. much better than other methods, on learning tasks that people had been struggling to do better on for a long time) in a lot of practical applications like speech and image recognition.

jesuslop• 12 years ago

For instance: speech recognition in android with unprecedented word error rate and facebook face recognition with human-like performance, and other language processing applications as sentiment analysis.

davidtgoldblatt2• 12 years ago

I'm don't have a ton of expertise in the area, but I chat sometimes with people who do, so take this as a reasonably informed but lay opinion.

By and large, most uses of machine learning don't get much value from better machine learning in the sense that, given finite time and resources, you'll probably get better results on your problem by spending time on feature engineering instead of on machine learning improvements.

Some of the new deep learning techniques though have achieved good results on a large variety of problems in more of a "just plug it in" sense. My favorite example of this is this paper: http://machinelearning.wustl.edu/mlpapers/paper_files/ICML20... , which handles both sentence parsing and scene decomposition. They get competitive results on wide-ranging problem domains which have traditionally required lots of fine-tuning to do well on.

So the hope is that deep neural networks will provide improvements in the same way that SVMs did 20 years ago; that people will be able to just dump numbers into a more or less black box algorithm and get better results than they currently do. And indeed, I'm guessing people were as excited about SVMs in the 90s they are about deep neural networks today.

vonnik2• 12 years ago

The advantage of deep learning is that you don't have to engineer features. It does the feature extraction for you, which saves engineers months of work.

ma2rten• 12 years ago

I disagree somewhat, in my opinion is rather more difficult to use. However, deep learning can solve problems that where previously insolvable. It's a step toward "real" artificial intelligence.

TeodolfoM2• 12 years ago

A link to a textbook draft from prominent ML researchers specializing in deep learning that is shockingly also about deep learning is not an example of hype. Yoshua Bengio has been working on neural nets for years and years and he leads the only one of the big three deep learning labs to not have a lot of commercial entanglements. I agree there seems to be some extra attention for deep learning recently. So who is hyping this stuff? Not the people in the big three (Toronto, NYU, Montreal) deep learning labs. They are just doing what they have always done, work on neural nets and solve problems and write papers (and in this case a textbook).

Google, Microsoft, Facebook, Baidu, and to a lesser extent IBM, have all made large commitments to using deep learning methods. Why? Because they solve some really hard problems particularly well. Are they hyping deep learning? Not as such. They are trying to bring attention to their products. Rick Rashid had a presentation that mentioned deep learning and some of the other firms have mentioned that they use deep learning in certain products (e.g. google voice search) but they don't really seem to be hyping the methods or techniques so much as their own particular successes with them. So who is hyping deep learning?

Deep learning techniques solve a lot of important problems and can almost certainly be applied to many more. But "deep learning" isn't a single technique, it is an attitude and approach to machine learning. So the reason you are hearing more about it is because big industry players are using it and some people in academia are getting a bit more traction in the broader ML community with the ideas they have been pushing, refining, and improving for years.

Shitty tech journalists and overoptimistic amateur "data science" clowns in the startup community that think these ideas are some sort of panacea are the people overhyping it.

TeodolfoM2• 12 years ago

Support vector machines are not particularly interesting. They are just perceptrons with a lot of math. So ultimately souped-up linear classifiers. If you start learning the kernel in an SVM you get a deeper model. A neural net using the hinge loss and an L2 penalty on the weights is basically a primal SVM.

abhgh2• 12 years ago

The math in "a lot of math" and the soup in "souped-up linear classifiers" are really not stuff to be casually swept under the carpet :) They still remain one of the best off-the-shelf learners one can use (look at "An Empirical Comparison of Supervised Learning Algorithms" by Rich Caruana et al).

Houshalter• 12 years ago

Deep learning has pushed up the state of the art exponentially in the last few years. On some benchmarks like speech recognition, deep learning methods increased the state of the art by what would have taken decades at the previous rate of progress. This years Imagenet winner achieved 5% error. Last year was 11%, the year before 15%. (Note percentage error isn't linear, each percentage point is harder than the last.)

Everything from Machine vision to natural language processing to speech recognition, is benefiting from this. We live in exciting times for AI, and everyone wants to get in on it.

Simple machine learning algorithms are interesting and even superior on simpler problems. But they don't scale up to complicated AI tasks like this. They are just doing simple correlations between variables, whereas NN's can theoretically learn any arbitrary function (theoretically Turing complete.) Usually this does make them overfit, but on more complicated problems underfitting is the bigger problem.

hyperbovine• 12 years ago

To be honest, I have never understood all the hoopla concerning the universal approximation theorem. Least squares in your favorite orthormal basis for L2[0,1] will do the same thing and it can be easier to match the correct basis with your problem domain. If there was some result showing that NNs converge at a faster rate (as a function of the number of parameters being fitted) that would be nice, but I am not aware of any. Also, universal approximation has nothing to do with Turing completeness.

fchollet• 12 years ago

Everything from Machine vision to natural language processing to speech recognition, is benefiting from this. We live in exciting times for AI, and everyone wants to get in on it.

Not "everything" is benefiting from multilayer CNNs; in fact their use is limited to a tiny fraction of machine learning problems. The majority of data problems in the industry involve small datasets with a limited number of dimensions; picture classification and speech recognition are outliers in their scale, although they are extremely important problems.

Also, the fact is that multilayer CNNs are not a recent development, even though the catchy term "deep learning" is. The recent development that has brought us significant performance improvements in image classification and speech recognition is mainly to put CNNs on GPUs, allowing us to scale the networks considerably. So you could say that what is going on here is really hardware progress, not any kind of theoretical breakthrough (there is surprisingly little theory behind multilayer CNNs).

At last, improvements over the state of the art on a number of large-scale ML problems is not the reason there is so much hype about "deep learning" (which is really the same kind of shallow learning that we've been doing for a while), both in the mainstream media and in the tech community. We have entered a new AI summer, where algorithms that have been around for decades are being hailed as being the "real deal" that works just like the brain (nope). A company like Vicarious has raised 50M not long ago on a promise to create "human-level vision" by 2015 and "strong AI" by 2018. Somehow I don't think investors will see much of a return on that money.

This is all nice and good, but after summer comes winter. The hotter the summer the darker the winter. It has happened before, and it has hurt AI progress very much.

vonnik2• 12 years ago

I hear O'Reilly's coming out with a book soon as well.

bentoner• 12 years ago

See also Michael Nielsen's book Neural Networks and Deep Learning

http://neuralnetworksanddeeplearning.com/

ajays• 12 years ago

Unfortunately, the chapter(s) on Deep Learning haven't even been written yet.

iandanforth• 12 years ago

Hrm ... there has got to be a better format for soliciting edits. Github, Google Doc, ???

discardorama• 12 years ago

Sigh. The authors have taken a PDF and generated HTML out of it. I can understand the desire to maintain copyright, but this is ridiculous.

gamesbrainiac• 12 years ago

First it was data mining, then machine learning, and now deep learning. What's the next name? Clairvoyant learning?

chestervonwinch• 12 years ago

The latter is a subset of the former and refers to particular architectures/strategies for doing machine learning/data mining. Think deep as in hierarchies not deep as in "whoa, man". Some ideas in deep learning are new; some are old. It's not entirely a rebranding.

fizixer• 12 years ago

Question: Does anyone have updated information about Hinton's 'trillion parameter network' progress (http://deeplearning.net/2013/05/20/g-hinton-wants-to-train-n...) from almost a year and a half ago?

It's been quite a while and even Ng has demonstrated that a billion parameter setup could be built for $20k using commodity hardware (https://news.ycombinator.com/item?id=5896684).

I wonder what's happening at Google labs as of August 2014.

shepardrtc• 12 years ago

Here's a talk by Geoffrey Hinton back in 2007 about Deep Learning neural networks:

https://www.youtube.com/watch?v=AyzOUbkUf3M

Deep Learning: An MIT Press book in preparation

Related Stories