Andrew Ng and the Quest for the New AI

http://www.wired.com/wiredenterprise/2013/05/neuro-artificial-intelligence/

by ivoflipse • 13 years ago

147 70 13 years ago

wired.com

kinofcain2• 13 years ago

There is a very quick reference to the person who inspired him, Jeff Hawkins, whose book is worth a read:

http://www.amazon.com/On-Intelligence-Jeff-Hawkins/dp/080507...

Edit: update link

netrus• 13 years ago

Link does not load CSS for me, but this does:

http://www.amazon.com/Intelligence-Jeff-Hawkins/dp/B000GQLCV...

kinofcain2• 13 years ago

Thanks, had stripped the tracking codes off the mobile link, but it doesn't load right on desktop.

blhack• 13 years ago

That book will change the way that you look at yourself.

lomendil• 13 years ago

I just saw Jeff Hawkins give a talk and it was quite interesting. I was a bit worried, however, that he is basing his theory of intelligence on the human neocortex, while claiming to go after general principles.

This is guaranteed not to be terribly general, considering the many bits of matter on this planet that exhibit intelligence without a neocortex. By many, I mean ones that hugely outnumber humans.

So very interesting stuff, but not the answer that I think he wants it to be.

Lambdanaut• 13 years ago

In "On Intelligence" he postulates that he's not necessarily going for a "human-like" intelligence or even a "life-like" one.

Basically he just wants something that's very good at recognizing patterns over time, which I can imagine the neocortex would be great at.

Though, he also references the thalamus and hippocampus in the books a lot, as very important parts of the brain to his framework. [http://en.wikipedia.org/wiki/Memory-prediction_framework#Neu...]

thomaspaine• 13 years ago

Grok (formerly Numenta) has a slightly more technical white paper that goes into more detail on the actual algorithms from On Intelligence:

https://www.groksolutions.com/htm-overview/education/HTM_Cor...

There are some subtle differences between HTMs and straight up deep learning, mainly the requirement for HTM data to be temporal and spatial.

I know Andrew used to sit on an advisory committee at Numenta, I don't know if he still does.

jhartmann2• 13 years ago

I've said this before, but deep learning is terribly powerful precisely because you don't have to spend lots of time doing feature engineering. Multi layer networks that are trained in semi-supervised, unsupervised and supervised fashions all can now produce networks that meet or beat the state of the art hand created models for speech, handwriting recognition, ocr, and object recognition. We are only just beginning to see what is possible with these sorts of techniques. I predict within a few years time we will see a huge renaissance in AI research and Neural Network research specifically as these techniques are applied more broadly in industry. My startup is building some cool stuff around this technology, and I know there are hundreds like me out there. This is going to be a fun ride.

wookietrader4• 13 years ago

This has been said about neural nets two times already. Sadly, they did never deliver.

There are still applications where e.g. random forests beat the crap out of all kinds of deep learning algorithms in (a) training time (b) predictive quality (c) prediction time.

We should stop hyping this. I am a researcher working in deep learning myself, but the current deep learning hype is actually what makes me worry that I will have trouble getting a job because industry will be disappointed a third time.

jhartmann2• 13 years ago

Well I think a big part of this is that right now we have finally gotten to where the algorithms + the required computing power are starting to become more widely available. Cheap graphics cards or things like Intel's Phi + techniques like drop out to prevent overfitting are really enabling much more sophisticated things to be done in a reasonable wall time. Granted multilayer neural networks aren't a free lunch that will solve everything, but there are large classes of problems that are falling to these techniques all the time. We are also finding that Neural Networks scale very well to very large architectures, better than some of the other techniques. I understand we should be careful not to overhype since we have seen previous excitement cause a mass exodus from this research before. I however think people like Hinton were always right, and this was awesome stuff. We just couldn't really take advantage of it because we could never train it for long enough and we hadn't learned how to do things efficiently yet.

aswanson• 13 years ago

Can you recommend any introductory papers in the area, or keywords to look for aside from the obvious?

visarga5• 13 years ago

It's a little offtopic, but can you recommend a random forest implementation to be used from the command line, like Vowpal Wabbit does for linear learning? I like VW for its speed and ease of use.

hvs• 13 years ago

OT: His online Machine Learning class last year was great. He is the best professor I've ever had, and explains things so clearly that you understand them the first time. You are lucky if you ever get to work or study under him.

rzendacott• 13 years ago

That's awesome to hear! I'm taking the Coursera class now, and it's been great so far. It just started a couple weeks ago, so it definitely isn't too late to join! https://www.coursera.org/course/ml

psbp• 13 years ago

I wish Coursera followed the Udacity model. I always find out about these classes after they're already weeks in progress or over.

etherealG• 13 years ago

thanks very much for pointing this out, joined the class now, hope i can catch up

primaryobjects• 13 years ago

I'm also taking the class. Just finished the logistic regression programming assignment. Great stuff. You can take those algorithms and kind of add a bit of magic to your software.

alanthonyc• 13 years ago

It’s great this year as well. Very enjoyable class.

mark_l_watson• 13 years ago

I second this! I have taken 7 Coursera classes, and most of them "lightly": just doing as much work as I needed to for passing the class, with just a few classes that I did put a lot of energy into. Andrew's class was in this second category: I kept taking the tests and tweaking the homework assignments over and over again until I got a 99.5% score in the class. His class was lots of fun and also very useful material. Recommended!!

visarga5• 13 years ago

Yes, unfortunately his partner, Daphne Koller, though very esteemed academically, made me feel very frustrated in the PGM class. Not the same great feeling of clarity.

ivanist4• 13 years ago

I have to blame you, because now I had to take the course in coursera! I always wanted to learn machine learning. Better late than never!

aespinoza2• 13 years ago

I thought the man behind the Google Brain was Ray Kurzweil (http://www.wired.com/business/2013/04/kurzweil-google-ai/).

drcross• 13 years ago

Kurzweil is a hack. (Source: I've an Msc in Robotics and read all his books)

konstruktor2• 13 years ago

Please think about what a source is. This isn't reddit.

aespinoza2• 13 years ago

Dude, I am sure you know more than me in the field of robotics and A.I. but I don't think Google would spent so much money on a hack.

SCdF• 13 years ago

drcross is a hack. (Source: I'm a space fairy)

If you're going to throw broad statements around some examples would really help lend you credibility.

gavanwoolery• 13 years ago

Hmm...I don't mean to be a skeptic, but I do not see any new theories here. Neural networking has been around for a long time, as have an abundance of theories and implementations around it...some people have gone so far as to build an actual brain replica (a digital version of the bio/analog thing). Neural networking is extremely powerful, but to be of any use, you need a lot of computing power. As it turns out, our brains are really good at massively parallel tasks like image and motion processing; these things can be done explicitly on a computer with some ease, but having a computer learn on its own from scratch how to do them is not easy.

raverbashing• 13 years ago

Exactly

Neural networks of course "mimic" the way the brain works, but it stops there.

Not to mention several modern AI techniques have nothing to do with mimicking biology (like SVMs)

This looks a reporter pushing his own agenda to make for a colorful story.

xtacy• 13 years ago

I think the requirement on enormous computing power is exactly the point. Since not many have access to such computing power, Andrew Ng and folks at Google (incl. Jeff Dean!) built a large scale neural network system called DistBelief (nice name!). This system allows programmers to think of really large scale neural networks without worrying about how to scale them, handling fault tolerance, etc. You can think of it as the MapReduce for neural networks. They demonstrated how a large scale neural network can do interesting stuff on its own (e.g. recognising cats and human faces from unlabeled youtube videos).

jbelanich• 13 years ago

You're correct in that neural networks as a model have been around for a long time. However, those networks were restricted to be shallow because backpropogation didn't work well on networks with many hidden layers. Only recently have researchers developed learning procedures that can learn these deep architectures efficiently, using some clever unsupervised learning techniques. And surprisingly, they are finding that these deep networks perform remarkably well, beating the state of the art in a number of benchmarks.

You are also right that you do need a lot of processing power to get neural networks to work well. But that is changing rapidly. Hinton's convolutional neural network has the state of the art in the ImageNet benchmark, yet was trained using significantly less power than google brain. Regardless, you don't need google scale computation to get deep networks to work well. The point of google brain is to see how far one could push neural networks.

jcrites• 13 years ago

> Only recently have researchers developed learning procedures that can learn these deep architectures efficiently, using some clever unsupervised learning techniques.

Would you mind naming some of these techniques, if you're familiar with them? I'd like to take a deeper look.

EthanHeilman2• 13 years ago

We should stop trying to claim every new method is "like the brain". We don't have any clear understanding of how the brain works. One can be inspired by a particular and likely wrong cognitive theory, but one can not say one is building "machines that can process data in much the same way the brain does" truthfully without a deeper, and currently unavailable, understanding of the functioning of the human brain.

Jach• 13 years ago

I feel the same way, that researchers should stop trying to mimic the brain, but not because we don't understand the brain. While I think there are still several decades before we'll be able to have mind uploads, I also think a lot of people underestimate the quality of modern brain science. In any case, I have the same reason as Dijkstra for why I think mimicking the brain isn't that great an idea. In http://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD1036... (really a great read to branch all sorts of thoughts off of) Dijkstra said, "The effort of using machines to mimic the human mind has always struck me as rather silly: I'd rather use them to mimic something better."

It's probably a harder problem, creating smarter-than-human intelligence on a machine, but research isn't as constrained by laws and ethics (they don't have to bemoan not being able to experiment with living human brains). I wish more people were active in the area.

visarga5• 13 years ago

They are trying to find THE ONE ALGORITHM that solves all A.I. problems. The brain has an implementation of it, but it is in wetware, hard to extract. Deep learning makes some pretty good approximations of the visual areas, though.

yankoff• 13 years ago

Great and inspiring professor. Taking his ML course on coursera and trying to follow his talks.

wookietrader4• 13 years ago

If an article says that Andrew Ng is "the man at the center of deep learning" it's just not right. Geoffrey Hinton's and Yoshua Bengio's impact were at least as high as his, if not much higher.

why-el• 13 years ago

So I am a little confused. Where are we on the learning part of AI? As I understand it, the current consensus is to throw as much data as you can at your model (millions of cat pictures in this article's example) to make it pick up patterns and yet still claim that we are closing in on how the brain works? As far as I can tell no human brain would need that many pictures to see a pattern. In fact, and this is probably more apparent in language, we humans tend to work with degenerate data and still end up with perfect models.

DennisP• 13 years ago

Bear in mind that the neural network is starting at the same point as a newborn baby.

piglop• 13 years ago

You may not have seen millions of cats, but how many images have your brain processed since you can see? A five years old brain has been trained on billions of images (along with many other simultaneous inputs).

cscurmudgeon• 13 years ago

Ah. The good old AI cycle.

Scientist: X can help us get full AI!

You: Why?

Scientist: Because of reason R.

You: But, reason R is a non sequitur...

More seriously, reasons similar to that for deep learning have been repeated multiple times in AI with failure (e.g. Thinking Machines).

I would suggest that these folks remain calm and build something on the scale of IBM's Watson using just deep learning..

niklaslogren• 13 years ago

Very interesting article, it makes me hopeful.

This might be slightly off-topic, but I'll try it here anyway: can anyone recommend any books/other learning resources for someone who wants to grasp neural networks?

I'm a CS student who finds the idea behind them really exciting, but I'm not sure where to get started.

pilooch• 13 years ago

It seems the relationship between the brain and deep learning has evolved in such a way that the later can help with insights into how the former works.

In this regard, I thought I would mention the extraordinary simple and elegant talk by G. Hinton last summer: http://www.youtube.com/watch?v=DleXA5ADG78

It starts from a simple and clever improvement to an existing deep learning method and ends up with beautiful (and simple!) insights on why neurons are using simple spikes to communicate.

Andrew Ng and the Quest for the New AI

Related Stories