Basics of Neural Networks with example codes and illustrations

http://natureofcode.com/book/chapter-10-neural-networks/

by coderjack • 13 years ago

231 29 13 years ago

pests2• 13 years ago

In regards to the first interactive demo, it seems to be adjusting the line to be parallel to the one drawn on the background. Was this intentional or are the supposed to converge?

coderjack4• 13 years ago

The line in the background is the function towards which the neural net is expected to converge.

pests2• 13 years ago

When I first posted my reply every time I watched that demo the neural net line was converging on a line perpendicular to the line in the background.

(rereading my original comment I think I described what I was seeing incorrectly but this new reply correctly explains what I was originally seeing)

NKCSS• 13 years ago

An awesome book; I've now started reading from the beginning of the book :)

One thing I've noticed though, is that img 10 of chapter 1 is missing.

http://natureofcode.com/book/chapter-1-vectors/imgs/chapter0...

CodeCube• 13 years ago

You can submit a pull request ;) https://github.com/shiffman/The-Nature-of-Code

I think that's so amazingly awesome, that it can evolve as a living document in this way.

mekarpeles• 13 years ago

The experience (specifically the careful choice of mediums + examples + presentation though which the concepts are conveyed) is pretty fantastic.

coderjack4• 13 years ago

You may also want to read the chapter on fractal programming in this book. Its pretty intuitive too.

gustavodemari• 13 years ago

nice article

Lambdanaut• 13 years ago

This book is everything I've ever wanted in a programming text. I'm sorry that I don't have much of anything substantial to say except praise, but seriously, thank you for writing this.

coderjack4• 13 years ago

Thanks for liking this post. But the actual credits must go to the author of this text as I am just another fan of this book like you are now.

JDDunn9• 13 years ago

Does anyone have any examples of areas where neural networks beat out statistical based methods, other than maybe image recognition? I can't even think of another major area where they dominate.

- Search engines use algorithms, not neural nets.

- The most popular algorithm on Kaggle (data analysis competitions) is random forests

- Google's self-driving car uses statistical-based methods

I can't imagine commercial aircraft would use a neural net. What happened if one crashed? They would analyze the data and ask questions like, Q: "What happened?" A: "I don't know" Q: "Can we fix it so it doesn't happen again?" A: "I don't know".

coderjack4• 13 years ago

Yes its true that neural nets are not used in commercial systems because its computation intensive and also needs huge infrastructure even for things like finding cat images out of youtube videos but the results are far more accurate then any of the statistical methods used in commercial systems. There is time-performance trade-off when it comes to choosing statistical methods over neural nets and also it's a field yet to be explored. I will be writing a series of article on neural nets for beginners and advanced programmers . Please stay tuned.

nomailing• 13 years ago

Why do you think that they are not used in commercial systems? As far as I remember from leckeres of Geoffrey Hinton they are now used quite extensively in speech recognition systems on smartphones, etc.

lightcatcher2• 13 years ago

Deep learning generally involves neural networks or very similar models. Do you want me to list some of the many tasks for which deep learning is state of the art?

Search engines could use neural networks (and possibly have better search results), but the research was just done a few years ago (ie 3-4 years ago) and search engines are much older than that.

As for the commercial aircraft point: I'm trying to find the youtube video I've seen before about this. The video is of a plane simulation where the plane has lost control of both of the wing flaps (or both of some other important steering element) and a person was unable to stabilize the plane but a neural network could. This doesn't mean a neural network should be used for autopilot all the time, but they can be useful for certain situations.

Edit: found the video, https://www.youtube.com/watch?v=aObBHXsc_iw&t=3m50s

level09• 13 years ago

We use Neural Networks to extract text transcripts from our video stream, the process is extremely complicated as the stream is in Arabic which is a fairly complex language.

the system is fed with the actual transcript for some videos so it can learn with time and minimize error.

the overall system is incredibly useful as you can search the video stream of a TV channel for a specific keyword.

I have also used self organizing maps before for creating a recommendation engine.

I think it works a better than typical recommendation engines because you can feed the network features based on multiple criterias (in addition to taxonomies, you can also feed user geographic location, and click patterns etc ..)

atlbeer• 13 years ago

Is the product publicly available?

dave_sullivan• 13 years ago

Glad you asked...

Definitely image recognition: http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf

Speech recognition: http://www.cs.toronto.edu/~hinton/absps/RNN13.pdf

Natural language processing: http://www.socher.org/index.php/DeepLearningTutorial/DeepLea..., http://aclweb.org/anthology/N/N13/N13-1090.pdf

If you're into kaggle competitions: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it...

I don't think there are going to be any further major advances in eg SVMs or random forests (famous last words maybe...) Neural nets, on the other hand, are just scratching the surface of what's possible. So right now they are state of the art in some historically very difficult areas. But these are early days still.

lightcatcher2• 13 years ago

You wrote the answer I was just a little bit too lazy to write...

As to the GP: Geoff Hinton (probably the most well-known neural networks researcher) said in his Coursera course that neural networks thrive at problems with a lot of structure that could be encoded, while simpler models like SVMs or Gaussian processes might be better for problems without as much deep structure to discover.

Also, a lot of the current research with neural networks involves using neural networks to learn better representations of data. These cleaner representations of data (which can be thought about as a sort of semantic PCA) often make classification far easier, which explains the great results. Learning representations also makes transfer learning (transferring knowledge from one domain to another) much easier/more possible.

catwell• 13 years ago

Actually, in many cases non-NN methods still beat NN for those use cases, at least if you consider things like the speed of the response and consumption of resources as relevant.

Recent papers show that NN are coming back, but I think most of the speech recognition out there is still Hidden Markov Models and most of the image recognition is definitely based on tailored detectors & descriptors. This is especially true when you do these things on the users' mobile devices.

p1itopre• 13 years ago

Transaction Fraud Detection

http://www.fico.com/en/Products/DMApps/Pages/FICO-Falcon-Fra...

mrbrowning• 13 years ago

They (specifically neural networks trained on the cross-entropy error, with the logistic function as your activation function) work very well in classification tasks where you want to interpret the output as a probability, which is useful when you're more concerned with predicting across datasets rather than individual instances. My lab uses neural networks towards this end in predicting pharmacological activity across large libraries of molecules and all the prospective validation we've done indicates that neural networks are far and away the best-performing option out there for this domain as compared to e.g. SVMs or random forests.

aryastark• 13 years ago

I can't imagine commercial aircraft would use a neural net. What happened if one crashed? They would analyze the data and ask questions like, Q: "What happened?" A: "I don't know" Q: "Can we fix it so it doesn't happen again?" A: "I don't know".

We already do. The pilot and co-pilot(s). Though I get what you're saying, I just find it humorous that we would worry about it in such manner.

AI neural nets aren't quite there yet. But we attribute that great unknown to vague catch-all "human error" all the time. I suspect one day we will simply attribute AI screw ups to "computer error" or something like that.

joe_the_user2• 13 years ago

Well,

The thing is that (at least several kinds of) neural networks and support vector machines are equivalent in their operations. SVMs themselves seem to be something like a general form of binary regression (a clever way to separate two complex sets with a complex curve on a high-dimensional space). So the choice of neural network or support vector machine or some other statistical device seems like it comes down to which problem is easier to formulate how, which tricks are available for what problem, etc.

For example: http://blenny.ncl.ac.uk/peter.andras/PAnpl2002.pdf (that's what I google just now, think there are results on "deep" networks also).

aheilbut• 13 years ago

Despite the name, there's nothing mystical about neural networks that make them not 'statistical based methods'

It's just a particular way of describing a mixture of nonlinear regression models. There are some nice, although very abstract analogies to biology, which is both a blessing and a curse.

joe_the_user2• 13 years ago

Yes,

"...there's nothing mystical about neural networks", yes, there's nothing "not statistical" in the reality of the construct (the NN classes are often given without reference to stat).

But in the way that people present them or think about them, there is tendency for them to be seen or used as indeed "black boxes"(per the gp) or mystical constructs.

A NN might indeed be a good fit for a given class of problems and its the mechanism might be revealing for said problems. But the external impression that their development gives, that their development is that of a "thinking process", is kind of a weakness for the field.

catshirt• 13 years ago

cool. just bought the book on Amazon! i know some small amount about neural networks (i was able to skim the article), but the book as a whole looks stellar.

nrox• 13 years ago

brain.js library is a NN implementation in JavaScript. It's very easy to use. https://github.com/harthur/brain

Here is a test with a model of a robotic arm: https://assemblino.com/show/public20123372.html

bcuccioli• 13 years ago

I wrote a simple neural network about a year ago for doing optical character recognition as a class project. I think looking over the code could be good for learning, as it has a pretty simple OOP structure: https://github.com/bcuccioli/neural-ocr

Basics of Neural Networks with example codes and illustrations

Related Stories