A Revolutionary Technique That Changed Machine Vision

http://www.technologyreview.com/view/530561/the-revolutionary-technique-that-quietly-changed-machine-vision-forever/

by lelf • 12 years ago

175 65 12 years ago

pavlov2• 12 years ago

It's interesting that, in the image showing the classes easiest and hardest for the algorithm, all the easy ones are animals and all the difficult ones are human-made artifacts.

Does nature tend to create forms which are easily detected by relatively simple neural networks? The evolutionary explanation could be that this allows animals with primitive neural systems to more easily distinguish other members of their species.

bane2• 12 years ago

The evolutionary explanation could be that this allows animals with primitive neural systems to more easily distinguish other members of their species.

It's probably the other way around. Neural systems optimize for natural objects.

pavlov2• 12 years ago

But isn't the neural network (like the algorithm described in the article) a mathematical concept? Are the computer-based neural networks so closely modeled after biological equivalents that they would inherit such an optimization?

(As you can tell, I know nearly nothing about neural networks, but curious to learn more...)

ma2rten• 12 years ago

I think it's because these human made object have no single form. For instance "letteropener" describes the function of the object, not the form. Compare that to "red fox" which is always gonna look more or less the same.

kastnerkyle15• 12 years ago

Yes, but the network is still differentiating between select breeds which means it is learning traits of the animal which are unique to the breed. And training at a higher level is perfectly doable, say "fox" instead of the fox breed.

Ultimately, if you are able to look at an image and say "letter opener" there are features which differentiate this from a knife/can opener/whatever - these are the exact things a convolutional neural network should be able to use (in theory) and has nothing to do with the label, which is typically unimportant as long as it is unique and accurate.

We could flip all the labels around and still get unique answers - the network is just learning a mapping from input -> some integer, and I would argue the variance in dog breeds and lighting in natural scenes is much trickier than the angle/shape of a letter opener.

I still think it comes down to the composition of this particular dataset. Augmenting this with images scraped from online stores would be very interesting as it is fairly trivial to get huge numbers of images for anything that is typically sold online - I think Google is way ahead on this one!

theoh2• 12 years ago

I'd say it's an entropy thing. The principle of "correlation of parts" means images of animals have low entropy relative to, say, machines. Intuitively a machine can contain an almost arbitrary set of shapes and components, whereas animal forms are more constrained.

kastnerkyle15• 12 years ago

See my earlier answer - I think that it is all about the data. And most things humans design have a similar "correlation of parts" due to the human preference for symmetry. Not all the images are face on headshots - lots of running/action, off centered, etc. ImageNet is a very "real" dataset in that sense.

Entropy isn't the right term here - low entropy would imply something about the compressibility of each image based only on whether it was a machine or a natural object, which I don't think is case.

possibilistic• 12 years ago

  > Does nature tend to create forms which 
  > are easily detected by relatively simple 
  > neural networks?

Poison markings and mimicry, for sure. Furthermore, large swaths of plant life in various spectra. reply

kastnerkyle15• 12 years ago

Chicken, egg. Maybe these things are easy because they are crucial to survival (and everything that was bad at recognizing these signs died), not due to being easier for simple visual processes.

kastnerkyle15• 12 years ago

It could also be that there are more labeled examples of animal, or that the translation/rotation of an animal is less disruptive than than the same transform to a car or other object.

I know for sure that ImageNet has a huge amount of animals in it, down to very select sub-breeds, while the object categories are usually at higher levels.

Houshalter4• 12 years ago

Much more training time was spent on animals (10% of the dataset is literally just pictures of dogs) and much more variation among man made objects.

oftenwrong• 12 years ago

Here is the challenge site: http://image-net.org/challenges/LSVRC/

The datasets can be found by clicking the links for each year's challenge.

drpgq• 12 years ago

Not to denigrate the techniques used here, but it is interesting as a computer vision researcher (face recognition in my case) how important good labelled training and testing sets are. Some of my successes over the years have come more from figuring out where to get good data than good computer vision techniques.

trhway• 12 years ago

good education does matter :)

kastnerkyle15• 12 years ago

Not only large datasets, but the elasticity of a community to revisit old ideas from a new angle. Computer vision has been very good in this regard overall, though there were some dark times (https://www.facebook.com/yann.lecun/posts/10152034328862143).

tzs• 12 years ago

Can you ever generate artifical training and test data?

For instance, suppose I would like to be able to take a photo of a chess game and turn that into a diagram of the position. I have no idea where I would get natural photos of thousands or millions of chess games to use for training and testing a chess piece identifying vision system.

Could I instead make 3D models of common chess set designs, and then generate and render photorealistic images of chess positions to use for training and test data for the vision system?

ertdfgcb• 12 years ago

Probably, but it might not be able to work very well with different conditions, like a picture of a chessboard in central park vs a chessboard in a library. This could probably be solved with more renders, and even if it can't, the renders would probably be a good starting point

Houshalter4• 12 years ago

You still need some natural data, otherwise the network will probably overfit on regularities in your CG engine that don't exist in the real world. It might do that anyway, even with natural data.

graycat2• 12 years ago

What I see in the OP, via my Web browser Firefox 27.0.1, is just an ad I can't make go away in the middle of the screen. So, without some special effort, say, grab and parse the HTML, I can't read the content. Anyone else have this issue?

inetsee• 12 years ago

I have Firefox 32 on Kubuntu 14.04, and I had no problem clicking on the close button on the ad.

graycat2• 12 years ago

Thanks. I couldn't see a button for closing the window. Apparently somehow asking for higher magnification of the Firefox window didn't give higher magnification of the ad window; thus I didn't see the close button.

Thanks.

With your feedback, I tried again and did see the big X close button outside of the main part of the ad window. The thing did close.

Read the article. Cute.

Thanks.

kastnerkyle15• 12 years ago

I think it is key to remember that the accessibility of huge amounts of labeled data is behind most of this. ImageNet is 1.2TB (~1.2 million images!). Convolutional neural nets have been around for a long, long time (early 90s? or maybe late 80s) and they just needed more data (and a few training tricks :) ).

As we enter the world of "big data" more and more companies are locking this level/size (or bigger) data away in closed vaults. This is a competitive edge in business, but also strangles the ability for academics to do "real world" research, which is often a huge criticism of academia at large.

Academic research can have a role in tech R&D if given a chance, so I hope that companies will open their data to private researchers to continue this kind of improvement - even if under secrecy clauses or some other safeguard.

All that said, the technological improvements have been astounding, and I hope this is only the beginning! There have been some incredible results using related techniques in NLP for machine translation recently, and speech has always been a great playpen.

If anyone is interested in playing with these things in Python, myself and another researcher have recently created a library for using these type of neural networks as a black-box WITHOUT ANY TRAINING!, in a similar style to scikit-learn. See [1] for more. We support only a few nets currently, but plan to integrate support for caffe and pylearn2 in the near future.

[1] http://sklearn-theano.github.io/

agibsonccc2• 12 years ago

Great work guys! Are these going to only be computer vision nets?

I think the huge problem with most deep learning frameworks out there are they are either hard to use or limited in scope.

There's nothing wrong with that per se, but I'd be curious to see what you guys intend. It's great that you guys are giving an sklearn interface to it.

Edit: I should add my criticisms come from a biased perspective: I write http://deeplearning4j.org/

I love comparing notes with others regardless.

Either way: Wish you the best of luck with it.

kastnerkyle15• 12 years ago

Support is planned for audio and hopefully text - I am working on building a million song dataset to recreate the work Sander Dieleman did for Spotify, and have had some possible support in October for the weights of a trained speech network! So yes, feature extraction from other domains should be on the horizon.

We are specifically trying to make it easy to say: I want to transform an image (or audio, etc.) using a pretrained net. Download the weights, extract the features for me, and give me the feature vectors so I can do something with those. This seems to be really, really, really hard in all the tools I have used and usually involves training yourself, which is not very useful for things on the level of ImageNet.

Of course, having nice examples and good docs is one of the great parts of scikit-learn, and is actually one of the things I have been working on most recently. Our docs aren't to that level yet, but I hope they can be one day.

DeCAF (precursor to Caffe) and OverFeat binaries were really kind of the first in this regard (about 1 year ago now), but IMO one of the limitations is that they lose interaction with the rest of the Python ecosystem for data munging, and simple algorithms for exploration. By wrapping the weights, we hope to leverage the support of the Python ML ecosystem easily, while still being able to use the power of these networks.

Right now the most compelling use case is as part of a scikit-learn pipeline i.e. make_pipeline(OverfeatTransformer, LinearSVC) or whatever. Feed images in, get predictions out. I am also working on a demo of "writing your own twitter bot" similar to https://twitter.com/id_birds , written by Daniel Nouri . I like sloths, so it will definitely be a slothbot.

I also hope to support recurrent architectures from Groundhog (https://github.com/lisa-groundhog/GroundHog) as several researchers here at the LISA lab have been using it to get pretty amazing results in NLP and audio, both of which are potential targets in the future. If we can leverage their work, it would be a very nice way for people to immediately play with SOTA architectures in different applications.

In any case, just loading in weights and extracting image features easily is nice, and was a benefit for both Michael and myself in a research project this summer.

aswanson• 12 years ago

This post reminds me why I fell in love with HN. I use Theano, awesome contribution!

michaelochurch3• 12 years ago

Convolutional neural nets have been around for a long, long time (early 90s? or maybe late 80s) and they just needed more data (and a few training tricks :) ).

The state of the art has become a lot better in terms of training deep (multi-layer) neural networks. The naive approach (start at a "random" weight setting, use gradient descent) that works on a convex error surface fails catastrophically on deep, complex neural nets. (Shallow neural net training is a non-convex problem as well, but seems to be "less non-convex" in practice.) In the past 10 years, people have become better at getting past that.

Theoretically, only one hidden layer is necessary for neural nets to be universal. Thus, for a long time, most research focused on single-layer networks because those were "good enough" to model any mathematical function. The problem is that convergence, for single-layer nets, can be very slow (especially given that you're often doing stochastic gradient descent when working large data sets). Single-layer nets are often very difficult to audit. There's a lot of "cross talk" where unrelated features are mapped to the same "space" in the network. So you have a "black box" that is hard to interpret.

Deep neural nets (which is what ML researchers call "deep learning", until MBAs start abusing the latter term, as they have with "big data") have come back into style over the past few years, due to recent research into how to make them actually perform, and findings about superior convergence with the right conditions. In deep nets, there's often a problem of signals either amplifying or vanishing as they propagate throughout the net. The former leads to saturation (the neural net moves very slowly from a suboptimal but flat place on the error surface) and the latter is too linear and unlikely to pick up interesting features. Even now, making deep neural nets not sensitive to initial starting conditions is an unsolved problem, but there's been a lot of progress.

I would hazard the guess that the convolutional technique is a lot more useful in deep neural networks than it is in single-hidden-layer neural nets.

kmavm5• 12 years ago

Hi, I am a research engineer in Yann LeCun's group at Facebook. I hate to seem to be picking a fight, but you're a prominent poster here, and this comment seems likely to garner a fair amount of attention. Unfortunately almost every sentence you've written betrays a subtle misunderstanding of the space, and the totality is quite misleading.

The naive approach (start at a "random" weight setting, use gradient descent) that works on a convex error surface fails catastrophically on deep, complex neural nets. (Shallow neural net training is a non-convex problem as well, but seems to be "less non-convex" in practice.)

Starting at a random (no scare quotes needed) point in weight space and SGD'ing is exactly what Alex Krizhevsky, and all of the derivative convnets over the last two years, did and do. It works just fine; I sit at work doing it all day long. You need to have enough data to train on, big enough models, and enough flops to train the big models on the big data before your interns' grandchildren die. We have all of the above now. Aside: even single-layer neural networks do not have convex error surfaces; convexity, and funky error surface geometry, is not a relevant distinction between shallow and deep nets. There have been no magical optimization breakthroughs, it's still SGD with the same herbs and spices that were used in the 90's (momentum, e.g.).

The problem is that convergence, for single-layer nets, can be very slow (especially given that you're often doing stochastic gradient descent when working large data sets).

"Stochastic" gradient descent just means doing lots of weight updates per epoch. Ceteris paribus, training on large, redundant data sets, stochastic converges faster than batch because it gets to consider more points in the weight space than batch per pass over the data. The problem with single-layer neural nets is not that they converge slowly; the problem is that the layer size needs to grow exponentially with the task size. Single-layer neural nets' universal approximation power is thus not of great practical consequence. The power of deep nets is the power of composition: f(g(h(x))) is a strictly more powerful model than f(x) holding the number of parameters constant.

Even now, making deep neural nets not sensitive to initial starting conditions is an unsolved problem, but there's been a lot of progress.

You just initialize with a Gaussian ball around zero and explore whatever valley in the error surface you happen to be in. Works 100% dandy.

I would hazard the guess that the convolutional technique is a lot more useful in deep neural networks than it is in single-hidden-layer neural nets.

It doesn't really make sense to talk about a "single layer convolutional net", because if you only have a single layer, and all you can do is convolve with it, then the output of your net will necessarily be a big pile of filtered versions of the input image. Unless your task is specifically to learn a target set of filters, it would make no sense to have a single layer convnet.

Houshalter4• 12 years ago

It can work the other way too. Ironically the academics who created ImageNet restrict who can download it and don't allow it to be used for commercial use.

kastnerkyle15• 12 years ago

Well, they have no choice. Because technically the copyright of each image is still held by the people who took the images (or in some cases the people in the images). Are weights of a trained network based on ImageNet a derived work, a cesspool of millions of copyright claims? What about networks trained to appoximate another ImageNet network?

There is a lot of legal gray area here unfortunately, so the signing of the license to work with the images seems like a CYA move to me.

transpy• 12 years ago

Could you provide a link to the results of NLP for machine translation? I'm very interested in MT.

lars• 12 years ago

I've been hoping for a scikit-learn like wrapper around theano for a long time. Thank you for this, I think it could be extremely useful!

a_e_k• 12 years ago

Can any machine vision researchers recommend a survey/overview paper or two that would make a good technical introduction to these particular techniques?

kastnerkyle15• 12 years ago

I gave a talk at EuroScipy 2014, in the back part there are a ton of references and resources to get started. See [1]

[1] https://speakerdeck.com/kastnerkyle/euroscipy2014

m_ke• 12 years ago

If you want to see a convolutional net in action checkout out the demo on http://clarifai.com/

AndrewOMartin2• 12 years ago

There are two irresponsible sentences in the article, "In other words, it is not going to be long before machines significantly outperform humans in image recognition tasks." and "Or put another way, It is only a matter of time before your smartphone is better at recognizing the content of your pictures than you are.".

The irresponsibility is in seeing the existence of a technique to solve a more complex version of a toy problem (e.g. find the location in this photo that exactly matches pattern x), and inferring that the same technique, given more power, will exhibit super-human behaviour.

It's a reasonable claim that, in the task as described, the technique outperforms humans but that's about as exciting a claim as saying how much quicker the latest supercomputer is at arithmetic than a human. The point being that human object recognition isn't about labelling a scene with nouns, but somehow instinctively knowing the relevant objects for a situation and, if required, the appropriate situation-specific noun.

I therefore request the final sentence of the article be rewritten as "Or put another way, it may only be a matter of time, funding and motivation before your smartphone (with equivalent computing resources of the likes of Google Inc. and University of Tokyo) can ascribe one or more nouns from a set of size of order 1000 to regions of a photo, given that huge amounts of pre-processing and man hours have been dedicated to the pre-processing of that exact set of nouns to create a training set, and that you take the photo in similar lighting conditions as the training set, don't apply any filters, and that the objects referred to by the appropriate nouns is neither small nor thin, better than you.".

blauwbilgorgel3• 12 years ago

It is just extrapolating the error rate reduction over the last few years. Spam filters have become better than moderators in labeling spam only in the last decade or so.

When computers first started to become faster than mathematicians this was really a breakthrough. The same is happening now with object and speech recognition.

The computer succesfully completes a task. That it is not how humans intuitively approach these same tasks is irrelevant for this accomplishment. What if the results were only half as good, but the system behaved more like humans, who does this satisfy?

The state-of-the-art is capable of detecting far more than 1000 objects, does not need labeled data, is robust to changes in light and does not care about the camera used. No preprocessing the data needed, features are automatically generated (preprocessing the target labels is a bit silly BTW).

So yes, in the very near future, algorithms will be better security guards than well... security guards.

AndrewOMartin2• 12 years ago

My point is that extrapolating error rate reduction only applies to this tightly defined task.

You can only make claims about machines being better at "general" pattern recognition when we make progress on the issue that's stopped all Cognitivist General AI projects dead, which is that of situational awareness.

Arithmetic operations, spam detection and the task described in the article have a much smaller, and static, problem space than most human activities. You can demonstrably already knock up an automated-barrier style security guard. However, I'd argue that there does not exist an algorithm or appropriately weighted n-layer network that can handle all the ambiguity, countermeasures and ill-defined or contradictory situations that human security guards, or even just their object recognition capabilities, handle largely instinctively.

ogrisel• 12 years ago

does not need labeled data

It needed training on 1TB of labeled images in the first place. Arguably it can be used to transfer that knowledge to other tasks with a much smaller amount of labeled samples but still requires supervision.

kskz• 12 years ago

When computers first started to become faster than mathematicians this was really a breakthrough.

Mathematicians do not compute numbers.

A Revolutionary Technique That Changed Machine Vision

Related Stories