How Google Cracked House Number Identification in Street View

http://www.technologyreview.com/view/523326/how-google-cracked-house-number-identification-in-street-view/

by programd • 13 years ago

85 34 13 years ago

sjtgraham• 13 years ago

Anyone else getting a lot of obvious door numbers in reCAPTCHA captchas lately?, I guess that's how Google trained these neural nets.

RDeckard• 13 years ago

Yes.

amjaeger• 13 years ago

was just thinking that google can now fool captchas

clhodapp• 13 years ago

Google is also the one serving them, which does allow them to cheat.

ezrameanshelp2• 13 years ago

One CAPTCHA to rule them all!

mholt• 13 years ago

I've been thinking that for a few months now. When reCAPTCHA was revised last year and turned into numbers, I immediately figured it was for Google to read the house numbers on street view images.

Brilliant, really.

VanillaCafe• 13 years ago

But in particular to create a seed training set (for the neural network) -- not as a standalone house number recognition system.

mtrimpe2• 13 years ago

Sorry for the downvote; I though you were off-topic before my brain processed the second sentence.

Edit: thanks for the downvotes for apologizing for an errant downvote ... I guess ...

blueskin_• 13 years ago

I noticed those when they first started appearing, my first thought was "must be Street View", followed immediately by wondering about the potential for poisoning some kind of machine learning effort with them (and yeah, I know, issue of scale, even a group of people dedicated to ruining the effort would be vastly outnumbered by background reCAPTCHA traffic).

danielweber2• 13 years ago

4chan has had campaigns to get "penis" and a certain racial slur inserted into books. I was going to post links but then decided I didn't want to get fired today for my web searches.

chippy• 13 years ago

Gooogle are being totally dishonest and misleading in their use of reCAPTCHA to digitize house numbers.

"reCAPTCHA is a free CAPTCHA service that helps to digitize books, newspapers and old time radio shows."

"Currently, we are helping to digitize old editions of the New York Times and books from Google Books."

They are lying and they are not benefiting the public interest - which is what reCAPTCHA relies on - they are only benefiting their company and their own shareholders.

userbinator• 13 years ago

I just put in rubbish for that part. OCR'ing books isn't bad, but I really don't find the idea of helping Google with house numbers appealing at all.

datawander3• 13 years ago

Like the "youtube cats" paper that only has 16% rate of success (which represented a huge improvement), this doesn't appear to be anything beyond what has already been done for text recognition, which is usually one of the first examples held up of a place Machine Learning has done exceedingly well with 20 years ago.

This line gives it away. If they can remove that assumption I will be impressed, otherwise I would say they reinvented the wheel and probably could have used something off the shelf and got similar results.

"To start off with, Goodfellow and co place some limits on the task at hand to keep it as simple as possible. For example, they assume that the building number has already been spotted and the image cropped so that the number is at least one third the width of the resulting frame. They also assume that the number is no more than 5 digits long, a reasonable assumption in most parts of the world."

DannyBee3• 13 years ago

1. They are doing multi-digit recognition at once, which, as the paper says "To our knowledge, all previously published work cropped individual digits and tried to recognize those". So i don't understand your issue with the 5 digit limit, when everyone else is sticking to "1 digit at a time".

2. Spotting the building number, as the paper says, is taken care of by a different algorithm.

I'm not sure why this is also a big deal, since spotting the building number is "not the hard part" in most cases.

3. Your assertion that they could have used something off the shelf seems directly contradicted by the fact that the paper says nobody has ever published a multi-digit simultaneous recognition paper.

So i'm very curious what this "off the shelf" thing would be. Could you elaborate?

darklajid4• 13 years ago

Disclaimer: I'm working in the OCR industry, which .. doesn't make me an expert on either state-of-the-art recognition algorithms nor do I know enough about neural networks to be dangerous.

That said: The GP has a point, imo. I don't doubt that the paper describes something new and interesting (and all engines I work with during the day do segmentation/recognize character by character), but localizing a region of interest is usually the hard job for me. When I identify the right region and crop it/scale it/rotate it .. my job's "easy" and I can run a multitude of generally good OCR engines (off the shelf, if you will) and get decent results (maybe vote a bit, use engine A to segment and engine B and C to recognize the characters etc.)

So .. ignoring the 'we trained a neural network' part (which makes me nod thoughtfully and mumble 'whatever they did there..'), which I _understand_ is the interesting thing here!, they did more or less what I do all the time. The preceding algorithm is what I do quite a bit less often and which in my environment is more interesting and often challenging.

Then again, it can always be labeled as PEBKAC I assume :)

datawander3• 13 years ago

I stand corrected and wasn't aware of the multidigit claim.

mtrimpe2• 13 years ago

It seems like another practical application of the latest improvements on neural networks.

You can watch this talk to get an idea for what they're about: http://www.youtube.com/watch?v=vShMxxqtDDs

yaroslavvb• 13 years ago

"Off-the-shelf" approach didn't work, which is the reason we did this. And the simplifying assumptions are things we could get away with while solving the task at hand.

Traditional OCR pipeline would be to use some heuristics to find line of text in an image, use some other heuristics to break line of text into candidate characters. Some candidate blobs may need to be merged to make a single character, so you use a separate character classifier pre-trained on correctly segmented characters to score the candidates, and then Viterbi/A* search on those scores to find the most likely interpretation of input.

Many problems with this -- how do you tune the heuristics? How do you recover from error in an earlier stage of pipeline? How do you get character level ground truth from image/text pairs?

With enough engineering time, you can solve those problems, but it's a lot of coding and tweaking. The point of the paper is that you can skip those steps and read OCR output directly off top layers of the network.

datawander3• 13 years ago

Awesome to hear from you! Good points and I believe you're right on all of them.

I thought there was more to it because the headline made me think that the general problem of taking a random StreetView location and finding the addresses in it was solved, which really set some high expectations.

This work is equally impressive and good stuff. Hope to see this headline again in the future soon.

magicalist• 13 years ago

I think the actual paper (linked at the end) is much more informative and interesting than this coverage: http://arxiv.org/abs/1312.6082

(I wouldn't call it blogspam, because it looks like they interviewed the researchers, but the summary leaves something to be desired)

aaronsnoswell• 13 years ago

The MIT guys generally do a pretty good job with their articles. Definitely not blog spam.

danielweber2• 13 years ago

This is unusual: an article about neural nets that actually uses neural nets.

marc0• 13 years ago

I must say I find this fascinating, esp two aspects: First, great idea to train on number sequences instead of single characters. That's certainly a lesson which is applicable in many other situation, too. Second, 11 levels and deep learning -- a bold approach, since (from what I know) the paradigm is that deep networks are generally not working well. They even found that performance improves with the levels (they just stopped at 11, probably because of resource limitations). As mentioned in the paper, this is probably due to the fact that the network ist trained with a huge dataset, so the size of the dataset really is the relevant factor.

How Google Cracked House Number Identification in Street View

Related Stories