Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

http://arxiv.org/pdf/1505.07909v1.pdf

by ilyaeck • 11 years ago

80 21 11 years ago

arxiv.org

jyzzmoe2• 11 years ago

And yet ...

http://arxiv.org/pdf/1412.1897v1.pdf%EF%BB%BF

Houshalter• 11 years ago

Not really relevant to this. That deals with images, where you can perturb a huge number of pixels slightly to exploit weird edge cases. Even linear classifiers break on it, e.g. https://karpathy.github.io/2015/03/30/breaking-convnets/

empiricus• 11 years ago

I'm pretty sure that if you move just a little the image, the convnet is no longer fooled. So this kind of exploit works only for completely stationary images, and it is very to overcome.

JuliaLang• 11 years ago

I can break humans too: https://en.wikipedia.org/wiki/List_of_optical_illusions

jyzzmoe2• 11 years ago

Humans are able to recognize these as optical illusions. (otherwise we wouldn't be calling them "illusions") Further, none of the optical illusions shown are so wildly "off" and "crazy" as the algorithmic goofs demonstrated in the paper referenced above.

breuderink• 11 years ago

Further, these inputs were /optimized/ to confuse the particular classification software. If only we would be able to optimise optical illusions for a given human viewer...

dschiptsov2• 11 years ago

In accuracy of decoding (pattern matching)? What it had to do with IQ?

The decades old system for handwriting digits recignition for US Post beated humans (now it is a few lines of Octave in Andrew Ng's course). Still it cannot write a reply to a letter.)

joe_the_user2• 11 years ago

I'm reading the article and it's talking about solving analogies, antonyms and similar problems ... as found on a standard IQ test.

I understand this isn't intelligence and the title doesn't imply it but I'm not sure what your reference to decoding is about.

dschiptsov2• 11 years ago

But it still seems like pattern recognition based on a training set only, which is, in my opinion, a task on a lever prior to intelligence, like what visual cortex does. It cannot make new reference between words to produce (infer) new antonyms and analogies (not presented in a training g set), which is intelligence.

huac• 11 years ago

How reliable is MTurk for an IQ test? I presume the respondents don't put in that much effort on these quizzes.

compbio• 11 years ago

MTurk is representative of the human population. I hear it often that people presume that MTurkers do not put in any effort for a few cents. But for most it is a hobby/nice pastime. They do it to make a few bucks, instead of playing chess or reading a book. Most people do not get a nickel playing chess, though really put in the effort. MTurkers are similar.

See here for an MIT experiment in blurry text transcription: http://groups.csail.mit.edu/uid/deneme/?p=329 for the unexpected accuracy resulting from crowd-sourcing.

clickok• 11 years ago

Interesting and cool. It's almost like a magic trick, particularly in the sense that an almost unbelievable result has been achieved through putting in a serious amount of effort. It's not just a gimmick, though-- projects like this demonstrate the steady creep of progress on seemingly unsolvable problems[1]. But of course, there's thousands of such problems out there, with millions of potential customers ready to make whoever solves one a billionaire, so even if "true" AI fails to appear, odds are good we'll make a fair approximation for specific applications.

One thing that popped out at me, however, was the distribution of the human scores: monotonically increasing with age, which was somewhat odd. Shouldn't it be more or less normally distributed?

1. I wouldn't have thought to make a serious attempt towards beating a verbal comprehension test with deep learning; the possibility of succeeding would've seemed tiny in compared to the work. Similar to the notion that trying to prove the Riemann Hypothesis is hard to think of as a real pursuit, at best it's like a quixotic hobby.

jphilip147• 11 years ago

It is great to know how deep learning is exploring various possibilities.

one-more-minute2• 11 years ago

Considering that IQ tests have been widely considered as a measure of intelligence, we think it is worth making further investigations whether we can develop an agent that can beat human on solving IQ tests ... we could be a further step closer to the true human intelligence.

This is a bit like saying, "the best runners tend to be pretty tall, and we've made a robot which is really tall – so running robots are just around the corner."

IQ tests certainly correlate well with intelligence (in humans), but they're a metric, not the thing itself. Another metric would be mental arithmetic; people who can do sums quickly tend to be pretty smart, but that doesn't mean that calculators are a step away from super-intelligences.

Interesting and cool work but let's be careful in the interpretation (and remember that people said the same things about chess playing not that long ago).

Jimmy• 11 years ago

People fall victim to this fallacy in their own thinking all the time. They want to get good at doing X, but task Y is a lot easier and success at Y is mildly correlated with success at X, so they start practicing Y instead of X. It's the reason why things like "brain games" are so popular. In reality, people's time would almost always be better spent just practicing the primary task they want to be good at.

That being said, making an AI that does better on IQ tests than humans is a rather interesting and worthwhile endeavor.

Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

Related Stories