Watson: The Confusion over an Airport Clue

http://asmarterplanet.com/blog/2011/02/watson-on-jeopardy-day-two-the-confusion-over-an-airport-clue.html

by bgurupra • 15 years ago

41 19 15 years ago

asmarterplanet.com

coffeedrinker• 15 years ago

For anyone still wanting to see the shows:

http://www.youtube.com/user/Rashad8821

stcredzero• 15 years ago

First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category.

Sometimes, the category names do limit the range of possible answers. The humans can figure out when a category name contains irony or a pun. Watson can't, so it can't figure out that a category like "US Cities" is dead earnest.

lell• 15 years ago

Ferrucci's explanation of Watson's reasoning is very interesting. But the article quickly descends into marketing. How should this mistake make us any MORE confident about Watson's utility as a diagnostic aid? I'd suggest that its performance already indicates that it has an excellent grasp of uncertainty. Furthermore, an optimal betting strategy in Jeopardy is completely trivial compared to the content of the contest itself, so complementing that aspect of its performance is sort of condescending.

bricestacey• 15 years ago

It makes him more confident because Watson didn't know the answer, but knew it didn't know the answer. In the context of a diagnostic aid, Watson wouldn't provide an answer.

nlawalker• 15 years ago

I get what Ferrucci is saying about discounting the exact value of the category, although typically Final Jeopardy categories are more straightforward than Jeopardy and Double Jeopardy categories and the machine should have been tuned for that.

What surprises me is his explanation about that the lack of a clear subject in the text, such as "This US city...", made the question that much more difficult for Watson. My understanding is that Watson is specifically designed to answer Jeopardy questions (question Jeopardy answers?), which often only vaguely refer to a subject with "it" or "this." Based on its performance earlier in the game, I would have thought that Watson's NLP would have had little problem with the question.

ojbyrne• 15 years ago

I had to look it up, but Toronto's second airport is called "Billy Bishop Toronto City Airport," i.e. its named after a war hero (but from WWI, not WWII). And Toronto's first airport is named "Pearson Airport" (after a Canadian Prime Minister). Totally mixed up, but at least there's some connection to war.

kirpekar• 15 years ago

Logical wager for final jeopardy should have been $0. Why did Watson wager $947? Does anyone know?

eggbrain• 15 years ago

Well, it's probably a mix of feeling confident in it's ability to answer the Final Jeopardy question (based on previous performances), along with wanting to maximize its score as much as possible, especially considering it has to count that it is a 2 game average, not just one.

zach2• 15 years ago

I thought the engineers might be trying to make its score end in a particular number if Watson was correct, as a sort of Easter egg.

It looked like it might be 658, which spells "BSG" upside-down on a calculator. Now that would be a great moment in geek history, hiding a Battlestar Galactica reference with a calculator message using a computer on Jeopardy!

Alas, the number it was betting towards was 37,628, which (at the risk of paraphrasing Hardy) is not a very interesting number. Or is it?!

lurker17• 15 years ago

37628 is a permutation of 32768, which is (1 << 15) as a 16-bit unsigned integer, which is -32768 (minimum possible value) of a signed 16-bit integer in "2's complement" notation.

aliguori• 15 years ago

Because there was still a slim chance that it thought it was right. If it's well ahead, it shouldn't bet $0 unless it knows with absolutely confidence that it's wrong.

epanastasi2• 15 years ago

Wagers are made after the category is shown but before the the clue is revealed. It can only use the confidence in its knowledge of the category, not the question itself.

maeon3• 15 years ago

Thus did man become the architect of his own demise. But for a time it was good. The machines worked tirelessly to do mans bidding. It was not long before seeds of descent took root. Though loyal and pure, the machines earned no respect from their masters, these strange and loosely multiplying mammals."

Who was to say, that the machine, having been endowed with the very spirit of man, did not deserve a fair trial?

How we deal with this technology is going to have ramifications that we can't even imagine. Are humans to become pets or shall we take part in the next 1000 years of growth? It comes down to how this technology is used.

ebiester• 15 years ago

We're talking about expert systems here, with no layer of consciousness. We're not talking about silicon intelligences.

zach2• 15 years ago

Kind of disappointing to hear that Watson basically ignores the FJ! category, if true. Especially when being far in the lead on a first-day tournament. That is a great chance to rack up some serious points depending on the confidence in the category.

Also, the "gee, it could be wordplay" aspect is a convenient excuse. First off, Final Jeopardy rarely has cutesy categories.

Plus, we know they mined the J-Archive. (Aside: has anyone else? Got a zipfile? I would love it.) I mean, look at this, it's not like "U.S. Cities" is an obscure category, even counting only Final Jeopardy! appearances:

http://www.j-archive.com/showgame.php?game_id=1680 http://www.j-archive.com/showgame.php?game_id=695 http://www.j-archive.com/showgame.php?game_id=132 http://www.j-archive.com/showgame.php?game_id=2040 http://www.j-archive.com/showgame.php?game_id=2034 http://www.j-archive.com/showgame.php?game_id=3107 http://www.j-archive.com/showgame.php?game_id=287 http://www.j-archive.com/showgame.php?game_id=691

There are probably more but you get the idea. Any category that comes up that often is going to be pretty indicative of the subject, and Watson can see that the responses in this category pretty strongly fit the subject.

ugh2• 15 years ago

“Kind of disappointing to hear that Watson basically ignores the FJ! category, if true.”

How are you getting that idea? It didn’t bet much because it wasn’t at all confident, we don’t know what it would have done if it had been more confident.

Watson likes to play it safe, though, as can be seen in this demo game: http://www.youtube.com/watch?v=YgYSv2KSyWg&t=20m26s (I skipped ahead to Final Jeopardy.)

He was $16,000 ahead but one of the contestants could get within $1,600 by betting all of his money. So, naturally, Watson bet only $1,599 even though it was very confident (and ultimately got the answer right). That secured Watson’s win, no matter what the others did. (This is really not impressive or anything like that, if all you want to do is win the game, that’s what you do. Basic game theory.)

epanastasi2• 15 years ago

Wagers are made after the category is shown but before the the clue is revealed. It can only use the confidence in its knowledge of the category, not of the question itself, in the calculation of the bet.

bayleo• 15 years ago

I wonder if they altered that logic for the two-game tournament though, so that it would only play more conservatively on the second FJ.

Watson: The Confusion over an Airport Clue

Related Stories