IBM Watson's team Q&A on reddit

http://blog.reddit.com/2011/02/ibm-watson-research-team-answers-your.html

by hootx • 15 years ago

85 28 15 years ago

blog.reddit.com

Some interesting nuggets in here, I had watched the Nova specials on Watson etc. I would have liked to have had a question about the team, their work stress etc., but otherwise a fun read. I especially enjoyed the step by step parsing and examination of a question in the process of how Watson would work through it.

trickjarrett4• 15 years ago

I also feel that they sort of jogged around the buzzing in question. Obviously Watson has to calculate and decide his answer, but there is no denying that he was very fast on the buzzer in the game.

judofyr• 15 years ago

From the Q&A with Ken Jennings: http://live.washingtonpost.com/jeopardy-ken-jennings.html?hp...

    Q: Seemed to me, for many of the questions, that the computer was just
    better at buzzing in. Does Watson have an unfair advantage for timing the
    buzz-in?

    A: As Jeopardy devotees know, if you're trying to win on the show, the buzzer is
    all. On any given night, nearly all the contestants know nearly all the
    answers, so it's just a matter of who masters buzzer rhythm the best.

    Watson does have a big advantage in this regard, since it can knock out a
    microsecond-precise buzz every single time with little or no variation. Human
    reflexes can't compete with computer circuits in this regard. But I wouldn't
    call this unfair...precise timing just happens to be one thing computers are
    better at than we humans. It's not like I think Watson should try buzzing in
    more erratically just to give homo sapiens a chance.

Jd2• 15 years ago

Homo sapiens didn't have a chance in the first place.

trickjarrett4• 15 years ago

Agreed. I don't want them to offset Watson's raw potential, but why not just acknowledge the processing power put him ahead of the buzzer.

ars2• 15 years ago

Seems to me they should have chosen harder questions.

They should try to pick questions such that the contestants only know about 1/3 of them.

Then lets see how the computer does.

This is (should be) a contest of knowledge not buzzing.

patrickod• 15 years ago

while yes human reaction times are nowhere near as good as those achievable with computers I think the physical pressing of the buzzer was enough to level the playing field. I can't remember where I read it but it seems that at most Watson had 100ms of an advantage if he was confident of his answer before the buzzers opened. I also can't think of a consistent way of getting rid of this time gap.

gojomo4• 15 years ago

Parts of the game were eliminated to give Watson a chance: audio/video clues, and categories that required extra explanation.

If the exact timing of buzzing in – and not 'first' but 'first after a light goes on'– gave an overwhelming advantage to Watson, that could be eliminated, too. For example, why not just 'first to buzz in'? (That is, no penalty for being early?) Let Watson's parsing of the question text race the humans' sight-reading.

It looked to me like the humans usually won the buzzer on short questions, which strongly suggests it was the slowness of reading long questions that put it on par with the humans in knowledge, and then it crushed them on buzzing-after-the-light precision.

ddlatham• 15 years ago

Agreed. I wonder out of the cases where both human and Watson had an answer ahead of time and were waiting for the buzzer light, what percentage did Watson win?

If it's much higher than 50%, I would say that a significant portion of Watson's advantage derives not from the language processing but the response time / mechanical advantages.

Jd2• 15 years ago

Question 3 was the most interesting but data on parsing remarkably incomplete. As far as I can tell, we have only lists of possible ways to break down the data without any explanation of how or why one possible way is preferred to another.

Case in point (1):How it decides to treat "Treasure Island" as a proper noun. We see only "modifies(Treasure, Island)" -- indicating that it treats treasure and adjective modifying island, then suddenly in the semantic assumption phase they are treated as a compound.

Case in point (2). We are given:

        island(Treasure Island)

        location(Treasure Island)

        resort(Treasure Island)

        book(Treasure Island)

        movie(Treasure Island)

I assume what he is giving us is method names written in Java with "Treasure Island" as the single argument that return a value indicating the likelihood that "Treasure Island" is what the method name refers to. This is extraordinarily interesting. However, it is not at all clear which methods are chosen and why, if they are run in some sort of sequence or simultaneously, etc .

Case in point (3) : "Builds different semantic queries based on phrases, keywords and semantic assumptions." This is very vague but indicates that Watson generate a set of queries which it runs against its own internal search engine ranking answers presumably based on the quality of the initial search and the confidence of the answer. Would be very very cool to have an example.

All in all, wets the appetite but leaves one wishing for more hearty fare (or a job at IBM!).

ppod• 15 years ago

A while ago I submitted a link to a blog post( http://bit.ly/igJeRB ) I did that goes into the system in a bit more depth, based on a paper IBM published - there's a link to the paper (open access) there too.

For your cases, from what I got from reading papers:

(1) You could spend two weeks only reading papers on noun-compound semantics. Try a google scholar search just to get an idea of the volume of research. A simple technique to test how idiomatic a phrase is would be a Bayesian type test to see how many times "Treasure Island" occurs in a corpus divided by how many times "Treasure X" and "X Island" occurs. In this case the capitalization probably cues it to look up Treasure Island in Freebase. Interesting thought actually - do the contestants also get the question as text? I think they do, so they get capitalization.

(2)I would be pretty sure these are not Java methods, I'd say they are logical predicates representing the fact that 'Treasure Island' is returned as being a member of the set of things indicated by the predicate, as returned either by syntactic processing (island) or from the knowledge bases (WordNet, Yago, Freebase, dbpedia)

(3) there isn't a worked example in the paper, but my idea of this is that it's basically Watson's way of figuring out what queries to type into it's unstructured text corpus (they have a corpus of web snippets indexed with lucene)

natrius• 15 years ago

For more on (2), see http://en.wikipedia.org/wiki/First-order_logic

lmkg• 15 years ago

> I assume what he is giving us is method names written in Java with "Treasure Island" as the single argument that return a value indicating the likelihood that "Treasure Island" is what the method name refers to.

The sample queries struck me as Prolog rather than Java, even before they mentioned it. Using Prolog would allow testing the combination of various propositions in addition to the individual values. This would imply that they derive a list of possible assertions, then try combinations of those assertions and see what values of X will match those assertions. How they avoid combinatorial explosion when testing all those assertion combinations, I have no idea, but it must involve ranking and pruning.

baddox• 15 years ago

One could make the argument that since Watson is trained with English information and English Jeopardy! clues, English is Watson's native language. Sure, there's Java down to Assembly beneath Watson's understand of English, but the same goes for native English-speaking humans. English speakers aren't biologically any different than, say, French speakers.

gojomo4• 15 years ago

Alas, my somewhat-skeptical question came in late and got little support:

http://www.reddit.com/r/IAmA/comments/fnfg3/by_request_we_ar...

What determined the use of exactly 10 racks of 9 maxed-out (32-core, 512GB RAM) 4U Power750 servers? For example, would Watson have done better with more hardware? Or could it have made-do with far less, after all the bulk pre-processing of, and training on, source material was finished?

(My intuitions about the necessary amount of reference data and topical associations – written up at http://redd.it/fnixm – made me think way less hardware should have been required, at least at the very end during the match.)

whatusername2• 15 years ago

Tony Pearson wrote a bunch of stuff about Watson. https://www.ibm.com/developerworks/mydeveloperworks/blogs/In... Had a bunch of details including some performance details...

Single core -- 1 -- 2 hours

Single IBM Power750 server -- 32 -- < 4 minutes

Single rack (10 servers) -- 320 -- < 30 seconds

IBM Watson (90 servers) -- 2,880 -- < 3 seconds

https://www.ibm.com/developerworks/mydeveloperworks/blogs/In... Had details on the data/storage/ram:

When Watson is booted up, the 15TB of total RAM are loaded up, and thereafter the DeepQA processing is all done from memory. According to IBM Research, "The actual size of the data (analyzed and indexed text, knowledge bases, etc.) used for candidate answer generation and evidence evaluation is under 1TB." For performance reasons, various subsets of the data are replicated in RAM on different functional groups of cluster nodes. The entire system is self-contained, Watson is NOT going to the internet searching for answers.

blhack2• 15 years ago

I wonder if this is marketing.

IBM makes BIG HUGE MASSIVE (tm) server clusters that have lots of blinkenlights and require lots of power and are so crazy and huge and awesome that the people working at IBM must be hyper-geniuses!

Watson runs on a laptop.

brisance2• 15 years ago

I can't see how that would be marketing. Most people would be more impressed if Watson could run on a laptop, today. I know I would.

whatusername2• 15 years ago

Watson can run on a laptop.

It just can't get the answers fast enough to win Jeopardy. https://www.ibm.com/developerworks/mydeveloperworks/blogs/In...

IBM Watson's team Q&A on reddit

Related Stories