IBM's Watson starts on Jeopardy Tonight

http://www.jeopardy.com/minisites/watson/

by KevBurnsJr • 15 years ago

245 143 15 years ago

Is there somewhere to watch this online?

andrewce2• 15 years ago

If you're okay with waiting a few hours/days, The Pirate Bay seems to have a good selection of torrents of recent Jeopardy! episodes, though I've not yet tried it.

You'll also be able to see a full transcript of the board, as well as all questions provided by contestants, at www.j-archive.com.

patrickod• 15 years ago

I had thought from the counter that IBM had run on the Watson page that there might be but it seems that us non-US residents will have to wait. Still very much looking forward to this. History in the making.

nika• 15 years ago

Hulu refers to crackle.com. Crackle has "minisodes" which seem to date from 2002-2003.

It is owned by Sony, distributed by CBS and seems to be syndicated, which means there is no network that carries it specifically, and thus no online distribution, as each market has a different station (Eg: Fox might buy it in one market while ABC might buy it in another).

This is near as I can tell looking into it.

jackowayed2• 15 years ago

Looks like IBM will have it online in a few days http://twitter.com/#!/IBMWatson/status/37223337453158400

andrewce2• 15 years ago

Thanks for posting this. I got called into work and missed today's Jeopardy! episode, so seeing this link was definitely a relief.

jc123• 15 years ago

Found http://www.youtube.com/watch?v=BfNBWJTGEEA and it is good quality.

umjames• 15 years ago

Much of success on Jeopardy is not just deciphering clues in the answers, but your timing on ringing in to give the question. I'd imagine a machine could get really good at getting the timing down. Does Jeopardy have a way of varying who rings in first to keep things more fair?

cj• 15 years ago

Watson won't ring in until it determines it has enough confidence in its answer that it is favorable to ring in.

If it rings in every time there's no way it will win.

cryptoz2• 15 years ago

The timing of when players are allowed to click in is controlled by a human. Someone backstage decides at what moment Alex is finished speaking and then opens the clickers. If you click too early, you get ~ 300ms delay penalty which gives the other players a chance to click in.

Click timing is indeed very important but I do not think Watson has any special advantage there.

jackowayed2• 15 years ago

But there's a visual cue letting the players know that clickers are open (a light flashes, I think), so the computer probably gets some notification too, at which point it could instabuzz.

kls• 15 years ago

Not it does not have a special advantage but it also does not interface with the questions in the way that humans do, it cannot hear Alex or read the question on the screen, the questions are messaged to Watson via something akin to a text message or an email.

kenjackson4• 15 years ago

Something I wrote from a previous thread. The one piece of info that wasn't in this comment is that Watson can NOT anticipate when Trebek is about to finish the question. Watson must wait until it gets a signal that the buzzer is now available, and only then can it begin the process to physically depress the button.

Remember that Watson also has to depress a physical button (the same buzzer everyone else uses). The eye to finger path for humans is about 200ms. It probably takes about 100ms for Watson to physically press the button. So Watson is about 100ms faster. But that also gives humans about a 100ms window in which to beat Watson. This means that you need to start your press 100-200ms before Trebek finishes his last word.

That's pretty good sized window for most people given you are reading the question along with Trebek. If the person who turns the light on is very consistent, I think a human who is good at this could consistently beat Watson.

fefzero3• 15 years ago

Another key to remember is that there is time after the button press to think of the answer - it's a common strategy to ring in when you have a certain confidence level that you can think of the answer quickly. Watson uses confidence levels, too, but he already has a particular answer in mind instead of a vague confidence level for the whole clue.

rkudeshi• 15 years ago

Interesting fact: during Ken Jennings' original 74-game winning streak, he felt it was an unfair advantage he had so much experience with the buzzers. He asked the producers to allow the other contestants (competing for the first time each episode) time to practice with the buzzers, which they did.

ugh7• 15 years ago

That’s Jeopardy. Whoever buzzes first gets to answer, I don’t see what’s unfair about that. Computers can react faster than humans, why should they be denied that advantage?

BoppreH• 15 years ago

Because the challenge proposed is not about reaction times.

kirpekar• 15 years ago

Thanks for the reminder. DVR set.

sambeau• 15 years ago

A valentine's day gift for us single geeks

jazzyb• 15 years ago

Speak for yourself. My girl-friend is sufficiently geeky that she suggested we watch it together on Valentine's.

3pt141592• 15 years ago

http://www-03.ibm.com/innovation/us/watson/what-is-watson/co...

Has a nice little video too.

quickpost• 15 years ago

I'm glad they touched on the idea that this is actually a "Human vs Human" competition... Really good Jeopardy players vs. a team of humans that built their own "Jeopardy Player".

Watson winning the tournament is a triumph for humanity, not just for machines.

kj12345• 15 years ago

One thing I'm interested in is any skew in the questions from normal. In particular I hope they ask linguistically tricky questions where you can't even figure out what's being asked at first. I felt like they went a bit easy on that front in the preview round:

http://www.youtube.com/watch?v=WFR3lOm_xhE

kenjackson4• 15 years ago

I doubt they are skewing the questions. That kind of defeats the point. I suspect Jeopardy is constantly adding to a pool of questions. My hope is that they simply grab from that pool just like any other episode.

Note, the hardest questions for Watson are short questions -- the reduced time hurts Watson more than most human competitors.

random42• 15 years ago

Also, smaller questions provide, much lesser context for Watson to work with.

fefzero3• 15 years ago

The writers had no knowledge of which clues would be used for the Watson episodes; they only mention that they don't use audio or video clues. I certainly haven't done any analysis of it, but I didn't feel like they were any less common in the preview round than normal. Between all the games I'm sure we'd see more "linguistically tricky" clues. I'm hoping for some good "before and after" questions.

brown9-26• 15 years ago

You can watch the PBS Nova episode about the building of Watson for free on their website (about 60 minutes long, aired Feb 9 2011):

http://www.pbs.org/wgbh/nova/tech/smartest-machine-on-earth....

One question I have about Watson that I don't recall being mentioned in any videos or articles so far - what sort of interface does Watson receive the questions over? Is Watson performing speech recognition or getting the text of the question via some sort of interface?

lrm2423• 15 years ago

This was answered in the Nova special, in passing. The question is fed to Watson as text at the same moment it appears on the display that the contestants can see. Likewise for correct answers. Watson is not performing speech recognition.

brown9-26• 15 years ago

Thanks, this is what I assumed (as ASR of these questions is probably just as hard as the interpretation).

How does Watson know when it can chime in then?

SMrF2• 15 years ago

The documentary mentioned that Watson does some speech recognition. It attempts to listen to the competitor's answers so that it doesn't give a wrong answer that was already given.

dstein3• 15 years ago

The question is fed to Watson as text

Aw this seems rather half-assed to me. I hope at a later time they do another one of these with a new and improved Watson. I really want to see a humanoid robot standing on the podium (connected by wifi to the supercomputer) and using OCR and speech recognition. I also think Watson's voice should be more authoritative. It's like they went to extraordinary effort to make Watson sound unimposing.

deltaqueue• 15 years ago

What about video clues? I assume there is a transcript, but sometimes there's a visual reference. Perhaps some sort of integration to Tineye.com or advanced video recognition? :-)

jasonkester2• 15 years ago

(if you're in the United States)

Amazing that PBS is locking down their videos. They're public television fer cryin' out loud.

xxpor• 15 years ago

So is the BBC.

Xpirate• 15 years ago

What's amazing to me is NPR, PBS, CBC and BBC haven't gotten together to make their own public domain codecs. It's kind of their job to do this stuff. I guess if you take contributions from "The Bill and Melinda Gates Foundation", you got make some compromises.

ck2• 15 years ago

PBS is basically "owned" by the corporations that advertise now in front of every show. Congress keeps cutting back their funding and viewer donations are a fraction of their operating budget.

In a few tiny ways their low budget is a good thing, ie. PBS Newshour doesn't have the insane stupid graphics/animations/toys that mainstream news does.

But in almost every other way their low budget in ruining the experience now.

smackfu• 15 years ago

You still have regional concerns. Some are purchased from the BBC (like some episode of Horizon or Imagine) or are co-funded (like the Earth series). Those contracts will only allow viewing in the US.

icefox3• 15 years ago

The real question I have is according to that episode Watson does not take into account the category only the 'answer' and the already shown questions. It seems _really_ odd that they would ignore this bit of information.

rst• 15 years ago

IIRC, they said they fixed that.

jarin• 15 years ago

Since there are a lot of pun-based categories on Jeopardy, I can understand why they would have problems with it.

mcav2• 15 years ago

It looked like they were using Java. (Unless that was just fabricated for the screen.)

brown9-26• 15 years ago

I was curious about the same thing - did they ever mention what languages were used in the documentary?

The screenshots looked like Eclipse with Java. The brief snippets I could make out certainly looked like they had to do with evaluating rules and scoring judgment, but that could have easily been stock footage.

caustic• 15 years ago

Apache UIMA is one of the building blocks of Watson -- https://cwiki.apache.org/UIMA/powered-by-apache-uima.html And UIMA is written in Java and C++.

crocowhile• 15 years ago

That link just showed me 30 seconds of some commercial and AFTER told me that I cannot access their contents due to geographical restrictions.

ugh7• 15 years ago

It’s a shame that this is (necessarily!) such an insular challenge. Everybody knows what chess is all about, I fear that the impact of this game will be limited to the US or the Anglosphere. Just as an example, there has been no Jeopardy on German TV since 2000, it’s not really a part of German pop culture and because of all the puns it doesn’t translate well.

(Question for native speakers: When watching the practice round [0] are you generally able to keep up and answer the questions? The speed with which the game was moving made it nearly impossible for me to follow or enjoy the game. I would like to know what the experience is like for native speakers.)

[0] http://www.youtube.com/watch?v=WFR3lOm_xhE

fefzero3• 15 years ago

I can definitely follow along with this or any game of Jeopardy!, but my problem is that I just don't know all the answers. To me this is part of the appeal.

This is definitely faster-paced than regular episodes of Jeopardy!, but I don't think it's prohibitively fast for native speakers.

It's too bad that Jeopardy! doesn't fit in with international pop culture better. I think it's a perfect fit for a lot of AI problems.

3pt141592• 15 years ago

Typically real jeopardy will show the text on the screen. I can get about 30 (sometimes up to 35ish) "questions" for a normal show but it is all about reading it to get those extra few brain cycles.

metageek4• 15 years ago

>Everybody knows what chess is all about,

Well, that's kind of the point. Chess is a hard game, but it doesn't require broad knowledge of a human culture. If you're going to build a machine that exhibits such knowledge, you've got to pick a culture.

ugh7• 15 years ago

That’s why I wrote “necessarily!” Jeopardy is certainly a great pick, I can’t think of anything better.

bayleo• 15 years ago

Don't fret; you can still tune in after another decade or so for a machine to beat us at GO. Given IBM's trending business interest in corporate AI I would not be surprised if it were another IBM sponsored team that finally builds the successful bot.

KevBurnsJr2• 15 years ago

Is there somewhere in Silicon Valley to watch this together in person?

wmf• 15 years ago

You could try to sneak into IBM Almaden.

kenjackson4• 15 years ago

The geek equivalent to the sports bar... There must be some website for doing this, right?

KevBurnsJr2• 15 years ago

Anyone have any connections at HackerDojo?

metageek4• 15 years ago

Actually, that might be your answer: if 100 geeks show up at a sports bar and ask them to turn one of the TVs to Jeopardy, maybe they'll do it.

ajays3• 15 years ago

I just walked into my local bar, asked the bartender to switch to channel 7, and voila! There weren't too many people around, but it was fun (and between breaks I chatted with a couple of drunk chicks who were trying to drown their V-Day sorrows in shots of vodka).

swanson• 15 years ago

"a';DROP TABLE 'knowledge'; This type of attack is commonly used by hackers."

Game over, Watson.

zach• 15 years ago

Ken should totally go for the Bobby Tables response if there's a Daily Double he doesn't know.

That or the Star Trek self-destruct sequence.

nostrademons• 15 years ago

Really, who would put a massive store of knowledge in a SQL database?

powrtoch• 15 years ago

Anybody know whether this will be live or if it's pre-taped like most jeopardy episdoes?

metageek4• 15 years ago

I believe I saw a report recently that they had finished taping it.

shortlived• 15 years ago

In case anyone is interested to know Watson's opponents: Watson will compete against Brad Rutter, the current biggest all-time money winner on Jeopardy!, and Ken Jennings, the record holder for the longest championship streak [source: wikipedia]

icedpulleys• 15 years ago

In case anyone missed it the first time around, the nytimes mag had a pretty good writeup on Watson back in June -- might be worth instapapering and reading later if you're going to catch the broadcast this week:

http://www.nytimes.com/2010/06/20/magazine/20Computer-t.html...

As an AI researcher I'm excited to watch this week. Even if it's not the most elegant artificial Jeopardy player imaginable, it raises the public profile of a lot of AI & ML topics and might encourage and inspire other groups to tackle ambitious projects.

tpatke• 15 years ago

I assume this is pre-recorded and IBM would have cancelled it if they didn't like the outcome.

T-hawk• 15 years ago

It was pre-taped a few weeks ago in line with Jeopardy's usual schedule. There were a number of untelevised practice sessions beforehand against other Jeopardy contestants and wannabes. IBM did their tweaking then and promised that the actual competition rounds would be televised regardless of results.

radioactive21• 15 years ago

Also if IBM were to cancel an episode that showed bad results, it'll eventually leaked out, especially for project of this caliber. It would be an instant negative publicity, since EVERY tech site would pick it up.

You'll have the scorn of all the geeks on the internet, which could be interesting lol

waxpraxis• 15 years ago

I'm currently finding out what information we're allowed to share about how the avatar works and what went into developing it. The problem is we're so far down the totem pole I probably won't know for a while yet. :-/

http://automatastudios.com/automata-gives-ibms-watson-a-face...

tremendo• 15 years ago

http://twitter.com/robotwisdom has some results from the show, don't know how he was able to see it already.

Kilimanjaro• 15 years ago

I'd like to see Watson as a web service.

Retric• 15 years ago

As would I. However, due to the computer resources it takes per question it would be extremely expensive to scale.

Watson is made up of a cluster of ninety IBM Power 750 servers (plus additional I/O, network and cluster controller nodes in 10 racks) with a total of 2880 POWER7 processor cores and 16 Terabytes of RAM. Each Power 750 server uses a 3.5 GHz POWER7 eight core processor, with four threads per core. and it still takes ~15 seconds a question.

PS: Rough order of magnitude: 90 IBM Power 750 ~= 180 EC2 High-Memory Quadruple Extra Large Instance * 15 seconds ~= $1.50 a question. (Edit: that's probably still low, each of those 750's have 16TB / 90 = 180GB of ram).

Edit2: Ouch 90 times 32 core IBM Power 750 list price is 492,210$ a month or 2.84$ per 15 seconds (http://www-03.ibm.com/systems/power/hardware/750/browse_aix....) and those only have 128GB of ram,

patrickgzill• 15 years ago

That would be a cheap promotion for IBM, who presumably is getting everything at wholesale price :-)

phren0logy• 15 years ago

SPOILER Link to final score for those of us who are curious, but not curious enough to watch:

http://twitter.com/robotwisdom/status/37200070742904832

ars• 15 years ago

Thank you for not spoiling it in the text of your message.

ugh7• 15 years ago

This is the final score for today, the match will continue tomorrow and the day after tomorrow.

Charuru2• 15 years ago

Shouldn't IBM start their own search engine if they have algorithms like this?

Or conversely, maybe Google should buy them?

yumraj• 15 years ago

Or conversely, maybe Google should buy them?

I wish you had checked the market cap of IBM and Google before suggesting that :). IBM > GOOG

Charuru2• 15 years ago

You're right. Looks like IBM is doing pretty well.

osipov• 15 years ago

Looks like Google and IBM are evenly matched in terms of market cap so a buy out is unlikely http://www.wolframalpha.com/input/?i=IBM+GOOG+market+cap

QA technology is very different than search because the former makes strong assumptions about quality of the underlying data. Google's search is much better at filtering out and dealing with low quality content (e.g. blogs) and spam.

alxp2• 15 years ago

Hah now Wolfram Alpha, I wonder if they're going to bother coming back in to work tomorrow. Watson makes it look pretty primitive.

icegreentea• 15 years ago

IBM wants to sell the stuff that's Watson's based on to companies. Companies will get their own 'answer engine'. I guess they think that licenses will make them more money than ad-money for search engines. Which could well be true in this case. Watson's insane knowledge still won't give me the cat pictures I want.

BenSS• 15 years ago

Internet search is a completely different domain than this sort of language lookup. Also, look at the hardware required to run this!

tastybites• 15 years ago

maybe Google should buy them?

The web is not computing.

stcredzero• 15 years ago

IBM should use this tech to one-up the content-scraper SEO junk sites. They should put out sites that are what the scraper sites purport to be and put them out of business.

jal278• 15 years ago

I'm sure there are those that disagree strongly, but I feel like this isn't as much an advance for AI as it is another interesting combination of filtered human-structured knowledge and computation power. Just as Deep Blue brute-forced chess, this is mostly a brute-force of another, albeit more open-ended domain (i.e. Q/A).

I'm not arguing that this isn't an impressive accomplishment, but that the statistical-learning stream of research is likely a conceptual local optima that yields the best results in the near term but is probably unrelated to the way we ultimately achieve a creative, general AI.

olalonde• 15 years ago

Suppose Watson scales cheaply (it doesn't), would it be serious threat to Google? My guess is that it would be but I'd be curious to hear HN's take.

bryanh• 15 years ago

I think it's fun to think about how Watson type intelligence will be at the average consumer's figertips (and affordable, to boot) in less that 20 years.

A quick look at the TOP500 supercomputers puts about a 10,000x increase in raw flops since 1993. Using that as a rough benchmark (a very rough one), we're looking at some impressive stuff in the next twenty years. It's not unreasonable to think that:

    current consumer processor in FLOPS * 10,000 < Watson's grid's FLOPS

Naturally, that is a very rough estimate and has no scientific bearing at all.

whatusername• 15 years ago

For Reference: The laptop I am typing on is more powerful (in terms of FLOPS) than Deep Blue.

(For that matter - so is every server in Watson)

lorax• 15 years ago

I hope someone writes down all the answers (questions?) and feeds them into google so we can have a Google-Watson showdown. (Oh, we need a third contestant, how about bing too).

SoftwareMaven• 15 years ago

Is it bad that I thought "Oh, so you are going to have Google compete twice"? ;)

It would be interesting to feed to clues into Google to see what comes out. There are easy ones like http://www.google.com/search?hl=en&source=hp&biw=115... but I'm sure there are also some that Google wouldn't have a clue.

pbhjpbhj3• 15 years ago

I was interested in this too - http://www.reddit.com/r/technology/comments/fl477/tonight_a_... was the first few clues fed in to Google.

What I found was that the answer was in the snippets on the first SERP but certainly for some of the questions it seemed pretty hard to parse it out.

Also I think Jeopardy got 1 or possibly 2 answers wrong

- White City was the City of the 1908 Olympics, it was in London Borough or Greater London but not in the City of London (http://en.wikipedia.org/wiki/City_of_London). White City is a borough and not a city, which makes this a really hard question for a computer to answer I think.

- Nagini killed whatever Harry Potter character was mentioned ; I'm not familiar with the series and didn't see this question this was just based on a comment on Reddit and a Google for "killed $CharName" (I have a really bad memory).

tocomment2• 15 years ago

Why does Watson need so much computing power? I don't know of many NLP algorithms that are so intensive?

brown9-26• 15 years ago

Because in order to answer a question Watson is really executing hundreds or thousands of different algorithms in parallel to attempt to find candidate answers, and then comparing and building confidence scores for each.

tocomment2• 15 years ago

Thanks. Is there anything I can read to learn more about how it works in detail? All the mainstream media articles I've read are very hand-wavy.

AARC233• 15 years ago

This is interesting. Way to mix it up Jeopardy

thought_alarm• 15 years ago

And be sure to stick around for Wheel of Fortune, where two previous champs will take on an Apple II+. Beep.

savrajsingh• 15 years ago

Did anyone else notice that the vignettes about Watson's creation featured IBM researchers using MacBook Pros? So much for "International Business Machines." ;)

bjg• 15 years ago

As a former IBM employee, you can pretty much use what ever type of machine you want for your work as long as it doesn't interfere with what you do. There is a significant Mac subculture internal to IBM, and they love and suppor it. However a huge percent of their 600,000 employee's still use IBM branded laptop's and desktop's.

gursikh• 15 years ago

(This is just for kicks.)

Assuming the space required to run watson halves every two years and v:: of the human brain is v1 = 1500cm³, and Σv of Watson's servers = v2 = 90 x (17.5cm x 44.0cm x 73.0xcm) = 5.05e6 cm^3.

So Watson will be the size of a human brain in t = - ln(v1/v2)/ln(2).

We're looking at 11.7 years. So, 2023.

* Source: http://en.wikipedia.org/wiki/Watson_%28artificial_intelligen...

"Watson is made up of a cluster of ninety IBM Power 750 servers..."

* Source: http://www-03.ibm.com/systems/power/hardware/750/specs.html

"175 mm x 440 mm x 730 mm"

IBM's Watson starts on Jeopardy Tonight

Related Stories