A rising sentiment that IBM’s Watson can’t deliver on its promises

http://gizmodo.com/why-everyone-is-hating-on-watson-including-the-people-w-1797510888

by artsandsci • 9 years ago

503 278 9 years ago

gizmodo.com

oculusthrift• 9 years ago

the articles content is more just background about watson than having to do with the the title.

as a side note: it just sounds wrong to have the words "hating on" in an article title.

throwaway_ibm• 9 years ago

I know someone who is intimately involved with IBM Watson, they are highly educated and constantly diss the system. Calling it, 'Just a large database'. If Watson was a true breakthrough, it should be gaining marketshare throughout it's specialities but it's not. Google is leading the industry with DeepMind; Facebook and Microsoft aren't far behind. I'd encourage others to be very skeptical of the PR that IBM is pushing about their Watson problem.

disclosure: I haven't read the article but wanted to share a related story.

peteretep• 9 years ago

A couple of years ago I was given a project that was essentially "Evaluate Watson APIs to see if there's anything there we could make use of", and came away with the distinct impression that it was largely smoke and mirrors, and there was very little that was either effective or interesting there.

drvdevd3• 9 years ago

Could you elaborate a bit? I'm not affiliated with IBM or anything, just curious what you were looking for and what was missing.

dr_zoidberg• 9 years ago

I'll give you my experience. Manager asked me if we could use the Watson Speech-to-text API a couple of years ago, because he was impressed with the ads he'd seen. I worked for a week. Turns out the "dialect" of spanish that is spoken in Argentina (and this was tested against porteños, not even the accents from the provinces!) is utterly unsupported. Even when speaking slowly, it'd throw a random set of words with even more random looking probabilities.

Manager was expecting that after a week we'd have something akin to Googles "hello google!" at hand, through Watson. It took a couple of coworkers to come up with even more impressive broken examples to finally convince him that the technology was bad.

maxxxxx• 9 years ago

Our IT department has been talking about Watson for a while but I never could get anything out of them what it's really supposed to do. From what I can tell it's a very slick marketing term with some mediocre tech behind it.

gaius8• 9 years ago

It sounds like Autonomy. Or Palantir.

dguaraglia• 9 years ago

As far as I understand (and I have only passing knowledge on this, as we've been evaluating machine learning technologies for our startup) IBM Watson is just a set of APIs you can use to build your own solution, and the APIs themselves are not top-of-the-line (Google's are.)

The other day I saw a TV ad for Watson for the first time: the ad showed a bunch of aeronautical engineers who instead of having to take all kinds of measurements of airflow around a hull or who knows what else (the TV was muted) now sat behind a single computer that would do the whole thing for them. It was such bullshit, I couldn't believe my eyes.

Imagine Amazon promoting SES as this amazing tool that'd automatically send marketing emails and create a marketing funnel and drive user acquisition cost down by a factor of 100X, without you having to do anything. That's the level of vaporware they are selling.

chrisco255• 9 years ago

My team and I were able to build a pretty good chatbot with Watson two years ago. We used Google for speech-to-text, however, as we found IBM's to be behind in that area. We had to use a Node.js proxy with a couple libs to parse out some natural language features (like time)...but overall it worked pretty well. At the time, it was the only open system I knew of to write things like this. Today, of course, there's the Alexa API from Amazon and Microsoft's LUIS, which are probably vastly superior for this sort of thing.

iampims• 9 years ago

Their text to speech API is pretty decent and was the cheapest available until Amazon Polly.

ams6110• 9 years ago

What? A brand name which is just a word meaning "IBM Enterprise Products and Services" doesn't really live up to the marketing hype? I can't imagine such a thing.

batmansmk• 9 years ago

You can try by yourself. https://alchemy-language-demo.mybluemix.net/

Imagine analyzing product reviews to determine if it was positive or negative. Type "I like it", and see the inaccurate targeted sentiment (neutral sentiment instead of positive).

guyfawkes303• 9 years ago

I tried this, and while 'targeted sentiment' may be neutral, 'document emotion' has Joy at 0.880435 vs the rest at 0.

If you change your example from 'I like it' to 'I like puppies', the 'targeted sentiment' changes from neutral to 'puppies positive 0.550431'.

While the first example you found does seem odd, the rest seems to work quite well. Maybe it just has problems attaching 'it' to a 'target'.

foldr3• 9 years ago

It has trouble with negation outside the relevant clause. So it is not fooled by e.g. "I don't hate it" vs. "I don't like it", but the first of the following two sentences is rated more negative than the second:

I don't seem to hate it.

I don't seem to like it.

Edit: In fact it can even be fooled just by increasing the linear distance between a positive term and negation. The following is rated +0.6!

I don't in any way shape or form like it.

tensor• 9 years ago

Google has a similar API:

https://cloud.google.com/natural-language/

oceanswave• 9 years ago

Microsoft too: https://azure.microsoft.com/en-us/services/cognitive-service...

te_chris• 9 years ago

Alchemy was an acquisition.

mwambua• 9 years ago

It was an odd acquisition considering IBM Watson's supposed sophistication. Why not just use Watson's components rather than acquiring a company that did roughly the same thing?

hervature• 9 years ago

I think this is actually very accurate as to how humans use the sentence "I like it". If you cook dinner for someone and you ask how it is and they say "I like it" it generally means it's not bad but also not good.

Edit: My suspicion was confirmed when I tried "I like it a lot" had a positive sentiment with 0.711.

foldr3• 9 years ago

"I couldn't live without it" is rated -0.6.

dboreham2• 9 years ago

Bundle up for the second AI Winter...

sjg007• 9 years ago

I think we have a good 10 years of engineering production systems based on current tech.

bitL3• 9 years ago

Third actually. But it's just Watson; Deep Learning and its derivations are here to stay as they are already state-of-art and close to humans.

z3phyr• 9 years ago

They are not close to humans at all. They are good for very specific tasks only.

Edit: Spelling

tannhaeuser• 9 years ago

What's the second AI winter then? I only know about the 1986-1990 era, where "AI" was used in a broad sense covering everything from Prolog, expert systems, LISP, heuristic search algorithms, fuzzy logic to ML, neural networks and their generalizations.

I've still got great shiny magazines from that time, though.

adangert• 9 years ago

I think it will be the third this time around.

dboreham2• 9 years ago

I must have sat one out..

alok-g• 9 years ago

I would not generalize from Watson to the current trends in AI.

I had researched into Watson when it had won Jeopardy, as much as I could publicly find, and concluded at that time itself that it won't stand up to the hype IBM is creating.

dpflan• 9 years ago

I like how IBM does very elaborate marketing ploys to hype their wares: like Deep Blue competing against Kasparov and Watson competing against Jennings to showcase IBM's engineering prowess. But it does sell the idea pretty well I think, but perhaps the idea is too grand/far ahead of the present.

forgot-my-pw• 9 years ago

Deep blue was just a brute force machine. Perhaps Watson isn't much different, just slightly smarter.

sgt10111• 9 years ago

Watson that won Jeopardy was a fantastic advance in information retrieval. Google introduced the knowledge graph a year later, but domain (jeopardy) specific watson was terrific. The sad thing is that they got captured by PR rather than sense; a focus on customer service and my bet is that they could have disrupted Oracle/Seibel out of 75% of accounts.

IBM would be ridin' high!

BTW. This is still on the table for IBM if they are prepared to invest 18mths and focus gung ho on Oracle. In 18mths it won't be as "other players" will have moved, like Sauron, to hit this market.

zone411• 9 years ago

Saying something like this does a big disservice to IBM's real accomplishment. Code a brute force chess engine, add some alpha-beta pruning, transposition table, or even some of your heuristics and let us know how it does. Just because it's not an adversarial RNN or whatever, it doesn't mean it's brute force.

tangue• 9 years ago

Crédit Mutuel (a french bank) has adopted Watson [0] and it's not encouraging : it was supposed to help answering emails, : they had to describe manually the concepts in emails and create topics in which looks a lot like decision-trees (and reminds me of this 1985 ad for Texas Instrument's Lisp AI https://www.scientificamerican.com/media/inline/blog/File/De... scroll to see the ad)

Indeed the whole thing looks like a database with basic AI as a sales argument...

[0 - in french] http://www.silicon.fr/credit-mutuel-non-ia-watson-magique-17...

Spooky234• 9 years ago

Totally. I observed a support chatbot project that didn't quite meet up with the vision.

Even based on the technical marketing claims the consultants were coming in with, it was a product less capable than the IBM Prolog-based expert systems that I built out 15-20 years ago to triage and correlate network events in a large WAN infrastructure.... and that was a product with a whole slew of major implementation and operational problems!

fapjacks• 9 years ago

Hey just wondering: Have you written about your exploits before? If so, where can I read about them? Sounds interesting.

yourapostasy3• 9 years ago

...product less capable than the IBM Prolog-based expert systems that I built out 15-20 years ago to triage and correlate network events in a large WAN infrastructure...

That sounds like IBM Tivoli Enterprise Console (TEC). They wrapped a big Prolog-based DSL that the users wrote their correlation rules in, around the underlying Prolog engine. You could dip into the raw Prolog if you wanted, but it was considered an advanced user technique. The vast majority of "advanced" users used the set of cookbook Prolog functions the DSL was represented as, and even more users only ever touched a GUI that represented a tiny subset of that cookbook. People figured out what parts of Prolog the Prolog engine actually supported mostly through trial and error at first, before IBM finally published more details about it.

This rules-based approach ran into the same challenges of expert systems: beyond a certain threshhold number of rules, reasoning about the rules logic became dramatically more difficult. It didn't help that IBM never supplied support tooling for the underlying Prolog engine to aid in performance profiling, debugging, build support, etc.

The lesson I drew from that experience is if you supply your users with an embedded language in your app, make sure the advanced users get access to the kind of tooling full developers want.

It is interesting to see that the Watson-Jeopardy system uses Prolog [1]. I have to wonder how much of that was re-purposed to the chatbot project you observed. If it was, then its disappointment might be predictable; the range of possible problem spaces in a support setting is going to be larger than the relatively more constrained NLP of the Jeopardy format.

I've yet to see a good NLP system oriented towards technical software support, backed by a support knowledge base, that functions substantially better than current text searching and result ranking technologies, so I'd be very interested to hear about other people's positive results with applying machine learning to this area.

[1] https://www.cs.nmsu.edu/ALP/2011/03/natural-language-process...

shpx• 9 years ago

Chatbots are glorified command line interfaces. With no man pages.

_wmd• 9 years ago

Side note: whatever happened to article-sized advertising from computer periodicals? Just like this one, they actually manage to be informative while remaining sober and honest about their intention. Not to mention it includes enough prose to develop a serious flow and style.. you can't fit that into a banner ad

sgt10111• 9 years ago

Which is odd as the Jeff Dean pattern of "small feedforward network to say if this mail can be answered and then deep rnn to create an answer" seems to actually work. So how come IBM haven't just lifted it?

kaimaoi• 9 years ago

That reminds me of this: https://twitter.com/iamdevloper/status/830070592611172357

pron• 9 years ago

You speak as if AI is something that actually exists. The term AI is always a sales pitch (or the name of a field of research). In the '50s and '60s they called various symbol manipulation programs AI; later it was expert systems; now it's statistical clustering.

moomin2• 9 years ago

Completely off topic, but didn't IBM have a system called Watson in the 1990s that was used by the police? Try as I might I can't find a reference for it anywhere.

rjsw• 9 years ago

Maybe you are thinking of the product from Harlequin [1].

[1] https://en.wikipedia.org/wiki/Xanalys

moomin2• 9 years ago

I think you're right... thanks.

sgt10111• 9 years ago

Scotland Yard had a system called Holmes

https://en.wikipedia.org/wiki/HOLMES_2

Retra• 9 years ago

Watson is the name of the founder of IBM, so there's a pretty good chance his name was attached to various things over the years.

currymj• 9 years ago

it's just a brand name at this point, which they attach to any machine learning they develop or acquire, and they should stop trying to sell it as a distinct technology.

strict9• 9 years ago

Many years ago when I worked for a company that decided our existing ecommerce app was too terrible to fix and would be too much effort to rebuild, we talked to a number of vendors, including IBM. The marketing materials and salespeople made a compelling case, but deeper dives into the app itself and the support engineers behind it convinced even the most enthusiastic internal cheerleaders to look elsewhere.

In recent years as news articles heralding the future of Watson for various industries (including healthcare and supply chain), I predicted a similar path. An amazing product in a very narrow environment designed specifically for marketing and selling purposes, and not very adaptable.

FTA: “And everybody’s very happy to claim to work with Watson,” Perlich said. “So I think right now Watson is monetizing primarily on the brand perception.”

This is painfully obvious, as this has been IBM for a very long time.

iamleppert• 9 years ago

The real crime is in using cancer kids to sell your product. I mean, who even does that? Even if you could cure cancer for kids, I find it incredibly tacky to go around making commercials about how you can cure cancer for kids, which aren't targeted at those who actually are in the position to use the technology, and its used to market to other tangential industries where the real money is. It's just despicable and you can tell right there its smoke and mirrors.

There's a special place in hell for anyone working at IBM or involved in the Watson project who is supporting this thing. It's damaging legitimate deep learning/machine learning industry and generally making a fool out of IBM, AND giving children with cancer false hope....just so IBM can try and stay relevant and make money?

orf• 9 years ago

The real crime is in using cancer kids to sell your product. I mean, who even does that??

Charities?

oblio• 9 years ago

Ah, yes, the noblest of giants, the most charitable of charities, Big Blue.

ryanmarsh• 9 years ago

Can confirm

Source: former president of children's cancer research foundation

In my defense we were actually trying to make a difference in the world not sell software.

davidreiss• 9 years ago

Kids are used by scams/charities to lure in people for all kinds of things. The latest one is the cars for kids scam which used kids to get donations and who knows what they did with it.

http://www.nbcnewyork.com/investigations/Kars-for-Kids-Cars-...

Jeremy1026• 9 years ago

It's amazing how easy it is to set up a scam NPO.

tylersmith• 9 years ago

Allegedly the money is used to help teach Jewish children more about their faith.

https://en.m.wikipedia.org/wiki/Kars4Kids

ceedan2• 9 years ago

Does IBM itself even "use" Watson?

marvel_boy• 9 years ago

Of course not. Watson is just a marketing trick to charge for consultants. I worked there.

ceedan2• 9 years ago

This has always been my assumption. If Watson did anything close to how it's advertised, IBM would have a big competitive advantage by actually using it.

outside1234• 9 years ago

You don't say! This is IBM consulting ware? Who would have guessed!

notfromhere3• 9 years ago

The dirty secret is that IBM Watson is just a brand for their army of data consultants, and their consultants aren't very good. In my experience working for a competitor in this space, IBM Watson was widely agreed to be smoke and mirrors without much going on

ibm_throwaway• 9 years ago

Former IBM Watson data scientist here.

At first I was annoyed by your comment, but then I realized how true it is and sulked instead.

The good consultants are leaving as soon as they hit the end of their first promotion cycle. The duds stick around and assume the upper-level manager roles. The group is in bad shape now and poised to get worse in the coming years.

ethbro7• 9 years ago

As someone who's worked in that space, I'll repost the short version of a comment I made before: it's an economics problem.

Services Profit = Bid - Employee Cost + Upsales

Bids are only so flexible. Employee cost is much easier to minimize. So inevitably, all consultancies trend towards providing the cheapest employees that will satisfy the customer. And if the customer isn't constantly vigilant, the quality can get pretty dire indeed (BAs who just learned to "code" yesterday).

mercer8• 9 years ago

Is it a bit like Deloitte? I keep hearing it's the shittiest of the bunch, but you do it for a while so you can move on and get a better job (or start your own company).

notfromhere3• 9 years ago

Don't feel bad. All the good IBM employees are former IBM employees

Bucephalus3552• 9 years ago

I was shocked to see this comment as it rang so true for me. I worked at Oracle for many years, and I think it's a pretty well run company, but this theme of "the smart people leave, the dumb people stay and promote other dumb people" is very true. Mark Hurd is the current CEO, and I think he's really good (even though all the people on www.thelayoff.com absolutely revile him), but there is a reason he came from outside the company.

speeder• 9 years ago

I actually love the idea of Watson being used for healthcare...

Sadly I think it is being used wrong...

IBM is focusing on using Watson to cure very specific diseases, like certain types of cancer.

I think a far better use for Watson would be to do initial diagnosis, for example my life got massively delayed because I got hypothyroidism as teenager, but only using internet data I could self-diagnose and self-treat (because doctors are still unwilling to help, not trusting data, and before someone come berate me for self-treatment, it is working...) as adult I could finally get my life 'started' (hypothyroidism affect physical and mental development, and slows down metabolism and the brain)

During my quest I met many, many, many people on internet, that had self-diagnosed with something using the internet as a tool. All of us would have been diagnosed properly if Watson was being used on the doctors office, using its data crunching capabilities and symptoms as input to find out what problem we had. (in my case: I have Hashimoto's disease)

vslira• 9 years ago

Why would professionals support tech that makes their labor less scarce? Unfortunately this change won't come from doctors

pwthornton3• 9 years ago

Doctors might fear it, but healthcare administrators would certainly love for a tool to help make diagnosis faster, more accurate and less expensive.

johngarrison3• 9 years ago

Because, in theory, if anything goes wrong during the course of a patient's care the doctor can just say "Watson agreed with me" and the chance of litigation drops dramatically.

Bucephalus3552• 9 years ago

There was a thread a while ago on HN where someone mentioned they design EMR and EHR systems and for a particular module they design they couldn't put in autofilled dates because the nurses union saw it as a slippery slope to them being out of a job. Considering good spectrometers embedded in the cameras of smart phones are only a few years away, perhaps full blood panels coming just from the camera of your iPhone, can't say they are wrong.

gaius8• 9 years ago

Why would professionals support tech that makes their labor less scarce?

That's an odd question. People in tech are always saying "teach everyone to code!!". There's your why...

rspeer• 9 years ago

All of us would have been diagnosed properly if Watson was being used on the doctors office, using its data crunching capabilities and symptoms as input to find out what problem we had.

No, unfortunately, I don't think there's any basis to believe that you could have been diagnosed properly by "Watson". Maybe by better data management in medicine. Not by an IBM brand name.

From the article:

“Why confuse people and make them think it’s going to find something that a physician couldn’t possibly find?... Then you’ve moved into what strikes me as unethical territory when you’re potentially giving hope to people who should never have placed hope in that kind of a system because it’s not a magical box that does that stuff.”

slackingoff20173• 9 years ago

IBM is a dying giant, I've seen it languishing for years. Their massive screw up was a decade ago when they decided shareholder value was more important than having good engineers. They've since gutted their R&D departments and all that's left are duds and underpaid undereducated consultants rented from places like Accenture.

The only good thing to come out of IBM in years is their Hyperscan regex library and unsurprisingly they don't market it at all or build practical applications with it

bsg752• 9 years ago

The same Hyperscan as here? https://github.com/01org/hyperscan

Project homepage suggests this is an Intel project.

thealfreds• 9 years ago

Also the LICENSE

Hyperscan is licensed under the BSD License.
Copyright (c) 2015, Intel Corporation

KingMob• 9 years ago

The Hyperscan website (https://01.org/hyperscan) says it was built by Sensory Networks Inc, and then bought by and released as open source by Intel. No mention of IBM. Are you thinking of something else?

slackingoff20173• 9 years ago

Oops, yeah. They haven't produced anything then :)

WalterBright3• 9 years ago

In the 80's IBM was an unstoppable juggernaut. In the 90's it was Microsoft. Then Apple. Now Google/Amazon.

Conventional wisdom is that, in a free market, a single corporation will grow to rule everything. But that never seems to pan out.

Part of the reason is that the larger an organization gets, the more inefficient and bureaucratic it gets, the less it is able to adapt. See "The Innovator's Dilemma", a book everyone interested in business should read.

ethbro7• 9 years ago

+Baidu/Alibaba

aphextron• 9 years ago

Oddly enough Microsoft is the only company I can see bucking this trend right now. The other big guys youve mentioned have essentially slumped down into their respective markets and drawn up battle lines, whereas MS really seems to be pushing into new territory. I wouldnt be surprised to see them come back into mobile with a Surface phone that gives Apple a run for their money.

notfromhere3• 9 years ago

The main reason is that corporations run like command economies internally, and have to compete(ish) in a market externally. So resource allocation in a corporation is a bureaucratic, political task. This is why big companies notoriously suck at everything after a while, since internal management just starts pillaging business divisions for resources and pet projects

baursak• 9 years ago

With existence of antitrust laws, or government regulations in general, it's not really a free market. Both IBM and Microsoft were hit with antitrust issues. Standard Oil or even AT&T are some examples that come to mind. I bet there are countless other examples where consolidation was either prohibited by regulators, or not even attempted in the first place due to antitrust issues.

E.g.:

https://en.wikipedia.org/wiki/History_of_IBM#1969:_Antitrust...

https://en.wikipedia.org/wiki/NCR_Corporation#Expansion

PhasmaFelis• 9 years ago

*Part of the reason is that the larger an organization gets, the more inefficient and bureaucratic it gets, the less it is able to adapt."

Which is a symptom of publicly traded companies, right? It creates this weird system where finding a niche and serving it reliably year in and year out is considered failure; you're only succeeding if you're constantly growing, which means you inevitably collapse under your own weight.

programmarchy• 9 years ago

Large firm sizes are encouraged by state protections like patent law, regulatory capture, and other monopolistic advantages granted by anti free market practices.

The economic calculation problem, described my Mises in his criticism of centrally planned state economies, starts rearing its head in large firms as well [1], leading to the inefficiencies you mentioned.

[1] https://fee.org/articles/economic-calculation-in-the-corpora...

forapurpose• 9 years ago

Conventional wisdom is that, in a free market, a single corporation will grow to rule everything. But that never seems to pan out.

The extreme scenario doesn't pan out - we don't end up with one eternal corporation that owns all capital. But certainly it pans out in a more realistic scale of size and time; there are and have been plenty of monopolies in many industries, and the overwhelming power and influence of large corporations, and the damage that power and influence does, is well documented.

duncan_bayne• 9 years ago

That's only conventional wisdom amongst people opposed to laissez faire capitalism. It is not an opinion held, for example, by the Austrian School.

adventured• 9 years ago

Their massive screw up was a decade ago when they decided shareholder value was more important than having good engineers.

IBM's management believed shareholder value was important, in about the same way Bernie Madoff believed his investor's well-being was important. Which is to say, not at all in actuality.

IBM's management was negligent in regards to shareholder value in almost every way possible.

Believing shareholder value was so important, would require simultaneously believing having good engineers is critical.

The management decided to fleece the company for their own near-term benefit, to the detriment of shareholder value (which is now being represented in the stock price accordingly, and it'll get much worse, soon).

jameslk• 9 years ago

Dying? How so? Lots of companies have their legacy IT infrastructure built on top of IBM's mainframes and products. Who's innovating fast enough to provide a compelling enough reason to replace all that infrastructure? Perhaps IBM is just stable and uninteresting, like most blue chip companies.

reednj• 9 years ago

It's not stable though is it? Their revenue has gone down 22 quarters in a row

gchokov2• 9 years ago

Yep, Dying. No new companies built any infrastructure or anything on IBM mainframes.

icc972• 9 years ago

IBM have been what seems like this bureaucratic monstrosity ever since the started. But that was back in the 60s.

I appreciate their perseverance if nothing else. If you look at the fate of companies like Nokia, IBM has some how managed to avoid this what has been meteoric shifts in their business.

Plus competition is always good.

Watson itself clearly was a great product and a big step in the development of AI. Just because the marketing department is making a mess of things shouldn't take away from the work that was done on Watson in the first place.

rsj_hn2• 9 years ago

It's not just dying, it's shedding customers. The only reason the stock price isn't in free-fall is because it's cutting costs rapidly via outsourcing and also buying back shares.

  IBM Real(1) annual Revenue (in billions of 2016 dollars):

  1986: 111
  1996: 116
  2006: 109
  2016: 80

  (1) adjusted by BLS CPI Calculator

potatoman2• 9 years ago

Did they really need to stick an insult at Trump in there?

optimuspaul• 9 years ago

yes

colejohnson66• 9 years ago

As much as I hate bringing politics into places it wasn't, this wasn't from the author; It was from a quote

"IBM Watson is the Donald Trump of the AI industry—outlandish claims
that aren’t backed by credible data," said Oren Etzioni, CEO of the
Allen Institute for AI and former computer science professor.
"Everyone—journalists included—know[s] that the emperor has no clothes,
but most are reluctant to say so."

etiam• 9 years ago

It's tempting to start whispering winter is coming, but I think one may reasonably hope that the current fashions at large have enough nuance to differentiate between this particular marketing gimmick and the broader developments in ML.

Personally I'd be happy to see the paragraphs/minutes at the beginning of far too many interviews about "intelligent" machines exchanged, from straightening out the misconception that Watson is an example of this new hot "Deep Learning" thing and one of the pinnacles of achievement in the field, for some type of more valuable type of commentary from leading researchers.

PaulHoule• 9 years ago

This was my opinion when they started running these ads. My opinion has actually softened a little.

Some of the cognitive services they are offering today are not half bad; also I can say their salespeople are doing a gangbusters job in places.

crsv• 9 years ago

Replace IBM's Watson with anything branded with "AI" right now and themes in the article still hold up.

bsg752• 9 years ago

It is likely that AI has not matured to the level its being marketed at currently?

And that the over hyping of it today will negatively impact its adoption when ready for general use?

macintux• 9 years ago

Seems like AI has been over-hyped for decades, just rarely with someone as prominent as IBM leading the charge.

scottlocklin4• 9 years ago

It happened in the 80s. https://en.wikipedia.org/wiki/AI_winter

It will happen again. The use of the phrase "artificial intelligence" itself ought to be forbidden until such a thing actually exists in the corporeal world.

riku_iki• 9 years ago

Any first firings for choosing IBM?..

et2o2• 9 years ago

I saw a very humorous twitter exchange between a bioinformatician and IBM Watson's twitter account. The scientist asked them to provide any peer-reviewed ML publications and the best they could do was an abstract at a regional conference no-one has heard of. And it was a terrible abstract.

It's completely marketing. IBM still has a good name among people who don't know much about technology. They're trading on this and the current saturation of 'machine learning' in the popular press.

sgt10111• 9 years ago

That's mad as IBM have loads of peer reviewed ML publications.

3 minutes of searching :

http://researcher.watson.ibm.com/researcher/view_group_pubs....

c'mon

et2o2• 9 years ago

I should have added: healthcare. No shit IBM research has publications, they have their own journal. Also, 15 publications in 3 years is pathetic. c'mon

blueyes• 9 years ago

IBM has almost zero credibility in deep learning and AI. They haven't hired anyone of note. They haven't produced any novel or influential research in the field in years. And yet they air these cheesy Dylan ads and the rubes fall for it. Watson is a Theranos-scale fraud, and it's finally coming out.

pas• 9 years ago

Was the Jeopardy think just a ruse? Was it something easy (something any ML/DL team could have thrown together in a year?)

martingoodson• 9 years ago

No it was a hard engineering challenge. But it solved a very specific problem that has no relevance to most business use-cases.

Jach• 9 years ago

http://jeffjonas.typepad.com/about.html

However he's no longer at IBM...

tCfD• 9 years ago

Obvious fix is for IBM to put Watson on a blockchain /s

samfriedman• 9 years ago

You jest, but they're not above "throw a blockchain on it" levels of buzzword-chasing: https://www.ibm.com/us-en/marketplace/z14

diego_moita• 9 years ago

"In the data-science community the sense is that whatever Watson can do, you can probably get as freeware somewhere, or possibly build yourself with your own knowledge"

Any suggestions about the freeware?

sgt10111• 9 years ago

MLlib

Tensorflow

Lime

There you go.

nl4• 9 years ago

MLib

The Watson of machine learning libraries.

As someone who has used MLib and raised a number of bugs and feature requests: Use Spark for feature engineering and then Scikit for ML.

simonh2• 9 years ago

This just goes to show just how tragically far away we are from even beginning to build the rudiments of a strong general purpose AI. For all the fantastic achievements of systems like Watson and Alphago, and they are amazing achievements, they are radically optimised special purpose systems fine tuned to solving one extremely specific and narrow problem, and that problem only.

Watson is a case study in this, but I know Google has big plans for applying the tech behind Alphago in medicine. I wish them every success, but I'm concerned they will hit similar specialisation issues.

FeepingCreature• 9 years ago

I would argue this only shows how tragically far away IBM is.

sgt10111• 9 years ago

Medicine - the biggest prize, the biggest trap. Every complication, huge, mind blowing rewards. Maybe, if Google were to go 100% they could break the beast... but it's a 100bn bet.

simonh2• 9 years ago

Untill you actually have a viable, testable plan to implement it almost allege money you throw at it is pretty much certain to be wasted.

The Manhattan project is often given as an example, but that project was based on solid theoretical models and repeatable experiments. They knew exactly what they were doing and how they were going to do it.

The problem in AI at the moment is mainly developing talent and expertise, not capital funding. I'm all for advanced research into technologies. However we need to advance that research on as many different fronts as possible in a flexible way simply because we don't know how we are going to solve the problem of general AI. Guessing on an approach and spending billions on it would be a complete waste. We're simply not at a point where that would do any good.

ChuckMcM• 9 years ago

When I worked at IBM I expressed concern that the television commercials depicting a HAL9000 level interactive dialog system were dangerously overselling what Watson could do.

The challenge, as I saw it, was that no matter how good the tools and products that were used to help companies with data analysis to improve their operations were, when they realize they can't talk to a cube and joke with it about misusing colloquial phrases their disappointment overshadows all the 'good' stuff it was doing for them.

No relationship works well if it starts with a lie and as this article shows, people do take those ads at face value and assume there really is a talking AI inside of IBM. Then they are hugely disappointed when they find out it doesn't exist.

ethbro7• 9 years ago

Over-promising has killed AI progress before (expert systems). So history's just repeating itself.

People want to believe. They'll pay for their belief. Then when projects start to fail (~5 years after commercialization) they form the opinion that it was never going to work and they'll never do it again.

... then we wait another decade, repeat.

olewhalehunter• 9 years ago

There's this trend of Expert Systems being commented on as a failure and starting some sort of AI winter, yet to my knowledge they were among the first commercially viable digital animation tools, the first high speed trading platforms, and are still being used successfully in military planning. Where's the evidence that they were a failure or that they were oversold?

throwaway7645• 9 years ago

Just like the "Lisp AI Winter" when the defense department stopped spending big on AI in the 80's.

protomyth3• 9 years ago

Wasn’t that the same type of conflict that doomed the Newton? The engineers didn’t want marketing to talk about the handwriting recognition (nice extra, look what other things we do) but marketing ignored them. The Newton never recovered.

Angostura3• 9 years ago

The engineers didn't want marketing to talk about the primary method of data entry and one of the most superficially innovative thing about it?

I can only imagine the look on marketing's face when they heard that one.

erdle• 9 years ago

The funny thing about that ... is that IBM's long time ad agency is Ogilvy & Mather.

It was founded by David Ogilvy.

One of his big things was "nothing kills a bad product faster than great advertising"... he would be so ashamed of these ads.

thinnerlizzy• 9 years ago

I worked there for a time and can tell you that many of the creatives I worked with assumed that if Watson won Jeopardy, then it could do literally anything (indeed talking to you like a human), so a lot of the creative concepts were pretty far fetched.

pryelluw• 9 years ago

I do think it was positioned wrong. Which is amazing given how clear their actual positioning was. Big data and ML are big business opportunities. IBM's marketing just jumped the shark and could lose a lot of market share due to it.

raverbashing• 9 years ago

Not a surprising position coming from IBM

d--b• 9 years ago

watson's mistake is to have gone the chat bot route. promising a natural language input for all underlying problems simply discredits everything else...

scottlocklin4• 9 years ago

Yeah, well, "duh." What boggles my mind is people will read this, nod sadly, and continue not to notice that a whole bunch of what they think they know about machine learning, autonomous vehicles and so on is also marketing department hype.

mattnewton2• 9 years ago

I don't know if that's fair. To take self driving cars as an example, I've seen cars driving themselves around mountain View and Waymo isn't promising much more than that. Watson had ads with an intelligent voice but they were selling a bunch of programmers to make expert systems like it was the 80's but in java this time. I think by comparison those other fields are promising less and delivering more. Even if there is still a gap between what people think will happen in the next 10 years and what we can likely deliver in the next 10, it's nowhere like what is described in the article.

scottlocklin4• 9 years ago

C'mon man! Google allows one to check on their promises and their progress. For example, from 2012:

http://www.computerworld.com/article/2491635/vertical-it/sel...

"Such "autonomous vehicles" will be a reality for "ordinary people" in less than five years, Google co-founder Sergey Brin said Tuesday."

How's that working out for you Sergey?

Self driving tech hasn't moved appreciably since the 90s. Computers are better. More people are trying it. Reality is, it's just collision avoidance with GPS. They rely on maps; what happens when the maps change? What happens when a kid runs out in front of one of these things?

Probooks• 9 years ago

Problem is deeper (and simpler). IBM does not look for clients, but rather victims. We clients end up being caught in an internal upsales fight. Nobody cares which is the best solution IBM as a whole can offer to you (their own people do not even know all their available tools!), but rather how much suboptimal stuff each salesman can load onto you. I'm on my way out of IBM...

dislikes_IBM• 9 years ago

IBM has a toxic culture. They are the vendor lock-in Gods. Every company I've ever worked for has cringed at the mention of IBM, never suggested them as a new solution, and always regretted whatever if anything they locked themselves into.

They are the only company that charges you to sample their API's. They are the absolute worst, an infection that needs to be cured.

johngarrison3• 9 years ago

Sometimes I wish I was wealthy enough to buy out horrible companies, just so I could have the power to dismantle them and end the suffering.

Method-X• 9 years ago

You wouldn't be wealthy for very long ;)

smugworth2• 9 years ago

I'm working on a team for a platform built on open-source software which actively replaced a previous IBM solution, and there are more services from other vendors being phased out in favor of internally developed tooling.

Seems like a good long-term investment, in an era where people are constantly criticizing CEOs for short-term cash-grabs. Why shackle yourself to a vendor if you plan to be around for awhile.

ghostly_s• 9 years ago

I overheard a good-'ol-boy businessman at a hotel bar a few months back. He bore an eerie likeness to Bosworth from Halt and Catch Fire, and was telling a younger gentleman about a project he worked on. "...so Watson comes in and they Algorithm the whole thing..."

I'm pretty sure he thought Watson was a person.

ExactoKnight• 9 years ago

Watson's Natural Language Classifier, in particular its categorization API, is actually pretty impressive...

jjm• 9 years ago

They had so much time to contribute but instead chose marketing and pushed into areas where they didn't really have a handle on yet. As in management didn't understand.

I mean all the datasets, dozens of libraries, stunning NN demos and training sets, TPUs (multiple versions at that!) all could've come out of the company.

Think if keras and tensor flow were from IBM. Or all those cars now running Nvidia Jetson, or mega datacenters running NV100s or Google TPUs.

Shoot they even had a chance to enhance PowerPC ICs for NNs.

Alas but nope.

filereaper2• 9 years ago

I'm quite late to this thread, but I worked on Watson very briefly (not on the core development, but overall system performance improvements).

I think there's a major misunderstanding of Watson which isn't helped by IBM's Marketing efforts. IBM Marketing has been slapping the "Cognitive" label on everything and is creating unrealistic expectations.

The Jeopardy playing Watson (DeepQA pipeline) was a landmark success at Information Retrieval, its architecture is built largely on Apache UIMA and Lucene with proprietary code for scaling out (performance) and filtering & ranking. I'm not an expert on IR so I won't comment further. This is very different from Neural Nets that are all the rage in ML today.

I'd like to point the following links from David Ferrucci [1] the original architect of Watson and this technical publication at aaai.org [2].

The DeepQA pipeline wasn't fluff, the intention was to take this question-answer pipeline and apply it to other verticals such as Law and Medicine, essentially replace the Jeopardy playing Watson's corpus of Wikipedia, Britannica etc... with Legal and Medical equivalents.

Given its runaway PR success, the Watson brand was applied to many other areas which haven't been successful but I'd like to point out what the original product was here.

[1] https://www.singularityweblog.com/david-ferrucci-on-singular... [2] https://www.aaai.org/Magazine/Watson/watson.php

nl4• 9 years ago

This is true. I've read all the Watson Special Edition papers numerous times[1].

The original Watson work was very good, and adding neural-network based models as alternative question parsers and answer rankers is a clear way forward.

It's a pity IBM decided to use the Watson name for everthing they do in ML now.

[1] http://brenocon.com/watson_special_issue/

denzil_correa• 9 years ago

The Jeopardy playing Watson (DeepQA pipeline) was a landmark success at Information Retrieval, its architecture is built largely on Apache UIMA and Lucene with proprietary code for scaling out (performance) and filtering & ranking. I'm not an expert on IR so I won't comment further. This is very different from Neural Nets that are all the rage in ML today.

The question I would like to ask is that if the architecture is built on UIMA and Lucence, what exactly did "Watson" do?

filereaper2• 9 years ago

Apache UIMA came out of the efforts from Ferrucci and others from TJ Watson Research Lab.

"Watson's" secret sauce was the filtering and ranking, you get multiple results from the models, which ones do you pick?

There was significant engineering effort put into reducing the latency of getting answers out of the pipeline as well, you had to optimize given the constraints of latency, accuracy and confidence from the pipeline.

dmreedy• 9 years ago

If you can find them, read the papers on Question Analysis and Answer Coercion. I am still impressed by how powerful these systems were, and how effective. UIMA is a framework for tying algorithms together, and experimenting quickly with different combinations.

(locked-down IEEE versions: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=617771...)

nl4• 9 years ago

There's a lot more there than just search. See http://brenocon.com/watson_special_issue/

ezy• 9 years ago

"Watson" was a Jeopardy playing bot.

The main trick behind Watson is to take the various systems (parsers, search, et al.) and hacks (constraints imposed by the rules of jeopardy) needed by a jeopardy playing bot and put them all together.

So, in some sense, you could say UIMA is what Watson did -- because it allowed a lot of flexibility for researchers to combine their efforts. Ranking and filtering becomes of ultimate importance in a system like that because at some point you have to make a decision. However, it is terribly reliant on the other modules at least getting somewhere in the ballpark -- and the ranking is also not, by itself, anything impressive.

So, it's an interesting case of how far you can get by just setting a single goal and slamming everything together -- but as it turns out, for every new domain you wish to apply something like that to, that magic ballpark is hard to reach without a significant amount of engineering & research effort to come up with new systems/hacks combined with a lot of relevant data. In other words, just like any other adhoc AIish system with a particular goal. Change the goal, change the system.

So, of course Watson was oversold, it was a PR and Sales effort from the beginning. Sort of like AlphaGo or DeepBlue -- you might be able to find one or two interesting ideas in the bowels of such a system -- but the system itself is not a generic one.

avereveard• 9 years ago

there's a major misunderstanding of Watson which isn't helped by IBM's Marketing efforts

you got cause effect wrong, since it's a glorified search engine, the marketing team did all they could do pass this off as intelligence and to muddle the water around what Watson could and couldn't do. when sales target still were falling short, they started dumping everything under the Watson budget to prop up the dept.

spydum• 9 years ago

I have seen this pattern repeated in many large companies. One big product is successful and builds a nice PR bubble. Executives pile on anything they had in the pipeline with a re-brand, diluting the new brand.

I wonder what the solution is? Stronger brand owners within the company who can defend and be the quality control of what gets associated? Or is it hopeless?

aweiland• 9 years ago

MS did this with .NET. They even tried to brand a version of Windows with it for a while. I heard at one point no on in MS could really tell you want .NET was anymore.

cirgue• 9 years ago

IBM has definitely created some cool stuff at various times. The problem, though, is that IBM misrepresents themselves in publicly-facing marketing and with paying clients during implementation.

laichzeit0• 9 years ago

I had problems with Watson to the effect that not even the documentation matches reality. There are some fairly basic things missing from their NERC offering. I can tell you that the functionality that is missing is so basic (e.g. negation) that without a doubt, no one in IBM has ever used this offering in practise beyond a toy example.

The idea that IBM Watson is some uniform AI in a box with a bunch of REST API's to "expose" its intelligence seems to be the sales pitch. It's not. It's just a bunch of acquired products (you can see this when e.g. Watson Knowledge Studio breaks and you see the Python scripts that glues everything together in the backend) that are poorly integrated, probably because the left hand has no idea what the right hand is doing.

Caveat emptor!

https://stackoverflow.com/questions/44796501/ibm-watson-know... https://stackoverflow.com/questions/44800879/ibm-watson-know...

chisleu• 9 years ago

I'm late but have something to add.

Until last week I was on a 6 month contract as a senior DevOps engineer for IBM/Watson. I was responsible for one of the huge real-time data ingestion pipelines that Watson receives. I left to work elsewhere in spite of being offered an excellent position. (If you guys are reading this, hi.)

I went to IBM not expecting much more than working as a cog in a lumbering giant.

Watson is the fastest growing part of IBM. If IBM has all of those eggs in one basket, it is the Watson basket. There were lots of jokes about cognitive in the office pool.

That said, it was by far one of the best managed companies I've seen. They have some fantastic data engineers and scientists. They are backing most of the open source projects related to AI and next generation tech. Spark, VoltDB...

The ads might seem sensational, but the concept of a black box that orders preemptive maintenance for an elevator isn't far fetched...

More over, Watson had so many current customers because it is valuable. The technical advisors that but products don't put faith in ads any more than we do.

megamindbrian• 9 years ago

Everything to do with IBM is too expensive for the average user.

JimsWonderWorld2• 9 years ago

The IBM Watson win on jeopardy was completely made up. IBM does not have an AI that can answer natural language questions now and they also did not have that 7 years ago. IBM'ers are delusional.

A rising sentiment that IBM’s Watson can’t deliver on its promises

Related Stories