Bing smacks Google in new usability test

http://thenextweb.com/microsoft/2011/02/11/bing-smacks-google-in-new-usability-test/

by zuzzo • 15 years ago

71 60 15 years ago

yakto• 15 years ago

I call sampling error. Bing users are easily pleased with results, take anything they're shown. Google users look harder, and often refine a search.

jeroen• 15 years ago

"‘success rate’ .. a measurement of how often a user executes a search, and then goes to a webpage from the set of links"

One simple example where this fails: http://www.google.com/search?hl=en&source=hp&q=circu...

The answer is shown three times in the results. There is no need to click on anything. Did the testers compare these results for Google and Bing?

markkat2• 15 years ago

Bing also shows the answer, however: http://www.bing.com/search?q=circumference+of+the+earth&...

Perhaps Google does this better, but I would be surprised if it accounted for the large difference.

snprbob86• 15 years ago

This is one of the rare queries where I'd try Wolfram Alpha first:

http://www.wolframalpha.com/input/?i=circumference+of+the+ea...

xentronium• 15 years ago

https://img.skitch.com/20110211-r3jmccgi1ejs24p1hbdnu69fwm.j...

vs.

https://img.skitch.com/20110211-cfdpqbca86fe1qgwdpbutdu153.j...

Google's are better, however, they both don't show the value for me.

tuxychandru• 15 years ago

I see it in the second result from Google!

Matt_Cutts2• 15 years ago

It's even worse than that. As I understand it, the definition is not "a measurement of how often a user executes a search, and then goes to a webpage from the set of links" but rather "a measurement of how often a user executes a search, and then goes to a different website." That is, I believe Hitwise has no way of knowing whether the user clicks on a link from the search results vs. just navigating to a different page.

thesash• 15 years ago

The explanation just might be the fact that google submits new searches as you type

electromagnetic• 15 years ago

From my experience back in the pre-google days was that I'd have to open up five or more links to ever see anything worthwhile and this was back in the day when page titles had zero relevance too.

Plus, I haven't seen the google front page to search in months, so that's a definite +10 for usability in my books. If I was smarter (or more pro-procrastination) I'd have a link to google news on my chrome splash screen.

georgecmu• 15 years ago

A study like this would only yield meaningful results if the user selection for all three search engines were the same and randomly selected from the general internet-enabled population.

In this case, one interpretation of these results is that google users look for something very specific and thus are less likely to be satisfied with the first result that pops up, while bing users are more casual and search for generally consumable content.

brudgers2• 15 years ago

>"bing users are more casual and search for generally consumable content."

That would seem to be bad for Google. Bing gets to sell ads alongside casual searches such as "cheap table lamps" and Google has to try to monetize "'Company A' 143rd Ohio Brice's Crossroads."

kunjaan• 15 years ago

"If you are like most people in tech, the last time you used Bing was never"

Woah. I am a SE and I use Bing's image and travel search all the time!

megablast• 15 years ago

Are you saying you are most people in tech? He did not say everybody.

markkat2• 15 years ago

Here's an interesting comparison:

Google: http://www.google.com/search?hl=en&safe=active&q=Fan...

Bing: http://www.bing.com/search?q=Fantasia&go=&form=QBRE&...

Yahoo: http://search.yahoo.com/search;_ylt=Agt64BoFz7.2ArgJZ_Io2BSb...

What stood out to me is that images and videos are not being served up front and center on Google.

No doubt more and more people are searching for video. Google really looks old-fashioned in this match-up.

xyzzyb• 15 years ago

My preference:

DuckDuckGo: http://duckduckgo.com/?q=fantasia

I prefer uncluttered and informative results at the top instead of news headlines.

Legion• 15 years ago

I do too, but I find for non-trivial searches, I often have to make use of !g.

More than once I've spent time doing multiple searches for the answer of a tech question, re-wording the search to try and find a hit, only to have an a-ha moment and say, "oh! hey! '(original query) !g'" and bam, first page, The Answer.

Cyndre• 15 years ago

Its amazing how close they all are. One of the key things thats different is the bar at the top - but all same layout, format, color......

mcn• 15 years ago

My strongly preferred solution to this problem is to search for the desired media by using a keyword, like "!images cat".

I'm not aware of a way to do this from within google, which creates an odd situation: setting my default search to duckduckgo increases google's usability.

eagleal• 15 years ago

But people are forgetting the hover text that Bing returns (see 1st result).For the search "circumference of the earth" http://oi55.tinypic.com/2jfx8ur.jpg

Compare that with Google's results https://img.skitch.com/20110211-r3jmccgi1ejs24p1hbdnu69fwm.j... (courtesy of xentronium). For this query Bing's results are far better that Google's (they actually specify circumference at equator and poles, alerting you that it's not the same; Google doesn't).

Also if you add (in meters|in miles) to the query, you get similar results to Google's.

brockf• 15 years ago

Potential confounds:

* Google submits many more searches with it's instant AJAX search-as-you-type

* Google's speed allows users to hone in on better search results by tweaking terms, whereas Bing/Yahoo users don't waste their time with a 2nd search

* Google displays the answer right on the results page with superior semantic analysis, nullifying the use of a clickthrough

* Google's tools (such as "define:", math like "4+1.078", and conversions like "45cm in inches") are well-known and people look to Google itself for answers, not secondary sites in results

* If Google's users are more "tech savvy", they may use search qualitatively differently than Bing users (related to the above explanations)

* Google users click on Google Ads more than Bing users click on Bing ads - are ad clicks tracked?

Of course, these could all be wrong. But worth looking into or acknowledging in the research if they already had.

akanet• 15 years ago

Another confound I would note is that Bing is highly specialized in certain queries, and that users may select Bing for the queries they know will result in specialized results, and use Google for everything else.

blauwbilgorgel• 15 years ago

If Google corrects a spelling mistake for a user: did you mean x? than that is counted as a bad user experience. If a user gives up on Bing and goes to Google to refine her search, than that is counted a good user experience for Bing.

corin_• 15 years ago

Another one based on the logic that Google's users are more "tech savvy" (which I believe they, on average, are):

* A larger percentage of Bing users are doing "time-saving" searches, such as "hotmail" and "facebook", where they know the first result is what they want even before they search.

And extending that logic - a tech savvy audience will be more likely to have more complex results. It's not hard to give searchers want they want when they're looking for "superbowl results" or "justin bieber".

gaius• 15 years ago

which I believe they, on average, are

Self-selection bias?

Here's what I observe among my non-tech friends: they have Google as their homepage. If they want Facebook, they type "Facebook" into the search box - even tho' typing it into the address bar would do the same thing. If they come across a PC that's not on Google, then they go to Google using the address bar and proceed as before.

moultano2• 15 years ago

Matt's response to this: http://www.google.com/buzz/109412257237874861202/A7d5joZ3tJJ...

nostrademons2• 15 years ago

There's another possible source of selection bias: users may be going to Google for queries that they know are a hard, because they assume that Bing won't have the answer. Of course such queries will fail more often, because they're hard.

Bing's marketing has centered around the head of the query distribution - common tasks where definite answers exist, like how to book a flight. It makes sense that such tasks would succeed much more often than long-tail ones like looking up an obscure error message.

barista3• 15 years ago

Well if they are "going" to Google for hard queries then wouldn't they try the queries on Bing first?

Or are you saying that people spend time deciding a query is simple or hard and then choose appropriate search engine for that?

notahacker• 15 years ago

A third possible source would be bias in user samples:

If Bing users are on average less sophisticated (plausible given the assumption that less sophisticated users tend to use IE with the default search engine) they are more likely to click through irrespective of search results' relevance or the potential to improve a query.

What does the "success rate" of AOL's Google-powered search look like?

natmaster• 15 years ago

Does someone have a link to the information on how Experian's data was gathered?

It looks to me like 'success' is defined as actually clicking a result - not the ridiculous things like switching to google - but their explanation on the report page is fairly nebulous.

Matt_Cutts2• 15 years ago

Suppose the user was on search engine A and then goes to site B. I believe Hitwise has no way of knowing whether the user clicked on a link in the search results to go to site B vs. just going to site B on their own.

As far as how the data was gathered, Hitwise gets clickstream data from ISPs: http://www.hitwise.com/us/about-us/how-we-do-it

barista3• 15 years ago

Here's the gist of Matt's reply: 1. Hitwise's definition of a successful search doesn't actually involve clicking on result.

2. MSN has pages that generate bing search while masquerading as regular pages.

brudgers2• 15 years ago

Mr. Cutts provides an example of Kim Kardashian videos on MSN linking to Bing as an explanation for the difference in success ratings between Bing and Google.

However if that were a significant factor, the similarity between the ratings for Bing and Yahoo would be somewhat unexpected without a demonstration that similar searches are directed to Yahoo. It would also seem that there is probably a significant enough behavioral difference between the user bases of Bing and Yahoo Search for Matt's "abandoned search theory" to be consistent with the similarity in success rates between the sites.

A more plausible explanation (but "creepier" in Googlespeak) for Google's results is that Google's sometimes practice of not linking directly to sites but rather to an internal redirecting page [as described by jacques mattheij here: http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+...] lowers Google's success rate. It would also explain why Google questions the value of the metric so vigorously and why their criticism is speculative and anecdotal rather than more scientifically argued.

http://jacquesmattheij.com/Microsofts+Bing+versus+Google%2C+...

bkhl• 15 years ago

I clearly see where he is coming from and I also believe that the metric Hitwise used is just BS...BUT! the more Matt becomes sensitive about Bing, the more pageviews Bing would get. I bet Bing got quite unexpected PR recently due to Matt's accusation (I don't want to talk about the issue since my opinion is mine). I'm curious to see metrics for Feb. ;)

happybuy2• 15 years ago

Very interesting that the first response most HN commenters have when confronted with data that Bing provides better search results than Google is to doubt the data.

This despite that fact that (I'm guessing) most commenters have not actually extensively used Bing day in and day out. Shows the huge barrier to entry that any new search engine has: even if they provide measurably better search results even tech savvy users have an almost unshakeable belief that Google results will always or somehow must be better.

There seems to be some infallible belief that Google provides better search results, always will and that there is some 'magic' Google has for search that no other company could possibly reproduce or exceed.

There is no logical reason that this should be the case and in many ways its more logical to suggest that Google is so wed to its current techniques and revenue models that it has no incentive to disrupt itself. The recent furore over spam (and Adsense which funds its existence) perhaps is the first evidence that this is actually the case. Google made some noise about rectifying the spam issue but really they didn't make any hard decisions in this space.

A personal anecdote - I switched all my browsers and iPhone to default to Bing about a month ago - the experience has been pleasant, the results excellent and haven't found the need or desire to use Google search since.

chalst• 15 years ago

Very interesting that the first response most HN commenters have when confronted with data that Bing provides better search results than Google is to doubt the data.

The very first reaction I have to most studies is to question first the methodology, then the data. Remember that 88.2% of statistics are made up on the spot.

mst• 15 years ago

To add to the anecdata - I switched everything across to duckduckgo.com about the same time ago and am incredibly happy. The fact that duckduckgo sends me to google when it runs out of ideas helps, but I'm finding myself needing that less and less as I go.

At this point I use google occasionally simply because I finger-macro Ctrl-T,Alt-D,google.co.uk,Enter before actually thinking "if I hit Ctrl-K I'd have duckduckgo search already".

aChrisSmith• 15 years ago

Does my 'search' for "skitsophrenia" as a lazy-man's way of spellcheck actually count as a query? IMHO this metric is flawed.

gojomo• 15 years ago

My greatest search success is when the information I need is in the displayed snippets. Others clearly agree: see DuckDuckGo's 'zero click info' and Google's 'One Box' results.

Next best is when the snippets make it clear I need to refine my search, rather than wasting my time wading through the result sites. Again, no clicks from the results, but a happier searcher.

So the Hitwise 'success rate' metric seems totally bogus to me – inversely correlated to the quality of snippets/OneBox.

I suspect Google's snippets are still the best in the business, though an objective analysis of that would be interesting news.

moultano2• 15 years ago

When powerset launched they had very impressive snippets. Now that they've been bought by microsoft, I wonder how much of the technology has been integrated.

jjcm2• 15 years ago

I highly doubt their tactics here. They're grabbing aggregate ISP data for users (http://www.hitwise.com/us/about-us/how-we-do-it) and running scripts to see if the search is followed by an address that appears on the search page. Also, it seems like they don't account for google's instant search, but it's hard to tell as their first "success" statistic was posted in October (Instant Search was released on September 8th). They do however, publish upstream traffic percentages for sites in various categories (http://www.hitwise.com/us/press-center/press-releases/google...) prior to September. Note that in this case, the ratio between the upstream click throughs (75% for google, 15% for yahoo, and 10% for bing for health related click throughs) and the ratios of the aggregate search traffic (71.4% for google, 15% for yahoo, and 9.4% for bing during that month) allow us to extrapolate the "success" rates during this period, and it seems to suggest that everyone is in line for search results (with google being slightly ahead, at least for health. I didn't run the numbers for the other categories as we dont know the search volume for each of those, and as such you couldn't find an average statistic that was relevant). Just my two cents, but it seems like the data is highly affected by google's instant search.

kqueue• 15 years ago

The number of users that use Bing is much lower than the number of users that use Google.

This comparison is similar to saying that book X that has a total of 4 ratings, 5 star each is better than book Y that has 100 ratings, 4 star each.

jjcm2• 15 years ago

Given the volume of both though (they grabbed these stats from 10 million searches) the statistical relevance of each should be pretty definite, even if you're using a bayesian average.

baddox• 15 years ago

I initially assumed that everyone around here would have tried Bing, then I realized that may not be true. Have you guys given it a try? When Bing came out I switched my browsers to Bing for a couple of weeks and noticed essentially no difference in usability or results quality. Alas, after reinstalling operating systems and browsers, Google gradually defaulted back to the browser I use, and it's just not worth the effort to change again. I really don't prefer one over the other.

arethuza• 15 years ago

I don't use Bing for search (I'm tempted to start using DDG) but I do use their maps quite often (though not exclusively).

I really don't have any brand loyalty to any of the search/map/whatever providers - if something seems to be consistently better I'll switch (as I did from AltaVista to Google).

bayshorecove• 15 years ago

Google sucks in results recently.

Uchikoma• 15 years ago

50% of my Google use, as a non native English speaker, is for spell checking, even for tricky native German stuff, USD to EUR conversion etc and I've also never been using a calculator beside Google for some years now. Google just does that fine.

rimantas• 15 years ago

Must have been wrong shade of blue.

ashishb4u• 15 years ago

i think they opened google.com because bing was not good enough to come back to!

johnyzee• 15 years ago

I am extremely disappointed with the new-fangledness in the newer Google interface changes.

The 'instant' search is just a jarring user experience in the way it does massive unpredictable updates to the screen at every keypress. Even worse is the new image search results screen - mouseover on any image makes the search field lose focus, it even does this when fetching the first page of results, so you have to click the search field again every time you do a search.

Even the suggestion search field they introduced a few years ago was a dubious value proposition - I personally got better value from the standard text field's 'previous entries' than from the suggestions.

Havoc• 15 years ago

>It implies that the user found what they wanted

No it doesn't. It will penalize a search engine where the user base like tweaking the search & unduly boost the score of one with sheep for users that click on the first thing they see.

kurige• 15 years ago

I also find it interesting that a few defunct microsoft webpages (such as http://popfly.com) redirect you directly to bing. Naturally, you'll follow the wikipedia or equivalent link to figure out what's going on.

I doubt this would generate a large amount of click-through data, but it makes you wonder what other tricks they have up their sleeves.

suitcase• 15 years ago

Wow, Google partisans need to calm down a bit.

The reduction in Big G's search engine monopoly is good for the market and good for tech in general.

mih• 15 years ago

Noticed how Yahoo! has a higher success rate than Google? That alone is proof that these numbers do not correlate to the real world experience.

Bing smacks Google in new usability test

Related Stories