Bing sets the record straight on recent accusations

http://www.bing.com/community/site_blogs/b/search/archive/2011/02/02/setting-the-record-straight.aspx

by thankuz • 15 years ago

169 186 15 years ago

> Google’s “experiment” was rigged to manipulate Bing search results through a type of attack also known as “click fraud.

hunh?

> So big and noticeable that we are told Google took notice and began to worry. Then a short time later, here come the honeypot attacks. Is the timing purely coincidence? Are industry discussions about search quality to be ignored? Is this simply a response to the fact that some people in the industry are beginning to ask whether Bing is as good or in some cases better than Google on core web relevance?

I can't believe this was written by someone with 'Vice President' in their title ... how childish.

vash_stampy2• 15 years ago

Read google's "Bing Sting" experiment on Google's blog. Their test had no control variable (ie: running the same test on another website that wasn't Google). If they had done their expirement on another site and Bing results didn't change then Google would have a very plausible case against Bing.

Google fell into the classic trap of confirmation bias with bad scientific method. And thus the test was 'rigged', they never had a control variable.

makmanalp• 15 years ago

What would be a suitable control variable? How would one know if Bing is using results from that website too or not?

moultano11• 15 years ago

Their assertion was not "bing is using clicks on google search and only google search." The assertion was just "bing is using clicks on google search." They've demonstrated that pretty conclusively.

haberman3• 15 years ago

Your "control variable" only matters if the hypothesis they are trying to prove is "Bing special-cases Google."

That is not the hypothesis. The hypothesis is "Bing has results in its index that it could not have gotten in any other way than from Google search results." Their experiment does indeed confirm that hypothesis.

scottmp10• 15 years ago

My understanding is that running the experiment on some random other sites would work as well as long as Bing user's were actually clicking the honeypot links. This is probably the intended functionality of the click stream data. That doesn't change the fact that using this type of data from competitor search engines results in "stealing" (for lack of a better word) search results, particularly for rare terms such as "tarsorrhaphy".

napierzaza2• 15 years ago

So what exactly would a control that satisfies you be? How about I submit search terms on the computer terminal that connects to a server that isn't even on the internet. There, those searches do not end up affecting Bing.

The fact is that they used unique search terms that link to unique subjects. They ONLY submitted searches using what they described. That means the only places those searches passed through were the OS, toolbar, IE and Google. They know Google received those searches, did it propagate to Bing?

That's it. They didn't need to offer a placebo to anyone. It's like throwing a ball and hearing an echo. You don't need to NOT throw a ball just to make sure it doesn't echo.

thankuz• 15 years ago

Not only that, there's too many hypothetical, open ended questions - they don't really address anything. Instead they leave it to the reader to imagine. Not impressed.

moultano11• 15 years ago

This seems to be mostly ad-hominem arguments in the classic definition of it: "You're wrong because there's something bad about you," rather than addressing any particular point. I don't see how questioning Google's motivations for uncovering this is doing anything to "set the record straight."

A useful post would have addressed at least 3 things:

- What are the specifics of the mechanism by which they ended up obviously copying google's results?

- Do they handle clicks from google differently than any other clicks?

- How different would their results be if they didn't use clicks from google's search as a signal?

ddemchuk2• 15 years ago

I wouldn't say it was obvious, mainly because this was something Google did specifically to cause this. I would imagine you could replicate the same results with other major search engines simply by doing the same thing with the same amount of volume as the Google dudes did.

Additionally, if things were reversed and Google was posed these questions, I would imagine would be lacking just as many answers as what Bing is supplying.

txxxxd• 15 years ago

> I wouldn't say it was obvious, mainly because this was something Google did specifically to cause this

There is a subtle point that many posts like this are overlooking. Google didn't run this experiment to cause bing to show bogus results, they did it to confirm the rise in suspiciously similar results produced by bing.

napierzaza2• 15 years ago

>Additionally, if things were reversed and Google was posed these questions, I would imagine would be lacking just as many answers as what Bing is supplying.

That's like saying you'd be evasive like OJ Simpson if you murdered a few people and were on trial.

So if Google used Bing to rank searches it would also be under scrutiny? Since that's not the case we can all avoid the useless thought experiment can't we?

earl3• 15 years ago

"- How different would their results be if they didn't use clicks from google's search as a signal?"

That's the important one. These Bing people are happy to run their mouths about 1k signals blah blah blah. Who gives a fuck? What is the weight on those signals.

Further, this case was made obvious because Bing couldn't create an answer to a query so they copied Google's answer. Dude admits as much. Why should we give Bing a pass on copying Google's results just because in this instance it was too hard to find their own results?

brudgers5• 15 years ago

>"What is the weight on those signals."

Low enough that it took Google nearly three engineers per successfully injected honeypot (7 honeypots per 20 engineers) and Google was only able to achieve a 7% success rate despite their extensive in-house knowledge of SEO.

Aaronontheweb3• 15 years ago

Have the Bing team expose their algorithms and criterion to the public so they can settle a spat with Google over 7/100 made up words appearing in a set of search results?

That sounds reasonable.

zzleeper5• 15 years ago

I've seen this 1k signals BS repeated over and over by HN commenters, as if somehow copying from 999 other guys makes this a nonproblem.

waterlesscloud• 15 years ago

The weight on keywords that exist nowhere else on the internet other than the fake searches Google injected?

Well, if there's one source of data, that data is weighted at 100%.

The whole experiment is meaningless.

megaman821• 15 years ago

- What are the specifics of the mechanism by which they ended up obviously copying google's results? They said is was user click stream data. User types search into Google, gets results back and clicks link. Microsoft gets sent the search term and link clicked.

- Do they handle clicks from google differently than any other clicks? Hopefully they do if they are using user clickstream data. Each domain should have something like a page rank to determine how trustworthy it is.

- How different would their results be if they didn't use clicks from google's search as a signal? Why would they not use Google. Google is a site on the internet where users click links, Microsoft is collecting data from sites on the internet where users click links.

rflrob3• 15 years ago

> Hopefully they do if they are using user clickstream data. Each domain should have something like a page rank to determine how trustworthy it is.

If there's some algorithmic reason to believe that Google gives more trustworthy results, that's one thing, and the pagerank weight can be empirically determined. If, on the other hand, there's some point where they explicitly treat google clickthrough data any differently (if site=='google.com' weight+=10), then that seems to have crossed a line.

I'm trying and failing not to have this sound like some kind of koan, but if you treat everyone differently in the same way, then that's (arguably) fine. It's only if Google gets some specific attention that it seems more malicious.

amalcon4• 15 years ago

I'm about 80% in the "Nothing fishy here" camp on this issue, but you have pointed out the 20% very well right here.

Obviously they won't explain the first one, just as Google won't explain the specifics of why StackOverflow scrapers sometimes outrank StackOverflow, and for good reason. Explaining the specifics of the mechanism would open it wide to spammers.

The second? Yeah, that's the million-dollar question. They should answer that. Definitely. It touches on the smelliest thing about the issue. If I had a search engine (I don't), and I wanted to copy Google's results (I wouldn't), and I had the ability to collect user click data, I could use that click data to create plausible deniability for the copying. This is exactly the sort of thing Microsoft has done in the past on other issues.

The third? It's certainly relevant, but I doubt Google would be willing to tell me how its results would be different if it didn't use a specific metric.

lionhearted• 15 years ago

Did we read the same article? I like Google and crew more than Microsoft, but this was pretty clear to me:

> Google engaged in a “honeypot” attack to trick Bing. In simple terms, Google’s “experiment” was rigged to manipulate Bing search results through a type of attack also known as “click fraud.” That’s right, the same type of attack employed by spammers on the web to trick consumers and produce bogus search results. What does all this cloak and dagger click fraud prove? Nothing anyone in the industry doesn’t already know. As we have said before and again in this post, we use click stream optionally provided by consumers in an anonymous fashion as one of 1,000 signals to try and determine whether a site might make sense to be in our index.

Beyond that, it's kind of crazy to think they'd open up on nitty-gritty details about their algorithm - nobody does that.

Anyways, I'm still with Google. I hope Google wins. But attacking Bing here was probably a tactical mistake - pretty much all marketing thought ever says "Don't attack-market against upstarts if you're the market leader!" You can't win if you're #1 and you do that. Google's #1. Attacking Bing was a really bad tactical move, though I'm still casually rooting for Google to win.

thenonhacker• 15 years ago

Google, Google. Stop crying. Just continue copying Bing. Infinite scroll images. And large image backgrounds on the search page. Copy the Maps features, too.

Groxx• 15 years ago

I boggle that people think infinite scroll came from (X). People have been hacking infinite scroll onto pages that don't support it for years, frequently through Greasemonkey scripts and there are a few very powerful extensions for essentially every browser out there. If you want to claim Google copied it from Bing, Bing copied it from creative users. I highly doubt there has been any significant amount (if any at all) of UI/UX original-creativity on any large website before someone else made a quick demo on their personal site or hacked it on top of existing sites through browsing tools.

Kylekramer• 15 years ago

This whole debacle is dragging both companies down. Overstatement and intentionally framing the story by Google, vague answers and changing the topic by Microsoft.

Though in the end, Google probably gets a net win cause they got "Microsoft copied us" headlines from some major sources with probably little follow up from the average reader. But the whole situation could have been handled better.

georgemcbay• 15 years ago

I can't speak for the masses but this is a net lose for Google in my eyes because it paints them as being extremely hypocritical IMO.

Keep in mind that we're talking about Google here. The Google of Google News, the web-corpus assisted translation engine, the Google book scanning project, wifi scanning, etc. Not that I'm negative on any of those projects, but based on the amazing degree to which their business piggybacks off the work of others it does make it difficult for me to take them seriously when they come out swinging at somebody else for piggybacking off of them, especially when the piggybacking going on here appears to be fairly minor and incidental.

bertil• 15 years ago

Now that we have confirmation that Bing uses clickstream data on sites that they do not control, I'm surprised that no one mentioned this makes them sensitive to click-fraud.

Copying a leader isn’t as objectionable as denying it and I would argue that Google could split their search engine into independent entities—say: crawling, algorithm, interface. That way, competitors could try to outrun them and innovate on one or the other, separately: Firefox, Siri and many others on the interface; Cloudera or Amazon on the crawl, DuckDuckGo and physics labs on the algorithm (just spitballing there). What Bing did wasn’t that different from a grease-monkey script that would supplement bing.com results with Google‘s, it seems — or at least, they could have had the humility to present it that way.

Now, their pride made them confess to being easily gamed.

gojomo3• 15 years ago

Do we have any information about whether Google uses clickstream data from users traveling between non-Google sites?

As I noted in a previous thread (http://news.ycombinator.com/item?id=2166256), Googler Amit Singhal's wording was a bit vague about whether URL trail data from things like the Google Toolbar, Analytics, Ads, or other systems ever affects rankings. (His careful wording, "put any results on Google’s results page", could mean merely that URL trail data never adds results to the set of all possible results. It could still affect rankings of pages found through other crawling.)

The use of such data is clearly allowed by Google's very broad privacy policy. I would love a clear answer from Google on this. It should be easy to give.

muro2• 15 years ago

this has been answered many times already - check the other threads.

mhansen• 15 years ago

http://twitter.com/mattcutts/status/32568287384571904

tzs• 15 years ago

> Now that we have confirmation that Bing uses clickstream data on sites that they do not control, I'm surprised that no one mentioned this makes them sensitive to click-fraud.

I would expect that they know this. The question is how susceptible does it make them? The clickstream is just one of something like a thousand factors that go into the ranking. Perhaps on any real page, it would take more fake clicks than a fraudster can generate to make a worthwhile difference.

Keep in mind that Google was experimenting with pages for which Bing had no information other than the clickstream, so the clickstream made a big difference there.

dooq• 15 years ago

Bing is admitting their results are affected by click fraud then??

pessimist• 15 years ago

Not quite - they are admitting that click fraud on google.com affects their results!

beej71• 15 years ago

And those of us with our own DNS servers know exactly where google.com is.

I wonder how smart the Bing Toolbar really is.

ot• 15 years ago

Yes, but only on hapax legomena :)

axiom• 15 years ago

From the Google blog post:

"This [torsorophy] example opened our eyes, and over the next few months we noticed that URLs from Google search results would later appear in Bing with increasing frequency for all kinds of queries: popular queries, rare or unusual queries and misspelled queries. Even search results that we would consider mistakes of our algorithms started showing up on Bing."

That's the key. Not the honeypot. The honeypot was just a test to see if they could catch them red handed.

Bottom line is that Google looked at the statistics and found that Bing results were improbably similar to Google results. Maybe they're lying about this, but I doubt it.

There's not enough information to figure out exactly to what extent Google impacts Bing results, but I would bet a lot of money that in the Bing code base there is Google specific code that behaves along the lines of "if on google, do x" rather than some generic code that just targets all sites.

HardyLeung5• 15 years ago

I'd play devil's advocate here and say that I give Bing a slight benefit of the doubt. Given they cannot perform a parallel Google search on every single user query (this would have been easiest to spot), the technique cannot be of the form "if on google, do x". Rather, it is likely they look at what users search and what results they get as one of the parameters, say normally influencing 0.2% of the search ordering on average (assuming 500 "signals"). In the case of "torsorophy", out of the 500 "signals" only one came up, the Google result (via Bing toolbar from a few users). So, their algorithm makes a note of that, like, hmm... I don't really have anything else so I'd just rehash what users have searched and subsequently visited. So the 0.2% becomes 100%, and internally Bing associates the word "torsorophy" with that website. Next time "torsorophy" is searched it returns that site.

Yes, Bing is likely using Google's data (if available, via Bing toolbar) as part of the ingredients. You can argue whether this is stealing or not, whether this is Google results or user search/visit behavior, and but I would say it isn't exactly "if on google, do x".

moultano11• 15 years ago

> I would say it isn't exactly "if on google, do x".

I wish bing had taken this opportunity to answer that question in this blog post. Since they didn't, it makes me more likely to assume the worst. All they would have to say is something like "We use anonymized click data and don't special case Google" to avoid most of the controversy. Then we would be back to debating whether it is ethical to do something theoretically ok even if you know that it practically means you'll be cribbing from a competitor.

Terretta2• 15 years ago

> "the technique cannot be of the form 'if on google, do x' ... Rather, it is likely they look at what users search and what results they get"

How do they look at what a user searches and what result they get? Ok, the "click stream" can see what pages a user visits.

But to get the search term, they are parsing something, and unless they implemented a universal search recognizer that will rank up results from any old site's search (allowing 20 SEO guys to push private SERPs to the top), it seems more than probable the parser indeed would start with "if on google, do x".

pacemkr• 15 years ago

The most I can deduce from the experiment is that Bing look at what people click on Google and that plays some role in their search ranking. If you synthesize a term that doesn't exist in nature I can see how a search algorithm can weigh the only datapoint it has, data coming from the toolbar, heavily. This may not the best approach, but a far cry from copying, which is what Bing are being accused of.

blahedo3• 15 years ago

I don't think anything about this is clear-cut. I can imagine a very simple algorithm, as follows:

* When query Q gets made more than N times at bing.com,

* Mine clickstream data for the next M urls requested after searches for Q

* Any url that appears more than T times (possibly spread across some number of users) is presumed to have been found relevant to Q, and derived either from later searches (corrected spelling) or curated sites or other search engines. Add to mapping of valid responses to Q.

It's not a very good algorithm, of course, and if you have any other source of information about Q you're probably better off using that instead. But it or something like it could explain the torsorophy example and every other part of Google's narrative, and it's not particularly suspicious or questionable, and it certainly doesn't involve targeting Google.

moultano11• 15 years ago

Except that in this case, the honeypot searches were not done at bing.com

zzleeper5• 15 years ago

I can think of an even better algorithm:

1) When a user enters google.com/search? , scrap the page (pay special attention to their spell corrector)

2) Send all the data back to MSFT

3) Profit nicely!

It's simpler, and probably works as well as the best competitor out there ;)

greendestiny• 15 years ago

The problem is that the honeypot doesn't prove anything. These are rare terms by design so clickstream data is all probably all Bing has to go on, and that clickstream data was voluntarily submitted to Microsoft by Google employees.

Google need to go to some extra steps to show that Bing is copying Google links for popular terms. If Bing is weighting clickstream data from Google searches very highly, that is more or less admitting that other search engines work better.

wrs2• 15 years ago

All search engines look for keywords in URLs. That's why domain names with keywords in them are more expensive. If you're going to mine clickstreams from the toolbar, it's not far-fetched to think you would mine the entire source URL, including the query string, for keywords. Then there would be no need to postulate special treatment for Google search URLs, since the keyword is in plain sight (google.com?q=xxxxxx).

If that is what's happening, then the moral/ethical argument against Microsoft would have to be that they should treat Google specially, by explicitly ignoring clicks on Google's result pages. That seems to me to open up quite a can of worms.

lysium5• 15 years ago

I'm wondering, if this is really the case, why does Microsoft not just point that out, but instead comes with ad hominem arguments?

las3rjock2• 15 years ago

To crawl Google URLs of the form google.com?q=x would be to disregard http://www.google.com/robots.txt , which seems like bad netiquette to me.

cryptoz• 15 years ago

> We have been clear about this for a couple of years (see Directions on Microsoft report, June 15, 2009).

That's about as unclear as anyone can be. Seriously Microsoft, this is the Web. When you want to point your readers to a document, don't fucking cite it like some printed academic paper! LINK TO IT.

pygy_• 15 years ago

It was published on Direction on Microsoft[1], which is supposed to be an "Independant Analysis of Microsoft Technology & Strategy".

The yearly subscription is $1500.

If [2] can be transposed, non-subscribers can also get the report for a whopping $750.

I couldn't find that report in their archive,though. The only article published on june 15 2009 is [3], which is unrelated.

[1] https://www.directionsonmicrosoft.com/

[2] http://www.reuters.com/article/2009/04/21/idUS223966+21-Apr-... :

[3] http://www.directionsonmicrosoft.com/update/50-june-2009/678...

fady• 15 years ago

after reading the first few paragraphs, i closed the tab.. a waste of space. all PR crap.

brudgers5• 15 years ago

Whether it was a type of attack also known as “click fraud,” or not, it does appear to be an attack. One can argue that the attack was justified - and one can argue that it's not - but it still was a deliberate effort to improve page rank in a way similar to one which Google actively discourages.

The fact that only 7/100 of the honeypots were successful may mean that Google's motives may have been more complex and slightly less akin to righteous indignation - such as determining how and to what degree Bing handles "click fraud" type attacks. I just have a hard time believing that Google would turn 20 engineers loose on a task that can be so easily automated.

rflrob3• 15 years ago

>Whether it was a type of attack also known as “click fraud,” or not, it does appear to be an attack.

I disagree with describing this as an attack. Based on the kinds of search terms Google has said they used, they aren't attempting to make the Bing results any worse. There is simply no good search result for "hiybbprqag", so Google pointing to the Wiltern Seating Chart is just as good as pointing to nothing at all from the purposes of what any user who happened to Google hiybbprqag randomly would see.

At best, you could make the argument that by revealing that Bing is susceptible to these kinds of attacks, they are making it easier for others to attack Bing. It's certainly illegal to rob a bank, but what about walking down the street yelling, "The bank guards are all gone! The combination to the vault is 1-2-3-4-5!"

brudgers5• 15 years ago

"what any user who happened to Google hiybbprqag randomly"

But it wasn't any user, it was a Google engineer and the circumstances were not ordinary. Those engineers were actively trying to get the results into Bing over the course of at least two weeks 12/17-12/31 according to searchenginland.com.

trunnell• 15 years ago

I have a much lower opinion of the Bing team after reading this non-denial denial. It's disingenuous to act surprised that using clickstream data from your main competitor might be called copying.

hackinthebochs8• 15 years ago

And I have much lower opinion of the Google team (and they had much further to fall). They intentionally use ambiguous and loaded words like "copying" specifically to stir up drama. Google is smart enough to use precise words when they want, and not to when they want. The ambiguity was intentional.

moultano11• 15 years ago

I don't think it's ambiguous. They are contending that bing is intentionally using this to recreate results similar to google's. I'm guessing you feel this is ambiguous because you don't think it is supported by the data, not because they are saying anything unclear.

bhavin• 15 years ago

One difference between Google's vs MS's approach. While Google's first post was much more rational (with proofs) and convincing to tech minds, this post seems more a politician's attack on Google with vague and scary sounding words like "click fraud". Only if some evidence for the above "click fraud" were supplied!!

draz• 15 years ago

First, I have friends working for Microsoft as well as Google. I like and hate both companies, for different reasons. Now, to the meat of what I wanted to get to: "We have brought a number of things to market that we are very proud of -- our daily home page photos, infinite scroll in image search, great travel and shopping experiences, a new and more useful visual approach to search, and partnerships..." Seriously?? Are you going to put "daily home page photos" as (first!) point to be proud of? Is it such a technological feat? Kudos for the rest of the items, but I just found it funny

jmaygarden• 15 years ago

The first comment to the blog post is humorous.

  Bing, "Powered by Google". It does have a nice ring to it.

deno• 15 years ago

Another, quite tongue-in-cheek, humorous app found in comments:

http://createasearchengine.appspot.com/

zzleeper5• 15 years ago

That's a nice one!!! Over 1000 thousand!

tristanperry• 15 years ago

>> "As we have said before and again in this post, we use click stream optionally provided by consumers in an anonymous fashion as one of 1,000 signals to try and determine whether a site might make sense to be in our index."

Erm, the Google test took pretty non-sensical keywords and put at #1 some completely unrelated website.

And then Bing copied this unrelated website result from Google.

The above quote from Bing (especially the bit in italics) is therefore a little confusing. Since, again, the entire point here is that Bing shown completely unrelated websites taking straight from Google. The results weren't relevant. Thus making the quote fairly contradictory in my opinion.

Yet another 'reply' by Bing, and yet again they seem to be trying to insult their way out of it via ad hominem and strawman arguments.

And yet again, Bing still haven't addressed the key point here: why Google results were taken and used by Bing.

zepolen• 15 years ago

Bing sees that I searched for a term, 'iwejhoihfe' on google and ended up clicking a link to foobar.com/page there, this gives foobar.com/page a relevance with the term 'iwejhoihfe'. Since the 'iwejhoihfe' doesn't show up anywhere else on the web, that single relevant site gains a lot of weight for that term.

This is how Bing's algorithm may have worked, and I wouldn't call it intentionally copying, flawed maybe and easily manipulated just like Google Bombing in the old days.

hexis• 15 years ago

Bing seems to want me to sign in to read this. Am I doing something wrong?

jpwagner5• 15 years ago

Great response. I wouldn't have used the term "click fraud," but the analogy is easy to grasp (ie, that the "test" purposely attempts to strengthen an association between things that are not associated.)

lysium5• 15 years ago

How is that a great response? It does not address the allegations at all, just claims the allegations are insulting.

The only great thing I see is that the blog post actually confirms what Google said. Because otherwise, it would have been easy for Microsoft to put the finger at the point where Google is wrong.

jpwagner5• 15 years ago

I think this response is convincing. Period. Full stop.

blauwbilgorgel5• 15 years ago

From a logical debating point of view, this post is really weak. If I were an engineer at Google, and I caught Bing at this, I would be upset, to the point I couldn't sleep. This is one of the lowest things you can do in academia, and even though this is business, the accomplishments of the engineering team at Google is at the highest point of information technology. As a businessman I could have taken this, not as a hacker/problem solver.

"It was interesting to watch the level of protest and feigned outrage from Google. One wonders what brought them to a place where they would level these kinds of accusations." Feigned outrage... one wonders... If you really not understand this, you are not a hacker, you are not an engineer. Maybe it was your actions that prompted two valued Google engineers (Singhal and Cutts) to make these accusations?

"Before we explore that (so you gonna prove this point later on?), let me clear up a few things once and for all.

We do not copy results from any of our competitors. Period. Full stop." The end result is a copy of the results of a competitor. If you borrow it, steal it from your users, copy it from a guy in a raincoat with binoculars, observing Google results and jotting them down -- It doesn't matter! OK, let's say you don't copy results. Your index still contains copied results for a fact.

"We have some of the best minds in the world at work on search quality and relevance, and for a competitor to accuse any one of these people of such activity is just insulting." Bing copied Google's fake results. Your engineers either deliberately or accidentally made Bing copy those results. If you want less insulting: Google claims your minds are the best at copying others.

" We do look at anonymous click stream data as one of more than a thousand inputs into our ranking algorithm. We learn from our customers as they traverse the web, a common practice in helping to improve a wide array of online services. We have been clear about this for a couple of years (see Directions on Microsoft report, June 15, 2009). " That will make me trust your index a lot less. If engineers clicking manually on some results can make a difference in ranking, imagine what a botnet or dedicated spammer can do.

" Google engaged in a “honeypot” attack to trick Bing. In simple terms, Google’s “experiment” was rigged to manipulate Bing search results through a type of attack also known as “click fraud.” That’s right, the same type of attack employed by spammers on the web to trick consumers and produce bogus search results. What does all this cloak and dagger click fraud prove? Nothing anyone in the industry doesn’t already know. As we have said before and again in this post, we use click stream optionally provided by consumers in an anonymous fashion as one of 1,000 signals to try and determine whether a site might make sense to be in our index. " Ad hominems: Honeypot, trick, quotes:experiment, rigged, manipulate, attack, click fraud, attack, spammers, trick consumers, bogus, cloak dagger, click fraud, prove? nothing. If you need those words in a paragraph to describe how you got caught with your hand in the cookie jar, then you already lost.

" Now let’s move the conversation to what might really be going on behind the scenes. " Yes, move the conversation away from your duplicate results, the key issue at hand.

" Bing was launched nearly two years ago .to break new ground and help move the search industry in new directions. We have brought a number of things to market that we are very proud of -- our daily home page photos, infinite scroll in image search, great travel and shopping experiences, a new and more useful visual approach to search, and partnerships with key leaders like Facebook and Twitter. If you are keeping tabs, you will notice Google has “copied” a few of these. Whether they have done it well we leave to customers. But more importantly, we take no issue and are glad we could help move the industry to adopt some good ideas. " copies marketing paragraph. Rebuttal: Google didn't use its toolbar to "copy" your pictures and add those to their own background.

" At the same time, we have been making steady, quiet progress on core search relevance. In October 2010 we released a series of big, noticeable improvements to Bing’s relevance. So big and noticeable that we are told Google took notice and began to worry. Then a short time later, here come the honeypot attacks. Is the timing purely coincidence? Are industry discussions about search quality to be ignored? Is this simply a response to the fact that some people in the industry are beginning to ask whether Bing is as good or in some cases better than Google on core web relevance? " Bing is absolutely horrible for international searches. You serve up 40 different language Wikipedia pages for some terms. But you are saying, that Google deliberately timed your outings? That is like saying your brother saw you stealing cookies, but waited till he got a bad report card, before telling mom. And I thought a bruised academic ego was childish. So your rebutal amounts to: But mom! He got a bad report card! And that while you are struggling making your grades yourself.

" Clearly that’s a question that will continue in heated debate as long as there is a search industry. Here at Bing we will continue to focus on our customers, and try to provide some great innovation for consumers and the industry. " That's not the question and certainly not core to this case, which is your duplicate index. This is unprecedented in 10 years of search engine competition, and you want to make it about the timing? Let's make it about the 1000 ranking factors vs. the 200 ranking factor from Google. Lets make it about the million other results this could be happening for, but are impossible to prove with a "honeypot attack" and specific keyword? Lets make it about search relevance, how we are spoiled with relevant results, yet complain about the most relevant index on the web, one you use to calibrate long tail terms and spelling corrections?

hackinthebochs8• 15 years ago

>The end result is a copy of the results of a competitor. If you borrow it, steal it from your users, ...Your index still contains copied results for a fact.

This is pure nonsense, and so is the rest of your post. [edit: The business of] search is not about academia, or algorithms, or engineering, its about providing users with relevant responses to their queries. The best way currently to do that is through crowdsourcing--which is exactly what Google does through Pagerank. Now Bing took that a step further and creates an association between what a user searches for and what they end up finding relevant: This is exactly what we currently call "search". Whether that initial directory of sites came from Google or from some hand culled list is inconsequential. This is not "copying", this is doing exactly what they should be doing.

The point is Google doesn't own the link, or the form input, or the fact that many users clicked on that link after posting that term. The only thing they could be reasonably construed to "own" are the relative rankings themselves. Bing did not "copy" this.

moultano11• 15 years ago

>Bing did not "copy" this.

People only click on the results that are there. Google put them there. People tend to click on results in roughly the order they appear on the page, which Google also determined. Getting the data through an indirect means does not insulate them from culpability.

blauwbilgorgel5• 15 years ago

"This is pure nonsense, and so is the rest of your post." Thank you. I don't really know how to respond to this, but maybe say that your own statement is so weak it just contains its own counterargument.

"[The business of] search is not about academia, or algorithms, or engineering, its about providing users with relevant responses to their queries"

And the food industry is not about the ingredients, the cooks and the recipes, but about providing users with a dish suitable to their tastes. In fact, we could do without the cooks, recipes and ingredients. er...?

"The best way currently to do that is through crowdsourcing". These are fine words to say that Bing uses rank on Google as 1 in 1000 ranking factors. Bing uses GoogleRank. Or CrowdRank in less polemic terms.

To me: it isn't about if Microsoft is "copying" or not anymore. You can win that semantic game. What you can't win or explain away is the end result: the duplicate results and grammar corrections. The process by which this happened (Microsoft can hack my microphone and record keystrokes) is irrelevant. Bing uses rank on Google as a ranking factor and Bing contains duplicate results.

antimatter15• 15 years ago

It's not Ad Hominem because the Bing team isn't personally insulting Cutts or Singhai. If anything, it's just using malicious sounding buzzwords. A honeypot isn't even isn't even a type of attack: it's a type of defense.

blauwbilgorgel5• 15 years ago

"It's not Ad Hominem because the Bing team isn't personally insulting Cutts or Singhai. If anything, it's just using malicious sounding buzzwords. A honeypot isn't even isn't even a type of attack: it's a type of defense." They not directly insulted Cutts and Singhal, but they offered a fake rebuttal of their actions/proof/views, peppered with malicious sounding buzzwords.

...have nothing to do with the logical merits of the opponent's arguments or assertions... The proof was concocted by these engineers.

...but can also involve pointing out factual but ostensible character flaws or actions which are irrelevant to the opponent's argument...

The way his statement could be read is: Matt Cutts and Singhal are no more than spammers, tricking Microsoft with cloak and dagger tactics that prove nothing. Which constitutes a clear Ad Hominem.

Matt_Cutts2• 15 years ago

The post said my feelings were feigned (definition: "to give a false appearance of"). My feelings weren't feigned.

(Also, it's spelled Singhal, not Singhai)

Matt_Cutts2• 15 years ago

"It was interesting to watch the level of protest and feigned outrage from Google."

This was the phrase that almost made me want to do another round of posts. As the person on the panel from Google, I can assure you: it was not feigned outrage. It was real frustration.

nigelsampson• 15 years ago

What I'm interested in, does the Google Toolbar send back the click stream when you use Bing?

noibl• 15 years ago

Feel like experimenting?

yalogin• 15 years ago

Going by what MS says, its clear why Google's experiment worked. They came up with very unusual queries for which by definition there will be no data to generate search results. So out of the 1000 inputs only the one input from Google is present and hence the output.

blauwbilgorgel5• 15 years ago

Yes, but then this also means that for every term in Bing's index, "how it ranks on Google" is 1 of the 1000 factors. It takes a very obscure term to show it, but we shouldn't conclude from that, it happens only for obscure terms.

BING!! uses GoogleRank as a ranking factor. Perhaps they even adjust the weights of this factor, according to what the press/blogosphere is writing about Google's index quality...

lysium5• 15 years ago

This is the usual FUD we are used to from Microsoft.

First just claim that the allegations are wrong (use 'Period.' and 'Full Stop.'). Don't try to explain!

Than redirect allegations as a personal attack (say it is 'insulting'). Don't try to explain why, either!

Refer to a document that requires paid membership, so only very few people if any can check (don't use a direct link). Gives you cheap credibility.

Claim that the experiment is fraud, although no one benefits financially. Don't back up the claim with data.

Counterclaim that your competitor is actually copying from you, ignoring the difference between ideas and data.

Now that the reader envisions you as the poor underdog that is insulted, is a fraud victim, and is copied, spread the doubt that Google is doing this because it starts to 'worry' (pose an open question like 'is it a coincidence'?).

End with the pose that your are not affected by this and concentrate on your business.

Never, ever answer the allegations with a simple explanation on why Bing shows bogus results because some Google employees search while the Bing Toolbar was active.

{Overlook that you admitted that your are prone to click fraud.}

billybob• 15 years ago

Thank you for summing up why posts like this one from MS make me queasy. Lots of spin and posturing and insinuation when a clear, technical answer is needed.

It's like, "Son, did you steal those cookies?" "Mom, Justin always accuses me of stealing cookies, and I've been getting good grades lately, and last week Justin hit me for no reason, and don't you think he should be in trouble too?"

Answer the question, son.

lysium5• 15 years ago

The post indirectly confirms Google's claims. Otherwise, it would have been easy for Microsoft to point out where Google is wrong and everybody would blame Google for being stupid.

Instead, Microsoft chose to attack Google 'personally' with non-substantiated claims.

BoppreH2• 15 years ago

"anonymous click stream data"

It can't be only that. The page where the clicks happen must be parsed for the click be linked to the query and reused at their search engine.

They transformed: "from google.com/q=X, people tend to click on example.com"

Into: "when searching for X, people tend to click on example.com"

andyv• 15 years ago

They not only parsed the search queries, they also copied Google's results and added it to their own results.

BoppreH2• 15 years ago

Monitoring the clicks and parsing the data should be enough to account for the results we see.

skarayan• 15 years ago

It would be interesting to compare the search results of a million or so keywords between Google and Bing using their APIs. I would like to see some numbers behind the overlap.

thought_alarm• 15 years ago

This is the record:

Searching for "torsorophy" on Google brings up the Wikipedia page for Tarsorrhaphy because of Google's advanced spelling and error correction algorithms.

Searching for "torsorophy" on Bing will (or used to) bring up the Wikipedia page for Tarsorrhaphy because of Google's advanced spelling and error correction algorithms, because Bing watches how people use Google.

Two different kinds of innovation at play, one much more legitimate and impressive than the other. But it's not like Microsoft is in unfamiliar territory here.

kenjackson2• 15 years ago

So does "fast forier twansform" -- did Google honeypot that term too? Or the other 20 terms I just tried with arbitrary mispellings?

true_religion2• 15 years ago

When Bing adjusts the results it shows due to user click-ranking, it does so without any notification. That's because its a ranking issue.

When Bing adjusts the results as due to a spellcheck done by their engine, it notifies the user so they can correct it.

On your example, Bing search does exactly this: > We're including results for fast fourier transform. Do you want results only for fast forier twansform?

bluelu• 15 years ago

All I can say is, that google engineers must be really jealous for not coming up with the same idea of ranking sites based on what people enter into search boxes and what links they click on. Sounds like Pagerank 2.0 to me.

blauwbilgorgel5• 15 years ago

There were AI algorithms in 1999 that described doing exactly this. Pagerank is perfect for the web. Agreed, tracking people like ants, the links they click on as pheromone trails, makes for a great program: 90% of users visiting website A about pancakes, went to site B about pancake recipes. The problem lies in tracking your users to this extend. On the web this isn't possible, unless you control vast web estate and are unscrupulous about using this data. I bet most Google engineers know about swarm intelligence computing, and aren't jealous at engineers like: http://www.youtube.com/watch?v=7GM4Lt5k24s who don't Bing! loud enough.

slig• 15 years ago

Yes, but from what other relevant search engine would them copy that?

ps: FYI, Google does record the clickstream within their own search results.

bmccormack• 15 years ago

Regardless of whether or not one agrees with Yusuf Mehdi's conclusions, I can certainly understand his passion in defending his team. Google's accusations yesterday were very accusatory and polemic (e.g. "We look forward to competing with genuinely new search algorithms out there—algorithms built on core innovation, and not on recycled search results from a competitor. " ouch).

naner• 15 years ago

Just once it would be great if tech companies could lay off the hyperbole and posturing and just give it to us straight. Bing copying Google? Not really. Honeypot attack? Give me a break.

Here's what appears to be the truth:

The Bing toolbar keeps track of what people click on after doing a search. Even when it is from sites that aren't Bing. If it is a search for something obscure (or made up) that Bing has no other data for, then that clickstream data can affect Bing results for that term. Most of the time, however, that is just one of thousands of inputs. It is useful data, but not a defining part of the Bing engine.

This was obviously a huge PR move on Google's part. The information was released to a massive search engine blog right before a big search engine event with both Google and Bing in attendance. Bing could have come out of this shooting straight and looking like the more mature party, instead the come across like they are unsure of themselves and have been backed into a corner.

Score one for the bullshit artists at Google.

gvb• 15 years ago

One really interesting thing about Bing watching google searches and changing their page ranking based on users clicking links... it implies one way to "SEO" (game) Bing page ranking would be to use Amazon Mechanical Turk to raise a page's rank on Bing by having lots of Mechanical Turks doing google searches and click-throughs on the target page.

It would be interesting experiment to estimate how heavily weighted the clicking is vs. the other 999 inputs. I would think a very obscure page would be fairly easily manipulated (Google basically showed that with their "honeypot"), but a more popular page with lots of non-zero entries in the other 999 inputs would be a lot harder to game.

contextfree2• 15 years ago

Apart from his flustered tone, Mehdi said nothing that Harry Shum didn't say in his post yesterday. I don't know why he bothered posting this.

InclinedPlane4• 15 years ago

Here's the problem in a nutshell:

The Bing toolbar uses clickstream data to extract information about the relatedness of urls, just like any search engine crawler does (page rank works this way). When a user clicks on url B from url A the Bing toolbar sends information back to MS about the relatedness of urls A and B including all the meta-data involved. If url A is ".../search?q=torsorophy" then Bing will make a note of that and will start showing results for B when you search for "torsorophy".

In principle this isn't necessarily a bad thing, it allows Bing to index sites that it wouldn't otherwise, letting its search index grow more organically. However, when search engines come into play things get problematic, because now the Bing toolbar is little more than an automated method for scraping search results piecemeal. Given that a great deal of modern web surfing falls into the "search for X, click links for X" pattern, this should have been something that the Bing engineers anticipated (if they didn't anticipate it that's bad enough, if they did and ignored the problem that's much worse).

Worse yet, the Bing toolbar is effectively a search indexer which does not respect robots.txt. Let that sink in for a while. Google.com/robots.txt has this line: "Disallow: /search", and yet apparently the Bing toolbar has absolutely no compunction about effectively ignoring that.

tl;dr: MS has created a search indexer which ignores robots.txt, this is bad.

recoiledsnake• 15 years ago

Err, robots.txt doesn't even enter into the picture. It would matter only if Bing's server was requesting pages off Google's site. That is not happening. The only things that are going are the Google URL and only the clicked URL(not the other Google results that the user didn't click on).

The user gave permission to Microsoft to use this info by installing the toolbar. I don't see why or how the Bing tool bar should visit Google.com/robots.txt to see the blocked folders unless they were crawling Google pages.

zzleeper5• 15 years ago

But it's the perfect way to circumvent robots.txt! With enough users the law of large numbers applies (and how many hundreds of millions of users are using IE?) so you can a.s. copy Google's index.

In particular, you copy the most popular results, so if you weight the % of google's index that you copied, then it's going to be a huge number, and almost for free.

It's evil. But devilishly smart.

InclinedPlane4• 15 years ago

The contention you're making, and apparently Microsoft as well, is that the Bing toolbar is not a search indexer and does not need to obey the conventions of robots.txt?

Personally I don't think that flies. Should I be able to create a toolbar that rips the data from hulu.com and hosts it elsewhere?

jpwagner5• 15 years ago

"...the Bing toolbar is effectively a search indexer which does not respect robots.txt..."

maybe i don't understand, but your logic seems correct until the above statement.

suppose you had a system that only used clickstream data. so you store a big list of url pairs (A,B) and a probability that B will follow A. i believe that your argument relies on the fact that it's possible for this system to violate a robots.txt file and i don't yet see it.

InclinedPlane4• 15 years ago

The intention of robots.txt is to tell search systems specifically "do not use information from the following pages in building a search index". The Bing toolbar's use of clickstream data from google, and no doubt many other sites, clearly violates that spirit.

This could easily be fixed, by checking the clickstream data against robots.txt files and discarding data that shouldn't be used. Microsoft apparently has decided not to take that step.

pedrocr3• 15 years ago

Yours is the first explanation I've seen that makes any case for Microsoft not being intentionally cheating. All the responses I've seen from Microsoft are in the form "we're not copying Google we're just using user click information" which is poor since what Google showed is that they're associating click information with the search terms that were put into Google. But since those search terms are in the referrer URL it could conceivably be an innocent general algorithm weighing the words in the URL.

As for ignoring robots.txt, that may not be the case. Conceivably you could get the url A->B link, save the metadata for both, signaling them as related, and then check both URLs against robots.txt to see if you should have them in the index. Then if url A is ".../search?q=torsorophy" Google's robots.txt disallows it from being indexed and only url B gets in but the link to "torsorophy" is still there from the metadata.

InclinedPlane4• 15 years ago

Indeed. Certainly it is technologically possible for clickstream based indexing to still abide by robots.txt rules. However, the Bing toolbar does not. That is the key issue here.

samuel• 15 years ago

In fact what Google users are really clicking in searches are "google.com/url?" URL's which are also disallowed in robots.txt(while the url they redirect to aren't).

ecaron• 15 years ago

Article appears to be down, fortunately you can still access it from Google's cache: http://webcache.googleusercontent.com/search?q=cache:www.bin...

bluesnowmonkey• 15 years ago

Why are people suddenly using the word "honeypot"? A honeypot is something intended to be broken into. Nobody broke into anything in this story. There were no honeypots involved. You might call the search terms "tracers" or something.

Another thing that did not occur is click fraud. That term already means something, it involves pay-per-click advertising, and it doesn't have much to do with Google's experiment.

Don't let this guy distort the language we use to discuss the issue.

sonoffett• 15 years ago

I mentioned this in another article, but I have yet to see evidence that demonstrates that bing is parsing google's queries either from a) the url of the referrer, or b) directly on the google results page, yet many people on HN have asserted as such, generally exhibiting a false dilemma fallacy.

An alternative is that the bing toolbar is collecting 2-tuples "<search_string_in_toolbar, next_href_clicked>" and sending these back to microsoft (regardless of the search provider selected). I would consider this "click stream data" and seems to agree with statements in the above article. Additionally, since it doesn't involve parsing hrefs it seems like the easier solution (and the one I'm going to tentatively assume by Occam's razor).

Just to be clear, it does not have to be the case that the bing toolbar is collecting data entered directly into the google search box. Indeed, google could have tested for this by having a control group where they entered search queries without using the toolbar search box, however from the details released so far we cannot conclude they did this (read: google hasn't released enough specifics about the test they conducted, and it's far from being reproducible with the current details).

Additionally, why do many in the HN community think this google specific? I understand that google isn't claiming "bing copied just google" but that seems to be the consensus within HN and the arguments of foul play (see the countless posts asking if there exists code that specifies google: "if string contains 'google' then..."). I'd like to see a test where users entered a pathological string in the toolbar with an alternative search provider specified (EDIT: not bing or google), clicked on a low ranking result (low ranking, or not appearing at all on bing for those keywords), and see if it pops up higher on bing at a later time.

jeisc• 15 years ago

'Imitation is the sincerest form of flattery.' But, 'Flattery is the worst and falsest way of showing our esteem.' Jonathan Swift

lazugod• 15 years ago

It isn't clear whether Bing was taking special interest in click data off of Google searches, or giving it special weight in its own results.

Google seemed to think so, but even if Microsoft was treating the data they received as neutrally as possible, it doesn't change the request made of them. Google wants an exemption.

425• 15 years ago

To me, it seems quite simple. Someone at Google used Bing toolbar and IE8 in following way:

a) Search from Bing toolbar for hiybbprqag.

b) Bing does not display anything.

c) Bing toolbar remembers that term internally.

d) Go to www.google.com and type hiybbprqag.

e) Google displays one intentionally seeded link.

f) Google engineer click on that link.

g) Bing toolbar notices that shortly after user searched for hiybbprqag, s/he clicked on seeded link.

h) Bing toolbar sends that piece of that to mothership: There is relationship between hiybbprqag and seeded link.

...

i) After some time, Google engineer searches for hiybbprqag from Bing toolbar.

j) Bing looks up its index and there only one piece of evidence regarding term 'hiybbprqag'. It is not much, but it is all it has, so it presents it to the user.

Google accusations strongly imply (using words like 'stealing') that Bing simply scraps google.com for results, while reality is not so simple.

Now, bigger Microsoft problem is that it employs VPs like Yusuf, who cannot express simple facts and easily fall into corporate speak.

UPDATE: Here is good summary of what I wanted to say: http://directmatchmedia.com/google-proves-bing.php

motters• 15 years ago

This doesn't seem like a terribly convincing response from Bing. The "honeypot" test seems like a pretty conclusive result to me.

ItsBilly• 15 years ago

The article is titled "Setting the record straight", but it is not fair to say that "Bing sets the record straight"

Bing sets the record straight on recent accusations

Related Stories