Why is Google so hysterically hypocritical about Bing using its public data?

http://www.roughlydrafted.com/2011/02/01/why-is-google-so-hysterically-hypocritical-about-bing-using-its-public-data/

by HardyLeung • 15 years ago

182 128 15 years ago

gildur2• 15 years ago

Google makes money out of being the #1 search engine. Concurrence from others is one thing. Concurrence from someone stealing your source code (open source or not) is a whole different story. That should be reason enough, in my humble opinion.

gildur2• 15 years ago

Oh I obviously misunderstood the article. I'm so sorry.

geuis• 15 years ago

I read half of this. Its late, and I don't have the time or energy to refute point by point every single wrong thing that is written in that half.

Its easy to sum up. Yes, Google indexes hundreds of millions of sites. It does so in order for other people to be able to search and find those sites, which is important to their owners. Its a symbiotic relationship, not theft of intellectual property.

Google has spent billions of dollars in manpower and physical capacity in order to be able to do that. Also, each and every one of those sites can very easily say "don't index me" with a simple robot.txt file on their site. Some do, most don't, because most sites find it valuable for Google to index them.

Meanwhile, Microsoft is trying to compete in the same space as Google. They are also spending lots of money and manpower to build their search engine. But, by using Google's search results to improve their own product, they are acting as a parasite. Google gets no benefit from Microsoft using their data.

So to sum up my own summary: Google is a mass symbiote, Microsoft is a parasite.

axod3• 15 years ago

Also websites can opt-out of having their content indexed.

How can Google opt-out of having its search index copied by Microsoft?

The funniest argument in this has been "Bing hasn't got the data, Google has an unfair advantage etc" as if Microsoft didn't exist before Google.

Microsoft have had years headstart to get their online division doing something worthwhile, but they still can't get it right.

kevin_morrill• 15 years ago

Put up a terms of use or force users to accept a license agreement. They do no such thing today. So I think you'd have a hard time saying Microsoft has done anything legally wrong here.

mooism24• 15 years ago

jonburs• 15 years ago

No one has shown any evidence that Microsoft has directly copied Google's search index, or even violated the directives in google.com's robots.txt file.

What Microsoft apparently has done is to infer relationships from Gooogle SRPs to various sites by way of information gathered through the Bing toolbar (and possibly other documented and user opt-in channels w/in IE). Using toolbar collected signals is well-established practice -- that's how Google obtains information regarding site performance.

dschobel• 15 years ago

Google gets no benefit from Microsoft using their data.

What happened to the Google line of "the more happy searching people we have online the better" (usually brought out in justification of their android investment)?

Shouldn't the same logic apply? Especially since at least half of the people using Bing aren't doing so voluntarily § shouldn't google be willing to trade that one search for having everyone happy on the internet and consuming more of it?

Is an incremental increase in Bing's search quality really going to take market share away from google? I'm skeptical.

§ 50% of Bing's traffic comes from fb, msn and windows live mail: http://techie-buzz.com/tech-news/google-is-4th-largest-traff...

greyman2• 15 years ago

Microsoft do NOT steal Google's search results. They use clickstream data from Bing toolbar users to improve their search results.

What happened is that Google engineers engineered a use case, in which signal from those clickstream appear like a stealing of search results. Or in other words, Microsoft uses Google users behaviour in their ranking algorithm. But that's not unethical, and Google is doing the same thing.

dekz• 15 years ago

Microsoft uses behaviour

I doubt they are only targeting Google. But I completely agree, the data is not being mined from Google, it's a relationship for search term a to page b created by the user, not stolen from Google.

dansingerman• 15 years ago

I know this has been posited, but has it been confirmed anywhere that this is the case?

luigi• 15 years ago

But the effect is the same as intentional scraping and outright stealing: search results that could have only come from Google are appearing as Bing results.

Bing needs to blacklist Google from its clickstream. Simple.

nailer8• 15 years ago

I do not understand this.

* Are Microsoft saying end users are naturally clicking on nonsense phrases invented by Google?

* If not, what are they saying?

bellaire• 15 years ago

Actually, Google denies that they do the same thing pretty emphatically. Basically, they claim that the same test in reverse would not work, because the data stream from their toolbar in a reversed test would not affect their algorithms in this manner.

But agreed that it seems like a very small nit to be magnified the way it has. Indeed, why they don't do this, and why we should care that Bing does, doesn't seem to be directly addressed other than by hand-wavery and PR speak about the research they've put into their algorithms and such.

chawco• 15 years ago

My issue with it is this: Had Google not created a better algorithm would those users be clicking on those links? If Google shut down today would Microsoft be able to associate those sites with the search terms?

I should think the answer to the above is 'no' in both cases, which is why this is cheating. It's probably not illegal, but I find the practice to be unethical.

Ask yourself this: If Google shut down today, would Microsoft be providing more relevant, equally relevant or less relevant results for those searches? If the answer is less relevant, then I think it's clear there's been a lapse of ethics.

haberman2• 15 years ago

> They use clickstream data from Bing toolbar users to improve their search results.

If that leads to Bing absorbing Google's results, and eg. suggesting spelling corrections they would have never figured out except that Google thought of them first, then they are indeed stealing results, whether they meant to or not, and need to stop.

George Harrison didn't mean to copy the song "He's So Fine" when he wrote his song "My Sweet Lord," but he lost the lawsuit anyway and had to pay damages. Whether you call it "clickstream" or "user behavior," Bing is incorporating an association that could only have come from Google, and Google's robots.txt makes it very clear that robots are not allowed to mine search results.

> But that's not unethical, and Google is doing the same thing.

As has been mentioned many times, Google does not take user behavior from the toolbar as a signal for ranking.

masklinn• 15 years ago

> I read half of this. Its late, and I don't have the time or energy to refute point by point every single wrong thing that is written in that half.

It's dilger, why would you bother reading that tripe at all?

toddmorey• 15 years ago

I don't know. I actually think he has a pretty good point here. I'm generally positive on Google but I'm embarrassed by their public tantrum. If I was advising a startup with similar grievances against a competitor, I'd strongly encourage them to avoid this distraction. It just feels like bruised ego. And so far, I think Google's cause has been hurt more than helped.

wingo• 15 years ago

Regardless of whether you think Google is more or less evil than Microsoft, what does it matter how many billions they spent?

It's silly to hold onto some murky moral click-ownership here. It would still be silly, even if it hadn't given Google billions of dollars in profit, every quarter.

jrwoodruff• 15 years ago

Read the rest of it. The last half ties the argument together quite nicely.

tree_of_item2• 15 years ago

I didn't even read the article. It's late and I'm not going to bother addressing any of the author's points.

But he's wrong.

I can't understand why people make posts like this.

gcb2• 15 years ago

Wrong. Google uses the link-out of other sites to rank the linked site.

So if I have a site that review sites about x, google will use my links to rank the sites about x. It will benefit google, sites about x, advertisers for x, users searching for x. But NEVER me.

now, replace "me" for "google" and "google"for "bing". And you will see how you and google are being hypocrite

ramidarigaz• 15 years ago

You may not benefit from the indexing of your link, but you certainly benefit from the indexing of other people's links. Unless, of course, you don't use search on the internet.

va_coder3• 15 years ago

This is starting to get interesting.

Google makes money aggregating other people's content. What happens when people aggregate Google's content? What's fair?

InclinedPlane2• 15 years ago

Explain what you mean when you say that Google makes money by aggregating other people's content? Google makes money by indexing other people's content and driving traffic to other people's sites. That hardly seems like a bad thing.

redthrowaway• 15 years ago

Google News is an aggregator, hence Rupert Murdoch's problem with it.

nhebb• 15 years ago

http://books.google.com is considered one of the more egregious examples, with many authors and publicists having complained about Google taking samples of the content and posting it online. I did a Google search a while back on a graphics issue, and got the answer I needed from a Google book search result. You could argue (as Google does) that they are really just helping promote the book. But I have no need too buy - I got the answer for free.

mooism24• 15 years ago

Google don't object to meta search engines (e.g. Dogpile) aggregating their results with those of other search engines.

Google are objecting to Bing seemingly passing off Google results as Bing's own (and not including their ads, no doubt).

It's clearly legitimate for Bing to index a Google results page if it follows a link to it (assuming absence of robots.txt etc) and visa versa.

Does it matter that Bing obtained the Google results via a search from a human not affiliated with Bing?

Does it matter that Google blocks crawlers from accessing its search results, using robots.txt?

Does it matter that Google's result pages don't include the noindex instruction?

Tbh, if I was Google I'd mark my results as noindex and see whether Bing respected that.

va_coder3• 15 years ago

That's why it's interesting.

You say Google adds value. Other's don't think so.

Why can't I add value to Google's content and present it in my own way?

tristanperry• 15 years ago

Hmm. It might just be me, but I get the feeling that they aren't exactly fans of Google?!

latch• 15 years ago

Which doesn't make his point less or more valid. Just because you dislike something doesn't mean you can't provide insight and valid opinions about it.

ZeroGravitas• 15 years ago

They're fans of Apple. It's the guy from Appleinsider writing and Google is clearly considered to be an enemy of Apple.

sajidnizami• 15 years ago

First article I've read in a while that's taking Microsoft's side in the whole situation. Every paragraph of this screams publicity. I am just wondering here if this is a PR tactic. Here I would be happy to see a balanced review of both companies behavior but this is just one sided bashing.

<quote>Shame on your pretentious, obnoxious, indefensibly egregious double standard in the field of using public information to turn a profit.</quote> <<-- Ironically, Bing is doing the same and unlike content creators who can opt out by putting a robots.txt file Google cannot opt out of this because like it or not toolbar is always sending back information.

Google indexes public content but how it aggregates it and displays it is their deed. The ordering, etc is Google's work not content creators.Copying it is just like copying an IP. I would love to see this go to court and be fought.

rbanffy• 15 years ago

I think that the Phineas Barnum assumption is pretty much valid and this article proves it.

Joakal• 15 years ago

http://www.google.com/robots.txt

First two lines:

User-agent: *

Disallow: /search

user242• 15 years ago

except that that's not what robots.txt is for. Otherwise your adblock extension could be excluded via robots.txt

edit: bing toolbar is effectively an extension for MSIE. robots.txt is designed to stop spiders. Spiders aren't extensions. Therefore that's not what robots.txt is for. What did I say wrong to get downvoted?

patrickas• 15 years ago

>Otherwise your adblock extension could be excluded via robots.txt

I think that's what most people though was wrong in your comment.

Care to elaborate on how an adblock extension could be excluded via robots.txt in that fictive scenario?

patio11• 15 years ago

There is no robot involved. An actual human instructs their user agent to retrieve a document on Google. A piece of code, running in that user agent, does something. If anyone has a complaint here, it is the user, not Google. (Google really really doesn't want to spin this as "MS is spying on our users" because as soon as they say that some person is going to point out "Google is spying on the entire world" and that person will be right.)

mechanical_fish• 15 years ago

Thank you - I've been looking for the metaphor to express my impression of this whole matter, and you've given it to me.

It's Spy vs Spy:

http://en.wikipedia.org/wiki/Spy_vs._Spy

...a pitched battle between the opposite-but-indistinguishable agents of two superpowers, on a plane so far removed from the realm of actual people and their concerns that the customers don't even appear in the frame.

motti• 15 years ago

Although strictly speaking you're right, it breaks the intent of robots.txt as the users' clickstream is fed into the server-side indexing engine alongside robot crawled data.

Arguing that according to the strict letter of the spec, robots.txt only has to be obeyed by true crawlers does hold water as strictly speaking Bing could disregard robots.txt altogether - it's not certainly not legally enforceable. The intent of robots.txt is clear and Bing should be trying to obey it wherever possible.

In this case it appears the keyword was associated because it was typed into the Google search box, and from the there to the faked destination. IMHO when the clickstream was analysed it should have disregarded clickstreams that pass a robots.txt-excluded page as these could establish associations that were not supposed to revealed by crawlers.

We aught to hold Bing and any search engine to the highest standard when talking about crawling etiquette.

Also, see http://news.ycombinator.com/item?id=2169817

haberman2• 15 years ago

By that logic IE could send the entire page of search results back to Microsoft and it wouldn't be a violation of robots.txt. Because hey, it was a user-agent doing it, not a robot! If this is so in the clear, why not have Microsoft do that also?

iqster• 15 years ago

I had to reread the original Google article to verify a point. The signal Bing is alleged to be using is that users CLICKED on the result that came up ... not the results that just came up first in Google upon a search query. And these are users who agreed to install the Bing toolbar!

Here's a hypothesis. I suspect Bing is making use of data that is typed in the toolbar or browser's search box ... not data captured from text typed directly into Google's search page. This might explain why their "honeypot-take" rate was only 8%. This is a subtle point. So ... imagine if you are the coder who wrote this signal collection feature. Would you capture "term in search box,next URL clicked" OR would you capture "if search box search engine == BING or Yahoo or something else then capture term in search box, next URL clicked".

Let me propose an alternative experiment. The test clickers should have clicked on the second or third (or some position N where N > 1) link. This would have demonstrated if Bing is using the actual click information or the search results themselves directly. The former seems completely fair on Bing's part. The latter might be debatable. My point is this ... the Google geniuses fail to distinct these two cases and are muddling it up for the PR drones. This waste everyone's time and productivity. Moreover, they belittle the hard-work of engineers and scientists. It is sad that this is what it has come to.

jamesgeck02• 15 years ago

From your comment: I suspect Bing is making use of data that is typed in the toolbar or browser's search box ... not data captured from text typed directly into Google's search page.

From the official Google blog: We asked these engineers to enter the synthetic queries into the search box on the Google home page, and click on the results, i.e., the results we inserted.

dangrossman• 15 years ago

The search text is in the referring URL no matter where it was typed in. It's captured without ever looking at Google.

tallanvor• 15 years ago

I find it amusing that Dilger mentions Overture, which bought alltheweb from FAST in 2003. FAST, of course, was later acquired by Microsoft in 2008. The number of connections among the major search companies is rather amusing, if not necessarily surprising.

coffeemug• 15 years ago

Oh jeez, not again. This is the Microsoft we know and love to hate, less so now that it is falling apart and must be pitied as an underdog.

Let's see:

Market cap: Microsoft - 239.49B, Google - 195.50B. Revenue: Microsoft - 66.69B, Google - 29.32B. Profit margin: Microsoft - 30.84%, Google - 29.01%.

Falling apart. Right...

andrenotgiant• 15 years ago

All it took to piece together the author's ignorance was the

"Daniel Eran Dilger is the author of “Snow Leopard Server (Developer Reference),”

quote down at the end. I'm sure his line of thought was: "All my friends at Starbucks have MacBook Pros, so Microsoft must be doing really badly."

Probably the same reason half his argument seemed to involve Android jealousy...

j79• 15 years ago

Goodness, your comment seems to imply you have an issue with Apple products, their policies, and/or Steve Jobs. Yeah, the guy writes about OS X and Apple, but he's been a tech consultant for nearly two decades. He's worked with various technologies, from various companies. A quick Google search (heh) would have revealed this. Or, what? Did you believe he was some 20-something year old blogger regurgitating the same "FUD" you believe all Apple fanboys are spewing regarding their "Android jealousy"?

megablast• 15 years ago

Microsoft is still a behemoth, with its fingers in a lot of pies, but unlike Apple and Google, it has not manage to grab onto the future, yet. The future if going mobile, and services. Microsoft has so far lost the mobile race, and the big services do not use MS servers.

They are looking in the worst shape to face the future than they have ever, in the last 20 years.

greyman2• 15 years ago

I partially agree, but with some caveats:

1. Future is also in the games, and here Microsoft has a very competitive product (xbox/kinect)

2. mobile race is not over yet, and WP7 is conceptually a good product, combining polished UI (aka iOS) with at least some hardware variety (aka Android). It just came late to the game, but it is not yet clear if too late. I personally think they can catch up with throwing enough money into it.

3. We will see when they release OS for tablets. They are late here as well, but similar to point 2), maybe not too late.

dexen2• 15 years ago

Are you kidding yourself consciously or just 8 years behind the news? @_@

http://www.wolframalpha.com/input/?i=goog%20msft

Note how MSFT is swinging around 0% growth, while GOOG is steadily between +20% and +60% growth, in regard to share prices.

The market has spoken some time ago, and the trend doesn't look like changing anytime near soon.

davidu• 15 years ago

You fail to understand how to interpret a "relative growth" chart. Google is not "steadily between +20% and +60% growth."

Since a random point in time picked by WA it shows what you would be up on another random point in time. If you look at the historical returns of MSFT vs GOOG there is no competition. You're original $10,000 in MSFT would be worth many millions and your GOOG stock would be worth less than $100,000.

known• 15 years ago

Microsoft did nothing wrong. http://en.wikipedia.org/wiki/Deep_linking

1010011010• 15 years ago

http://createasearchengine.appspot.com

fosk• 15 years ago

I have a feeling that people's reaction on this is biased by the fact that one of the two companies is Microsoft. What happened here it's not ethically correct, but come on guys, we must be honest. This is business, it's about billions, it's about market shares. It's not a news that money is not ethical.

Google collects data too, Google does its own shit like everybody else, like Microsoft. Google has its own toolbar too. And all this story seems more a marketing move against Bing.

What I want now to happen is to have more competition, instead of crying Google should work hard to be unbeatable and competitive. History teaches us a copy it's never better than the original.

zerd• 15 years ago

Dilger first asks "Bing is using Google's search results to improve its own. But what’s wrong with that?"

If Bing uses Google's search results, then we effectively only have one major search engine. Yahoo has switched to the Bing engine, and Bing gets a big chunk of it's results from Google. So we have a search engine monopoly. That's wrong with that.

> "This is the company that indexes blogs [...], and then makes all this information available without consent"

He first asks what's wrong with Bing copying Google's results, because it's public content, then says it's immoral to index public content. Double morals? Logic fail?

tree_of_item2• 15 years ago

Where did he say it is immoral to index public content?

Google complains about Bing using public content. Google uses public content.

Snarky question? Fail?

napierzaza• 15 years ago

Not really. If Google wants to publicly shame Microsoft it can and is. They're basically saying that Bing is the new "Let me Google that for you". Since they're not currently proceeding with legal motions it's basically just a smear campaign to make people think twice about Bing.

And there's truth to it.

andrewcooke• 15 years ago

Imagine a playground. Young, pre-alpha nerds writhing in chaos. Little arms flailing. Faces red. "Poopy face". "Your mommy smells".

Nerd fight.

tnorthcutt• 15 years ago

Is anyone else bothered by the fact that the article switches back and forth between serif and sans-serif font?

droz• 15 years ago

Nope. Just you.

ddemchuk2• 15 years ago

The world's largest scraper is complaining about being scraped...when Yelp and all the rest of the local businesses directories realized they were being scraped and the data was being used to google's advantage, it doesn't appear that Google gave two shits. Tables have turned, and it's like someone stole the handball from the recess yard.

SeanLuke• 15 years ago

> when Yelp and all the rest of the local businesses directories realized they were being scraped and the data was being used to google's advantage,

http://yelp.com/robots.txt

Not a mention of GoogleBot.

ddemchuk2• 15 years ago

well, they're presented with either 1) killing their organic search traffic by blocking Google or 2) allowing the crawlers but not being able to stop their data from being aggregated and included in local results.

Yelp's API specifically says their data cannot be aggregated with other sources. When presented with that, Google simply said they weren't using the API, they were scraping.

Apparently scraping is a free pass to do whatever the hell you want if you're Google.

tungwaiyip• 15 years ago

I'll try to make my point again. There is one school of thinking that Google is profiting by farming information from the Internet, literally on the back of many people's labor. You can prove this by running a "Google Sting", that is known as "Google bombing". By having many people setting up honeypot to link an arbitrary keyword to certain URL, you seem to be able to fool Google to artifically associate such keyword with the URL, thus "proving" Google is profiting from your labor. I don't really buy this logic. But Google's accusation of Bing steal from them seems very close to this line of framing.

earl• 15 years ago

Responding to merely two points, one hopes Daniel isn't as stupid as his writing. "This is the company that indexes blogs, newspapers, and both digital and physical books, and then makes all this information available without consent in the contexts of its ads and paid search space, and is dismissal of anyone who objects to Google’s ultra liberal sense of copyright. "

robots.txt and noindex http://www.google.com/support/webmasters/bin/answer.py?hl=en...

And "Install the Google Toolbar and do a search of Bing, and Google actually directs your clickstream back for its own analysis"

Google/Matt Cutts have been very open about what they do and don't use that information for, and they aren't using click tracking.

zohebv• 15 years ago

I think Google is being disingenuous with their "bing is copying my data" story.

However, please avoid linking to roughlydrafted.com. At the risk of going ad-hominem, I just wanted to point out that if you have read any of the material on the site, you will know that it is just full of flamebait articles with an absurd Apple fixation.

Everything Android does is copying Apple. Everything Microsoft does is wrong and stupid. If flash is not supported on iPhone it is because it is the morally correct thing to do. When Apple announced the iPhone without an SDK roughlydrafted called developers stupid for asking for an SDK and that javascript on the web is the new SDK. This post however takes the cake, you must side with Bing because Google is a greater threat to Apple than Microsoft. Seriously, how does Overture even enter into this discussion. It is tiresome to see this site linked to all over the web.

Why is Google so hysterically hypocritical about Bing using its public data?

Related Stories