Microsoft's Spam Bot has been Blocked

http://getclicky.com/blog/159/microsofts-spam-bot-has-been-blocked

by fallentimes • 17 years ago

37 15 17 years ago

getclicky.com

fallentimes• 17 years ago

I saw this issue come up on a few HN threads. Hopefully other analytics programs follow Clicky's lead.

csomar3• 17 years ago

Webmasters also had this problem with Cuil.

aristus2• 17 years ago

Cuil did (does?) a lot of deep searching but that's not the problem here. MSNbot is pretending to be an IE6-using human, including a fake referer header from their search engine.

Microsoft have been inflating their search traffic for years via many tricks. I think they still count every misspelling in IE that kicks to their MSN page as a "search".

briansmith• 17 years ago

Like Google Chrome?

csomar3• 17 years ago

if that the case it will be real SPAM and a shame for a such company!

jonknee2• 17 years ago

Any idea what they are doing? If it's just looking for cloaking you think it would be better to not have the search.live.com referrer. That makes it easy to game if you were somehow trying to cloak for the Live bot which I don't think is a real problem anyway.

Harkins• 17 years ago

The want to see if the page will attempt to customize a page full of spammy keywords and ads in response to the perceived interests of the visitor.

It's a smart check to make, but Microsoft Live does it so heavily that it bugs the hell out of webmasters.

ssn• 17 years ago

robots.txt won't work?

Jem• 17 years ago

If I remember correctly, it doesn't. The crawler ignores it.

The problem is, some people still want to be indexed by the Live crawler, but without the spam in the referrals. At least by blocking the crawler purely in the stats, they still have a chance at ranking with Live without the pain of sorting through hundreds of fake referrals.

eli• 17 years ago

Not only does it not work, but at least in my case the bot was hammering the site with many requests per seconed.

I blocked it with ModSecurity over a year ago.

CalmQuiet• 17 years ago

They say, "Anytime we see a referrer from serach.live.com, we check if the IP is in the range of known IP addresses for their crawler, and if so, we just ignore it."

I filter some stats according to user agent name: Anybody know if these bots use regular (honest) agent names? Or do I need to start searching for "range of knw IP addresses for their crawler?" (Anyone care to share such a range?

jonknee2• 17 years ago

It masquerades as IE 6 last I checked. It's not marked in any way as a bot, so IP addresses are the way to go.

schammy• 17 years ago

Correct, the user agent in no way indicates it's a bot. Just a standard IE6 UA.

Microsoft's Spam Bot has been Blocked

Related Stories