Twitter just updated its robots.txt to exclude all scrapers

by sb057 • 11 years ago

16 19 11 years ago

sb057• 11 years ago

Same file as of a few hours ago:

kenkasan• 11 years ago

so what does that mean?

minimaxir• 11 years ago

Absolutely nothing from Twitter should be appearing in search engines.

chc• 11 years ago

Except don't they have a special deal with Google to use the firehose?

teaneedz• 11 years ago

So only firehose agreements get data? What's the impact to general SEO or 3rd parties? Doesn't sound good.

Paulods2• 11 years ago

Actually i believe its only from the "www" subdomain. Take a look at the robots without the "www"

This is likely just to prevent content duplication/nudge users to visit without the "www".

searchmartin4• 11 years ago

Nope. No. They didn't.

What they did was some perfectly legitimate duplicate content protection.

Will write it up in a bit more detail...

mattbasta• 11 years ago

Nah.

They blocked robots on their marketing pages.

searchmartin4• 11 years ago

searchmartin4• 11 years ago

Here's a write up as to why they have made the changes. Was going to write it yesterday, but life got in the way:

sp3324• 11 years ago

Why would you not use 301 redirects or just rel=canonical?

searchmartin4• 11 years ago

There must be some platform issue is my best guess...

Related Stories