HNGlance

Top Best New Show

Google boutiques.com has no robots.txt

http://www.boutiques.com/robots.txt

by techvibe • 16 years ago

6 5 16 years ago

BO

Related Stories

Loading related stories...

Comments

5

yread• 16 years ago

Could it be user agent filtering?

ithkuil• 16 years ago

Well, you can try it with:

wget -O- http://www.boutiques.com/robots.txt -U "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

and you get the same error.

I tried a couple of other bot user agent strings from http://www.useragentstring.com/, same result

mooism2• 16 years ago

So?...

AndrewDucker• 16 years ago

Why would it?

andrenotgiant• 16 years ago

they have NOINDEX, NOFOLLOW robots META tags that tell search engine crawlers to go away.

Maybe it's their way of playing nice with all of the merchants they are listing.

Source preview

boutiques.com

HNGlance

© 2024-2026 HN Glance. Story content belongs to its original sources.

Privacy
Terms