Here is the robots.txt of Google

by coliveira • 17 years ago

40 33 17 years ago

jacquesm4• 17 years ago

I always figured that the company that spiders everybody elses content should have a more relaxed policy towards being spidered itself.

After all, google is datamining the web on an ongoing basis, in return it should willingly consent to being mined in return.

brk• 17 years ago

Yes, but...

For the most part "Google" is a condensed version of the web. In theory, you could "spider" a single site (google.com) and build your own search database without having to go out and crawl the web at large.

It's an odd paradox, but I think one search site crawling another search site is not a good idea. And there is probably an infinite spider loop hiding in that process somewhere.

antipax• 17 years ago

Not like anyone writing a spider has to obey the robots.txt file.

stcredzero• 17 years ago

And there is probably an infinite spider loop hiding in that process somewhere.

And then, this strange loop attains self awareness... (GEB reference)

pedalpete2• 17 years ago

Though you could end up with a load of URLs, titles, and a summary text, crawling google won't really get you to the important data in building a search engine. No follow-on links, only a subset of the text, no referrals, etc.

Love the 'paradox' of the infinite spider loop, hadn't considered that before.

digitallogic• 17 years ago

Actually a lot of search sites reguarlly pull down large searchs result sets from eachother to do comparisions to their own, though of course this isn't tecnically 'crawling' in the follow every link sense.

dschoon• 17 years ago

I think the goal is to prevent other robots from accidentally crawling the Google corpus. At the end of the day, robots.txt is a stab at comity. There are many well-known services which use crawlers that fail to obey it (or identify themselves). The file merely announces, "This is all recycled content." -- notice that none of the Google corporate pages are disallowed.

Aside: let's say you've whipped up a spiffy new ranking algorithm, and you just need an index to launch your search engine. What's faster: crawling the web, or crawling Google? I don't think such an entrepreneur would pass on a big speed up just because of a text-file.

sp332• 17 years ago

What is http://google.com/unclesam ?

Edit: must have just mistyped it, works fine.

eli3• 17 years ago

try it: http://google.com/unclesam

jacquesm4• 17 years ago

it searches the .gov domain, because 'uncle sam' employees can't remember to put site:.gov in front of their searches ?

Think of it as a shortcut.

mattmaroon• 17 years ago

I'd guess over 90% of Google users (and maybe 99%) don't know about "site:" Hell, most people can't even come up with a reasonable definition for "search engine".

alexandros• 17 years ago

this is even weirder.. http://www.google.com/microsoft

benprew• 17 years ago

I would guess it's the compliment of:

http://www.google.com/linux

qeorge• 17 years ago

Looks like the default XP background, no?

midnightmonster• 17 years ago

It's not weird--it's very useful for looking up Windows support info or IE dev documentation, e.g..

mr_luc• 17 years ago

http://www.google.com/unclesam?q=wtf

eru• 17 years ago

What's so interesting about the workforce training fund?

jedc• 17 years ago

Since nearly the very beginning of Google proper (aka, not BackRub) they've had a separate search for government sites and for Linux. And also Universities... I vaguely remember both a University of Michigan and a Stanford search. (Larry Page was an undergrad at UMich).

westside1506• 17 years ago

We actually have a surprising number of customers come to us at 80legs wanting to crawl google search results. I don't think most of them are trying to reverse engineer google or anything like that. Most probably just want a fast way to find relevant topics to crawl.

They are disappointed when they learn we obey robots.txt, so we have them manually do searches to pull out seed lists for their 80legs crawls. It's a pain, but there's not really a way around it within the rules.

twoz2• 17 years ago

Does Yahoo! not have a robots.txt file?

http://yahoo.com/robots.txt Sorry, the page you requested was not found.

jdrock• 17 years ago

If that page doesn't exist, then according to specification, they don't and you can crawl any page.

twoz2• 17 years ago

Update: Some Yahoo! subdomains have 'em.

Yes:

    http://search.yahoo.com/robots.txt
    http://groups.yahoo.com/robots.txt
    http://realestate.yahoo.com/robots.txt

No:

    http://maps.yahoo.com/robots.txt
    http://omg.yahoo.com/robots.txt

whalesalad• 17 years ago

Not that this was difficult or anything, but here's a list of all of the links as links. Easier to investigate ;)

http://dpaste.org/TqnU/

Here is the robots.txt of Google

Related Stories