Google says there are now 10^12 unique URLs. I am proud to announce the 2nd trillion.

by bdr • 18 years ago

44 32 18 years ago

lpgauth• 18 years ago

I guess this goes to show how this metric is useless on it's own but if you compare with other previous years then you can understand how far the web as evolved.

kleevr2• 18 years ago

http://ianab.com/trillion/1099377140735.html

mynameishere2• 18 years ago

http://ianab.com/trillion/1099510313975.html

LogicHoleFlaw• 18 years ago

http://ianab.com/trillion/370743989334.html

dhotson• 18 years ago

I worked on a web crawler once before.. you've got no idea how annoying these kinds of websites are to detect.

http://en.wikipedia.org/wiki/Spider_trap

.. although I'm sure they've got some smart people working on it at Google.

aristus2• 18 years ago

You detect them like Unix detects a symlink loop: it punts on the problem and just errors after 8 symlink traversals.

The equivalent is a crawl depth limit, which could be a hard limit (dumb) or a function of page-rank (smart) and the trustworthiness of the inbound link (smarter) and also the data quality & diversity of the traversed pages (best).

There are very good reasons why Googlebot seems to hit you from one IP at a time -- it's a long-running thread that is making all sorts of decisions about your site as it crawls.

13ren• 18 years ago

Google has indexed 38 so far: http://www.google.com.au/search?q=site%3Aianab.com%2Ftrillio...

Tackling the data quality & diversity of the traversed pages (best):

Producing English text with the 40 bits, by driving a generative grammar or a markov/travesty generator, would make it harder for Google to detect that the pages are auto-generated. It's unlikely to infer the function f(URL) -> text (or even to attempt it), but would limit the recursion for the other reasons you mention.

(guessing) sites like hackernews are indexed primarily by recursion (few direct inbound links to specific stories).

anamax• 18 years ago

http://drunkmenworkhere.org/ did an interesting study of how some search engines handled infinite sites.

snprbob862• 18 years ago

I submitted http://ianab.com/trillion/0.html to http://www.google.com/addurl/ and http://search.msn.com.sg/docs/submit.aspx

Please log requests from the Google and Microsoft bots and let us know how long it takes the respective bots to figure out that every page is the same :-)

bdr• 18 years ago

http://pastebin.com/m54b7d354

Looks like it just hit a bunch of links from the first page.

snprbob862• 18 years ago

So it looks like Google grabbed about 40 links before giving up? I wonder what a good "score" is? At first, I'd guess less is better, but too few might be running the risk of throwing out potentially good pages. Too many, and the bot is just wasting effort. The 40 score could vary as well based on parallel conditions assuming many bot instances are sharing a task pool. Be sure to post the Microsoft results if/when they crawl you.

hhm2• 18 years ago

Thanks a lot for sharing this. Very interesting indeed.

mrtron• 18 years ago

Hilariously clever, and very easy to create with something like Django's regex urls.

kleevr2• 18 years ago

^trillion/(?P<number>[\d]{1,13}).html$

hhm2• 18 years ago

Regex? What for? You only need to convert the number in the url to binary, and then 1s are whites, and 0s are blacks.

attack• 18 years ago

They actually did explicitly say that this measurement ignores URL's which are non-unique enough or of insignificant worth as best that they could determine. But ah well, I guess it's just for fun.

akrito• 18 years ago

So, if the number modulo 13 is 0, a taunt, "Maybe you could try drawing something," shows up. Any other easter eggs?

jrichmond• 18 years ago

http://ianab.com/trillion/1000000000000.html ("I think you win something... ") http://ianab.com/trillion/00000000000000.html (sort of an easter egg)

unalone2• 18 years ago

If you go past a trillion, they give you a free bonus page.

jrockway• 18 years ago

I like the first message: "It's true: you can click on the boxes."

Maybe I've had too much sugar today, but that made me laugh out loud.

unalone2• 18 years ago

http://ianab.com/trillion/344940631126.html

Do all the URLs leave snarky messages?

eru• 18 years ago

No.

ajkirwin• 18 years ago

http://ianab.com/trillion/840689566653.html

UFO :D

zevu• 18 years ago

Ok, funny, but Google has indexed only 40 of the pages. Where is the rest?! huh

Chocobean• 18 years ago

the red spiders that work the gears of google got bored after 40 i guess

drewcrawford• 18 years ago

Let's play pong http://ianab.com/trillion/547597778686.html

Google says there are now 10^12 unique URLs. I am proud to announce the 2nd trillion.

Related Stories