HNGlance

Top Best New Show

Do Not Crawl in the DUST: Different URLs with Similar Text

http://www2007.org/papers/paper194.pdf

by quilby • 17 years ago

9 1 17 years ago

WW

Related Stories

Loading related stories...

Comments

1

quilby• 17 years ago

I just found this paper in my school's library. Its not new and it looks like google + yahoo + msn have maybe given up on trying to find 'DUST' because they now let you do that

( http://googlewebmastercentral.blogspot.com/2009/02/specify-y... , http://ysearchblog.com/2009/02/12/fighting-duplication-addin... , http://blogs.msdn.com/webmaster/archive/2009/02/12/partnerin... )

Its still interesting that

1. Many sites have a lot of 'DUST'

2. It is not very hard to find the 'DUST'- which obviously reduces crawling time.

Source preview

www2007.org

HNGlance

© 2024-2026 HN Glance. Story content belongs to its original sources.

Privacy
Terms