Building a respectful web crawler in PHPhttp://www.dancryer.com/2013/03/handling-robotstxt-in-php-bots by dancryer • 13 years ago 1 0 13 years agoDAdancryer.com