johnmu

And one more thing ... you have some paths that are generating more URLs on their own without showing different content, for example:

http://www.languagespy.com/politics/uk/trends/70th/70th-anni... http://www.languagespy.com/politics/uk/trends/70th/70th-anni... http://www.languagespy.com/politics/uk/trends/70th-anniversa...

I can't check at the moment, but my guess is that all of these generate the same content (and that you could add even more versions of those keywords in the path too). These were found through crawling, so somewhere within your site you're linking to them, and they're returning valid content, so we keep crawling deeper. That's essentially a normal bug worth fixing regardless of how you handle the rest.

FWIW I think the main problem is that you're essentially creating an "infinite space," meaning there's an extremely high number of URLs that are findable through crawling your pages, and the more pages we crawl, the more new ones we find. There's no general & trivial solution to crawling and indexing sites like that, so ideally you'd want to find a strategy that allows indexing of great content from your site, without overly taxing your resources on things that are irrelevant. Making those distinctions isn't always easy... but I'd really recommend taking a bit of time to work out which kinds of URLs you want crawled & indexed, and how they could be made discoverable through crawling without crawlers getting stuck elsewhere. It might even be worth blocking those pages from crawling completely (via robots.txt) until you come up with a strategy for that.

Having a page that goes nofollow/noindex and back is fine, when we recrawl it, we'll take the new state into account.

Hi, I work with the Google crawling & indexing teams. Let me check with them to see what their recommendation would be. At first glance (based on the pages cached), it seems like we're just following the links within your site, like we would with other websites. In general, there are a few things one could do in a case like this (I might have more from the team later; these are in no particular order):

- Use rel=nofollow on links you don't need to have followed (this prevents passing of PageRank, which generally means we're less likely to crawl them)

- Use 503 for rate-limiting crawlers. 503 means we'll just retry later.

- Use the crawl rate limit in Webmaster Tools (I see you submitted the report there, so that should be active soon)

- If the content is fully auto-generated, you might choose to use a "noindex,nofollow" robots meta tag on these pages to prevent them from being indexed separately. It's hard for me to judge how useful your content would be in search directly.

Unblocking via robots.txt is fine and won't cause problems.

It's important for us (I work at Google on web-search) to be transparent about these reports, and we use them to remove / block content that is malicious too (just like other sites can use the Safe-Browsing API to get information about sites they host). With regards to where it's hosted, there are two main elements involved: a site that actually hosts the exploit (which could be a Windows EXE file, etc), and a site that sends the user to that exploit. Often these are separate. Sometimes it's not even a direct embedding of a known malicious site, for example, it could be that a counter/analytics-tracking site is hacked, which could result in all other sites that use those counters/scripts unknowningly sending users to malicious content.

From talking with webmasters, I have seen almost no false-positives in this flagging, but it's sometimes very hard to find the actual exploit. It sometimes hides from some visitors (direct visitors - like the webmaster - might not see it, it might only be visible for those coming from search), sometimes is limited to geographies or devices. This makes finding the exploit hard sometimes, and fixing the website so that it's no longer vulnerable to the attack that dropped the exploit isn't easy in many cases either.

I take these warnings very seriously when I see them in the browser, even when accessing a site with a fairly locked-down & up-to-date browser. I would recommend never skipping them, even to diagnose an issue (use other tools for that).

Handling URLs with multiple slashes in them is tricky, lots of websites silently fold them into one and return the same content, so this seems like something we should handle in the same way in search.

If you want to send me specific queries (the more general, the better) and what went wrong in the search results for them, I'm more than happy to forward them to the team that works on that. I'm [this-user-name] AT google.com

(I work with the search team at Google) This was a bug on our side, and should be resolved now.

This is just a bug, sorry.

(I work at Google on webmaster issues) Your site has a manual spam action probably based on something a previous owner had on it ( https://web.archive.org/web/20110202220759/http://smitecamp.... shows scraped Yahoo Answers content). You can see that in Webmaster Tools under Search Traffic / Manual Actions. Submitting a reconsideration request should do the trick.