Ask HN: Who is running the abusive scrapers?

https://news.ycombinator.com/item?id=48844680
by jonatron • 13 days ago
9 2 13 days ago

Personally I once had to triple app capacity and double database capacity just for just one scraper. I remember Cloudflare once figured out that Perplexity was one of the bad guys ( https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ ) but has anyone else figured out who else is running large scale and poorly behaved scrapers?

Related Stories

Loading related stories...

Source preview

news.ycombinator.com