Ask HN: Who uses web scraping?

https://news.ycombinator.com/item?id=5211587

by AlexMcP • 13 years ago

6 3 13 years ago

news.ycombinator.com

A few survey questions about collecting data via the web:

Do you have experience with people using web scraping commercially?

What field or industry was it being used in?

Do you think it fell in a 'grey' area, or was it explicitly disallowed or allowed?

Thanks for any info!

sheraz• 13 years ago

[1] - Yes

[2] - Lead generation, selling data, consumer + market research

[3] - I think it is always a gray area with regards to another site's TOS. But the burden of proof is on them, not the scraper. That and masking your scrapers is too easy.

ankitml• 13 years ago

Yes I had done web scrapping commercially, sometime back when I had just left college. I had to scrap website(s) for getting data about all pharma products available in india. So basically market research. But as soon as I realized this was against TOS of the website, i deleted all data and communicated this to client. I had to discard the contract and return even the previous payments. Although I am happy that I did. :)

ihopngo• 13 years ago

Eventbrite used scraping in the early days. They first argued it was a publicly available website. "Public" in the sense that the site was browse-able without restriction.

The plaintiff responded that it had anti-scraping language in its user agreement. The court said "the Terms of Use are not displayed on the website in any way in which a reasonable user could be expected to notice them."

Ask HN: Who uses web scraping?

Related Stories