HN user
As long as you're scraping publicly available data (i.e. not going behind a login) and avoiding copyrightable content and personal data, you should be mostly fine. This article should answer most of the web scraping legality questions: https://www.crawlnow.com/blog/is-web-scraping-legal
SEEKING WORK | Web Scraping, Web Data Collection, Data-as-a-Service (at scale) | Remote
- LinkedIn: https://www.linkedin.com/in/hafizhamid/
- Website: https://www.crawlnow.com/
- Email: hamid@crawlnow.com
I can source publicly available data from anywhere on the web - any # of websites, at any scale; and provide it as regular daily/weekly feeds. Fully-managed and hosted solution, it will save you time and money you'll spend building/maintaining the tech in-house.
I said goodbye to my big tech job to start a business in the data collection space. For the past 14 years, I worked as a full-time software engineer/architect/engineering manager at companies like Lyft, Microsoft and Salesforce.
I also provide consulting services at $300/hour to startups and enterprise companies. You can take advantage of my expertise in building internet-scale applications and distributed systems. I built web crawlers & data pipelines for bing.com for 5 years and I was the first/founding engineer on Lyft's data infrastructure team.
That guide from Akin Gump is listed under the Further Reading section at the end of the article. So yeah, read that too if you can!
By the way, a call to action, especially at the bottom, doesn't make the content any less credible. Nobody works hard to create quality content on the internet for the sake of it. There is always a direct or indirect motive/promotion involved. Let the readers judge whether the content is authentic or not.