Top JavaScript libraries for web scraping

https://serpapi.com/blog/top-javascript-web-scraping-libraries/

by fullofdev • 3 years ago

1 2 3 years ago

gajus• 3 years ago

I am obviously biased, but Surgeon is by far the best abstraction for data extraction I've ever seen or have written.

For context, it was created to support my previous business Applaudience. We had to build scrapers for (literally) thousands of cinema websites. At some point we migrated from custom scrips to Surgeon routines and reduced overall codebase size by 70% LoC. It was a huge time saver in terms of both writing new integrations and debugging when things go wrong.

The reality is that data extraction is a highly specialized task and you need specialized software to do it well. Tools like Surgeon can abstract a ton of complexity, but they have steeper learning curve.

I still use it whenever I need to scrape anything and you can combine it with anything that outputs HTML e.g. Playwright.

Ultimately business died (covid was brutal) and I moved on to even more exciting things, but this remains one of those technologies that I wish would have received more adoption.

fullofdev• 3 years ago

Will take a look at Surgeon. Thanks for sharing your experience. The syntax is very interesting.

Top JavaScript libraries for web scraping

Related Stories