Ask HN: Web Scraping with Node.js

https://news.ycombinator.com/item?id=11940165

by ErikAugust • 10 years ago

5 6 10 years ago

news.ycombinator.com

Anyone out there have a cachet of good Node web scraping libs, and/or other resources? Opinions?

I have the intuition that my Google searches are a bit inadequate. Thank you!

pesfandiar• 10 years ago

Should it be necessarily with Node.js? I've tried to find a good toolset in JS, but it seems other languages (e.g. Python and Java) are a lot more mature in scraping.

jneumann004• 10 years ago

I haven't used this for web scraping, but have you tried out http://phantomjs.org/. It's a great tool that lets your process webpages; I don't think it would be that difficult to set it up to scrape sites.

Nilef• 10 years ago

Try this out https://github.com/IonicaBizau/scrape-it

iends• 10 years ago

Check out https://github.com/cheeriojs/cheerio

tmpfs• 10 years ago

I found https://github.com/cgiffard/node-simplecrawler to be good, I used it for https://github.com/tmpfs/linkdown - a cli for validating web pages etc.

paulzerkel• 10 years ago

Here is a place to start: https://github.com/lorien/awesome-web-scraping/blob/master/j...

You might want to check out the Nightmare project (https://github.com/segmentio/nightmare) which makes automating an Electron instance pretty simple.

Ask HN: Web Scraping with Node.js

Related Stories