A Practical Exercise in Web Scraping

https://www.petekeen.net/a-practical-exercise-in-web-scraping

by zrail • 13 years ago

52 16 13 years ago

mtrimpe• 13 years ago

Wasn't there some parser other than Nokogiri which was focused more on being lenient than correct?

zrail2• 13 years ago

Maybe Hpricot[1]? It's been deprecated for a long time but from what I remember it was much more lenient on the structure of the incoming document.

[1] https://github.com/hpricot/hpricot

famousactress• 13 years ago

Python, not Ruby.. but Beautiful Soup is a lenient parser and a really awesome library: http://www.crummy.com/software/BeautifulSoup/

FBT3• 13 years ago

The author very carefully avoids mentioning what web serial this was precisely, but from context I am pretty confident he is talking about Worm [1], for anyone who is interested.

[1] https://parahumans.wordpress.com/

Tyr42• 13 years ago

How'd you figure that out?

Dru89• 13 years ago

More to the point, I think, why would you post it? The author clearly didn't want fan-made ebooks out there and the blog basically gave a step-by-step converter, given the correct url.

FBT3• 13 years ago

The author of this post drops enough hints. Also, Worm is enjoying a large surge of popularity right now, as it was just completed.

danso• 13 years ago

I've always thought web scraping was one of the best ways to teach the practical use of code. My first practice with Ruby was to scrape apartment listings from Craigslist and concatenate it into one page for easy browsing. Simple to do, and useful to your real life, and as a bonus, more practice time with programming.

(obviously I don't endorse scraping Craigslist because of TOS. Just confessing to a past infraction back when the NYC housing market was even more absurd than it is today)

wslh• 13 years ago

It is also a good way to help friends because usually you give a lot of value to others very quickly.

In the same way, I also helped a birdwatcher to download thousand of bird calls. My own rant: http://blog.databigbang.com/the-call-of-the-web-scraper/

tmikaeld• 13 years ago

Web scraping is nice to learn and when you hit that wall where it starts getting really complex - i.e you need to login to a page, have cookies, etc.. there are really good tools available, lite Outwit hub that uses the whole browser as a scraper.

prolways• 13 years ago

Isn't this wildly illegal? I thought all web scraping violated CFAA... not to mention copyright...

hfsktr• 13 years ago

In many cases how is it different from visiting each page and printing/saving the html? It's useful to create a backup of the site[1].

I'm not saying it's good or bad. It could be used for both. I know at my previous job we used it (a bit shady I think) to scrape and compare competitor prices to ours.

[1] Though I can't understand why you'd want to backup your own site but I am pretty sure I've heard of it being done.

krapp• 13 years ago

I thought all web scraping violated CFAA... not to mention copyright...

I hope this doesn't apply to meta tags and outbound links otherwise one of my projects just got a lot more awkward.

mimiflynn• 13 years ago

Isn't it more convenient to use something like Pocket for this?

zrail2• 13 years ago

Does pocket follow all of the Next links and build an ebook for you?

A Practical Exercise in Web Scraping

Related Stories