Web scraping: Reliably and efficiently pull data from pages that don't expect it

http://pyvideo.org/video/609/web-scraping-reliably-and-efficiently-pull-data

by zmitri • 14 years ago

124 45 14 years ago

i found node.js is a phenomenal scraping tool. There's also a pretty simple / easy framework for these tasks called node.io (http://node.io/).

Being able to pop up a jsdom, and extract data from the page using jquery is a lot of fun.

rhizome6• 14 years ago

If the home site for a language or framework returns 404/500s, it's not professional quality.

mistercow5• 14 years ago

I don't think that's quite fair. No server is 100% reliable, and that says nothing about the quality of the code of the project itself.

lancefisher• 14 years ago

I've found jsdom to break for quite a bit of real-world crap HTML. Next time, I'm going to try driving phantom.js with node using https://github.com/sgentle/phantomjs-node

I've had pretty good luck with phantom.js, but it is somewhat difficult to debug.

chrisohara4• 14 years ago

FYI, JSDOM isn't the default parser in node.io. It uses a faster and more forgiving parser if you can survive with a subset of jQuery functionality (https://github.com/chriso/node.io/wiki/API---CSS-Selectors-a...)

superasn2• 14 years ago

My favorite language when it comes to scraping the web has always been Perl. With tools like LWP, Mechanize and Win32::Mechanize (OLE) scraping any site is a breeze. Unfortunately, I haven't seen many good modules on CPAN for DOM processing. Of course there is the Tokeparser and XPath but those generally don't work well with street-HTML (which most sites are) and are no-where near as fast or friendly as jQuery selectors. By the way, there is one module called pQuery which is Perl's port of jQuery but that only supports a handful of selectors and doesn't work with Mechanize.

If only there was a module in Perl which could marry Mechanize and jQuery (without using IE or OLE) it would make the best scraper in the world!

mapgrep• 14 years ago

Regarding Perl DOM parsing, you might want to check out HTML::Query. Its selectors still fall short of jQuery but when I was evaluating it against pQuery it came out ahead (IIRC).

mickeyp2• 14 years ago

Actually, libxml (lxml for Python) is very, very good at handling real-life HTML content.

I'm currently on a contract where I do a significant amount of ETL and web scraping as part of the project, and I almost exclusively use lxml and XPath for parsing real-world HTML.

XPath is, without a doubt, the best tool for DOM manipulation that I can think of. And that's just XPath 1.0 -- 2.0 is reputedly even better, but no 2.0 support is forthcoming for lxml as near as I can tell.

sureshv• 14 years ago

+1 for lxml and related features (XPath, Html Cleaner & Soup-parser).

gauravk922• 14 years ago

Xpath is good when the content is well formed, but usually doesn't work well with even slightly messed up tags. Mojolicious supports broken content much better. And it has full css3 selector support, which we all know is hands down the best way to access Dom elements.

chimeracoder• 14 years ago

beautifulsoup for Python is miraculous. It takes a different approach from lxml (http://lxml.de/elementsoup.html), so sometimes one or the other will work better depending on the input, but I don't know what I'd do without it.

gauravk922• 14 years ago

Mojolicious is exactly what you are looking for. Lazy and forgiving DOM parser, and CSS3 selector built in. I've used it for scraping a couple of sites, after trying so many different libraries ranging from curl to asihttprequest for objc to mechanize for ruby, I don't see myself trying anything else now.

superasn2• 14 years ago

Mojolicious looks good. Though I've heard about it before I always thought of it as a MVC framework and never as a web scraper. Anyway, looking through the docs, I see that Mojo::DOM is what seems most useful. But does this integrate with Mechanize? Would I be able to fill/submit a form or is Mojolicious just for parsing only?

staunch• 14 years ago

Web::Scraper is great. It supports CSS selectors and XPath expressions.

http://search.cpan.org/perldoc?Web::Scraper

epoxyhockey2• 14 years ago

My last web scraping activity consisted of using PhantomJS to drive and injecting my own javascript that submitted the pertinent data to my own web service. http://phantomjs.org/

mistercow5• 14 years ago

Wow, I had used node+jsdom+jQuery to do scraping, which was a big step up from anything I had done previously. But after playing around with PhantomJS, I feel like I finally found the hammer in the toolbox after years of trying to pound nails in with duct-taped-together screwdrivers.

epoxyhockey2• 14 years ago

Check that.. I wanted to use js to submit to my own web service, but forgot about the cross-domain issues, so I just used console.log to output the data. I suppose I could have used the iframe querystring parameter trick.

easp• 14 years ago

Hmmm, seems like something you could have worked around by running phantom.js against a proxy that intercepted and rerouted requests to the data collection server.

uptown2• 14 years ago

"I suppose I could have used the iframe querystring parameter trick."

How's that work? Pass whatever data you want to process with your web service as URL parameters to the web service src URL of your iframe and it circumvents cross-site-scripting protections?

hk_kh2• 14 years ago

Also, tor + privoxy as a rule-of-thumb when your scraper IP is getting banned.

And - I know, it is (now) so old - memcached as a good place to store things.

duskwuff• 14 years ago

When your scraper IP is getting banned, that's typically a sign that you should stop, talk to the site owner, and/or reconsider what you're doing.

hk_kh2• 14 years ago

That's not always the case.

While government or private contractors do not allow to gather that data by any citizen (there are examples of cities allowing it), I find myself in the right to do so, and also, in the right to redistribute that data freely so other developers can play / investigate / learn with it.

Why? Well, for starters:

1) Their own apps suck (or are inexistent)

2) They don't want to help their own users.

3) It's fun

4) Their services are paid with public money.

5) It raises awareness on the need of public data legislation.

There's a lot more to talk on the subject.

If you want to check it, http://citybik.es

I am helping projects and visualizations like: http://bikes.oobrien.com/ or my own http://citybik.es/realtime/

nicksergeant• 14 years ago

Kind of surprised I haven't seen PyQuery in here yet: http://pypi.python.org/pypi/pyquery -- highly recommend. I've used it on several projects.

uptown2• 14 years ago

Recently I've been using this method to inject JQuery into pages that may not already have it available, then executing some custom javascript to extract the page contents:

http://www.guru.net.nz/blog/2009/06/screen-scraping-with-jqu...

elchief• 14 years ago

This one uses webdriver to get the full html from ajax pages: http://vancouverdata.blogspot.com/2012/02/less-painful-ajax-...

salvadors• 14 years ago

https://scraperwiki.com/ is a great source of thousands of pre-written scrapers to use / copy / extend etc. It's sort of like github except you can actually schedule the scrapers to run at regular intervals, and then just access the scraped data over a standard API.

ricksta• 14 years ago

Anyone recommend some equivalent tools for Ruby?

rb2k_• 14 years ago

There is Mechanize (http://mechanize.rubyforge.org/) which uses Nokogiri (nokogiri.org/) internally and thus nicely supports CSS and XPATH expressions.

There is also Capybara, usually used as a testing framework, but you can easily navigate, chose a backend (Selenium/Webkit for compatibility, mechanize for speed): https://github.com/jnicklas/capybara

tszming• 14 years ago

We at Rewritely (http://www.rewritely.com) have quite some experience in dealing large scale content migration for clients (not just a single HTML page, but a whole site) mainly using scraping techniques.

We have seen so many invalid, funny, ugly markups and edge cases and eventually, we found out the only reliable parser is actually the "browser" you use everyday - which is heavily tested by hundreds of millions of users! We chose to run the scraper in a real browser so what you scrape is what you see, not to mention you can fire up events, Ajax requests, inject JavaScript or even play Flash video... they are possible only when you have a real browser.

Also, the scraper codes need to be concise and expressive (keys: maintainable & testable), because your codes are going to break sooner or later - if you are doing serious scraping business. Less LOC = more easy to change. JavaScript & jQuery are the obvious winners in this category and what is more, they are fun to work with.

Disclaimer: we have no relationship with PhantomJS :)

Web scraping: Reliably and efficiently pull data from pages that don't expect it

Related Stories