Web crawling and downloading ebooks with phantomJS

http://debuggerstepthrough.blogspot.co.il/2012/06/having-fun-web-crawling-with-phantomjs.html

by gillyb • 14 years ago

61 26 14 years ago

debuggerstepthrough.blogspot.co.il

radagaisus2• 14 years ago

Phantom is awesome. I tried to use it for testing, but it's too slow (10 seconds for one test). Anyone else tried it? Any tips?

wslh• 14 years ago

Is using Google Chrome faster?

radagaisus2• 14 years ago

What do you mean? Qunit testing? Yes. always.

gillyb4• 14 years ago

At my job, we do A LOT of UI testing using selenium. We mainly use the FireFoxDriver, and i did some basic tests comparing it to phantomjs, and phantomjs is much faster! If we use the plain HtmlDriver than it might be as fast, but that doesn't use WebKit, nor has good support for C#... :(

simonw• 14 years ago

We use it for hooking our JS unit tests for http://lanyrd.com/ in to our Jenkins continuous integration server. Our tests are written in QUnit (so we can run them in a regular browser), but Phantom is used to execute them from the command line as part of a Jenkins task.

We also have a separate Phantom/Casper script which we use to test that our Twitter login flow is working.

smoyer3• 14 years ago

This technique is certainly useful in a variety of instance and I've done the same thing with both HTMLUnit and JWebUnit in Java. The "great site you know of" appears to be filled with books that are copyrighted and for-profit so I'm not sure you'd really want to publicize what you're doing on your blog.

lince3• 14 years ago

In my country there is no problem in sharing copyrighted content as long as do not receive economical profit. Also, I have found the example very practical as introduction.

smoyer3• 14 years ago

"In my country there is no problem in sharing copyrighted ..."

I guess that means that each e-book only has to be sold once in your country? The U.S. laws might be overly protective of IP, but that's an interesting problem for publishers who in theory need to earn a profit if they're going to continue as entities as well as for authors who need to feed their families.

This obviously wasn't a problem when the books were printed on dead trees, because you'd only share copies that had been purchased, and if you're friend was reading your book you no longer had access to it. Curiously, I could rent my copy of a book to you in the U.S. without violating copyright laws.

gillyb4• 14 years ago

Good idea, and thanks for the comment! I removed the website i was scraping from the screenshots. btw - All the old pages on the site i was scraping were broken links anyway. But I could now make small modifications to the script and have it working on a bunch of other sites as well... :)

er354yerty• 14 years ago

Wouldn't Node be helpful here?

gillyb4• 14 years ago

Don't know, I'm not familiar with node that much... ...but it's definitely on my 'todo' list of stuff i need to learn more about :)

petsounds• 14 years ago

https://github.com/chriso/node.io

looking forward to being able to distribute jobs across multiple machines

inDesperateZone• 14 years ago

What is the benefit of using phantomJS in this case? I understand that it is very useful if content is dependant on JS running.

But that doesn't seem to be the case here. With Python I would have used a parser like lxml or BeautifulSoup (and I'm sure there is something comparable for JS) coupled with Requests async methods. That would probably not only end up with shorter and more concise code, but also be a lot faster.

danneu• 14 years ago

The bulk of the code here is establishing the spider behavior, not parsing html or launching a request.

mistercow3• 14 years ago

The script appears to rely on jQuery (which is presumably already included in the pages being scraped in this case). If you're already familiar with using jQuery for DOM manipulation, then using it for scraping is incredibly easy.

One advantage is that it's not always instantly obvious if you'll need JS to execute before you can scrape a page. If you start out with a simply html parser and then find out that you needed the JS to run first, you're going to have to start over. If you start out using phantomJS and then find out that you don't need any of the original JS to run, your script still works.

rb2k_2• 14 years ago

When you say 'rely on jQuery', I think it would be more precise to say that it relies on CSS selectors. Most libraries will actually provide you a way to access an HTML parse tree in that way.

I guess phantomjs is a good a tool as any, but there is really no need to evaluate Javascript for a bit of plain HTTP+HTML parsing.

zdwalter• 14 years ago

phantomJS + CasperJS make crawling easily. I build http://sp.iderman.info to help scratching easier.

veverkap• 14 years ago

You could also use the CasperJS wrapper and have the script automatically download those files for you.

See http://casperjs.org/api.html#casper.download

gillyb4• 14 years ago

Wow! I just looked at the casperjs api, and it looks amazing!! Tons of great utilities to help you work with the DOM. This could be great, since i would like to implement the json wire protocol for phantomjs, and using casperjs will be of great help for that task! :)

malandrew• 14 years ago

If you like PhantomJS, be sure to also check out CasperJS. I use it with jQuery, Underscore and Underscore.string.

I just wish that jQuery had support for XPath style selectors as well. Chainable XPath would be hella sweet.

Web crawling and downloading ebooks with phantomJS

Related Stories