Phantompy - Headless WebKit engine for Python, like phantomjs

by juanriaza • 13 years ago

171 45 13 years ago

neo2001• 13 years ago

Looks phantastic

boothead• 13 years ago

Is there anything wrong with the python bindings to phantom? I used phantomjs (it's great)a few years ago and I thought there were some python bindings to the original phantom C++ library - could be wrong though.

waitwhatwhoa• 13 years ago

Ariya Hidayat & co have put a monumental amount of effort into exposing the inner workings of webkit in phantomjs - I'm not sure I see the necessity for a python rewrite. Taking a quick look at the code, most everything I use phantomjs for outside of screenshotting is not available. A great start for sure, but unless there are some fundamental shortcomings with phantomjs that I'm unaware of, I agree with boothead that python bindings are probably the way to go.

heydenberk• 13 years ago

One thing that jumps out at me is that phantompy uses Qt5 internally, while phantomjs uses Qt4. There are lots of new WebKit features and optimizations in Qt5.

Herald_MJ• 13 years ago

There were a set of Python bindings for phantom.js named PyPhantomjs, and it was included in actual phantom releases, but the maintainer moved away from the project, no-one stepped forward and it was ultimately removed from the project (https://github.com/ariya/phantomjs/issues/10344).

So this project is really useful.

kanzure2• 13 years ago

> no-one stepped forward and it was ultimately removed from the project

Here's the source code, if you are interested:

https://github.com/kanzure/pyphantomjs

woadwarrior01• 13 years ago

Nice. There's also https://github.com/jeanphix/Ghost.py which I remember using a couple of months ago.

killahpriest3• 13 years ago

Ghost.py seems promising but is painful to work with. Installing the dependencies took about an hour (compiling PyQt itself took about 25 minutes). When we did finally get it working, PyQt would crash after the third or fourth scrape and would take between 15 to 30 seconds per scrape. Where we used it: https://github.com/createch/PriceChecker.py/blob/master/pych....

My advice to anybody looking to do headless webkit in Python: Don't use Ghost. Try out PhantomPy and try using the Selenium webdriver for Phantom.

We switched over the Selenium webdriver for PhantomJS and found it to be much more stable and fast. See this SO answer by another Ghost.py user who gave up: http://stackoverflow.com/a/15699761/854025. Where we used the Selenium driver: https://github.com/createch/PriceChecker.py/blob/phantom/pyc....

arikfr2• 13 years ago

Just to give another perspective: I'm not sure on which OS you had those issues, but I'm successfully using Ghost.py on Ubuntu and OS/X. On OS/X it was harder to install PyQT and sometimes there were weird issues, but on Ubuntu it all went smooth.

By now I've ran it on thousands of different websites with little issues. It's true that I'm "recycling" the processes after ~10 sites, but that's mainly because of memory leaks.

EDIT: and to clarify on Ubuntu installing either PyQT or PySide (Ghost.py now supports both) was as easy as apt-get install...

acron02• 13 years ago

For those of us on the outside, what's the typical use cases of a headless browser engine? Testing?

juanriaza• 13 years ago

And web scraping

meerita3• 13 years ago

We use it to generate images from a website. We do a product with HTML5/CSS3 and webfonts, layout it all then "screenshooting it" to make a final image with the proportions we want. It's cool because its server side and we can generate endless images based on what the user want.

Addendum: If you want to use webfonts, specially from Google, you need to recompile Phantom to support WOFF.

acron02• 13 years ago

I was going to ask about off-screen rendering. What kind of speeds are achievable? 30fps? Could I use this to stream a web texture into a game?

DenisM2• 13 years ago

Can you capture PDF that way, or is it only png?

elmomalmo• 13 years ago

Running JavaScript unit tests headlessly in CI.

yashodhan• 13 years ago

I use phantomjs to login to my bank's website and grab my balance for me.

Unfortunately my bank decided to outsource their web operation to the cheapest bidder from India and as a result their web application is beyond shit. It requires javascript to work, and therefore I needed a headless broswer engine to scrape it.

justincormack• 13 years ago

All the banks I have used in the last 10 years have required hardware tokens. Recommend you change banks even if that makes screen scraping your balance impossible....

tlarkworthy• 13 years ago

angularjs and other JS rendering techniques make the plain HTML look a mess. Google will not be able to present the page or get its content properly, so you use something like this to take snapshots of your website post rendering and expose it for indexing & website previews. (https://developers.google.com/webmasters/ajax-crawling/docs/...)

rjh292• 13 years ago

I would be surprised if Google was not doing this themselves to some extent.

saryant• 13 years ago

I recently built a service using PhantomJS for a client. They have graphs and charts generated with d3.js that they wanted to add to their user-specific update emails.

I built a backend service using Akka[1] and PhantomJS which would grab the appropriate user's page, render it to HTML, splice up the images and save everything to temp files. Then I had a Java process that read those back in then format and sent an email to the user.

[1] http://akka.io

hackerboos• 13 years ago

Cool you can take screenshots with it, see the test_capture_page example.

shadowmint• 13 years ago

OT I know, I cant help but feel a little smug when I see cool projects like this and they use cmake, and not autotools.

Thank you cmake, you've made working in the C ecosystem fun again.

greggman• 13 years ago

Check out gyp and ninja

https://code.google.com/p/gyp/wiki/GypVsCMake

https://github.com/martine/ninja

randallu• 13 years ago

CMake is pretty horrifically inefficient at compiling WebKit. The makefiles it generates fork to call CMake (to do things like printf in color) far more than they fork to call the compiler. The ninja generator is better, but it has various issues with long command lines on WebKit currently (trunk CMake fixes some of these, but then it doesn't make some directories in the output directory).

Also, the CMakeLists syntax is pretty nasty IMO, but it feels like gyp, cmake and autotools are in some kind of syntax ugliness competition... (Maybe because nobody ever wants to work on a build system, so they'll just do this one little hack which inevitably grows tentacles).

_seininn• 13 years ago

I've been working on a project like this that's nearing completion (it works, and the api is mostly complete). It differs from this in that it uses any normal browser to fulfill requests (via extensions).

The idea is to have a pythonic API to any browser that has reasonable support for extensions; i.e. a cross-browser scripting.

laurencerowe• 13 years ago

What does it offer over Selenium Webdriver? Yeah, the API is not terribly pythonic, but it's battle tested and stable, works across almost all browsers and there's always wrappers like Splinter (http://splinter.cobrateam.info/) if you prefer a more pythonic interface.

fauigerzigerk• 13 years ago

I wonder if Phantompy is truly headless so it works on servers without GUI libraries and without xvfb.

QtWebKit needs access to some GUI features out of the box, but phantomjs includes platform abstraction code that avoids the dependency on an X server. That's the feature that made me migrate my own C++ crawler code to phantomjs.

kanzure2• 13 years ago

Actually, you don't even need xvfb these days (and I have no idea why anyone is telling people that they do). You could use something like xserver-xorg-video-dummy. Of course, as you have pointed out, PhantomJS doesn't require an X server (as opposed to these other projects, wtf) so that it is "really really truly headless".

druska• 13 years ago

Will this render Flash content?

salimmadjd• 13 years ago

OT - I've been looking for a headless WebKit for Go (Golang) I've searched all over with no luck. If anyone wants to become famous, please make one :)

rkrzr• 13 years ago

Does this actually offer full DOM access?

I see a 'cssselect' in tests.py, but I wonder how much of the DOM Api is actually supported?

jessedhillon• 13 years ago

Take a look at lib/webelement.hpp -- there's not too much yet, notably absent is a way to iterate children. But you can evaluate Javascript.

encoderer• 13 years ago

There are so many half-baked headless browser projects, many building in some way on PyQT. I've made heavy use of Spynner and Ghost. Spynner especially. It always seemed to offer exactly what I needed but the inconsistent docs and endlessly-deep stack always seemed to leave successful implementation just beyond my grasp.

Now, as an OSS maintainer myself, I sympathize. But I'd love it if this or another one of these projects started breaking away from the pack and could build some critical mass. I'm all for democratization but in this case I'd gladly trade 10 mostly working projects for a single solution ran by an iron-fisted BDFL if it meant decent docs and a mostly stable API.

Phantompy - Headless WebKit engine for Python, like phantomjs

Related Stories