Python 3 comes to Scrapy

https://blog.scrapinghub.com/2016/02/04/python-3-support-with-scrapy-1-1rc1/

by ddebernardy • 10 years ago

249 74 10 years ago

blog.scrapinghub.com

djm_• 10 years ago

Many congrats on the release & also thanks to the Scrapy team for the effort involved.

As far as I am concerned, this was the last package I used heavily that still had not made the upgrade.

For the Python community: which packages are you still waiting for/working on?

crdoconnor• 10 years ago

Mechanize would be nice, but it's practically abandonware at this point.

kevinwang• 10 years ago

same

random_upvoter• 10 years ago

A wrapper for the exiv2 library (for editing metadata of picture files) that works on Windows

vially• 10 years ago

I've been waiting for Fabric to make the migration to Python 3. Fortunately I no longer use Python that much (I jumped on the hype train to Go and I've been very satisfied with it) so I no longer need Fabric nowadays.

xemoka• 10 years ago

Esri's ArcPy module and ArcGIS Desktop's "built-in" python environment. :(

ArcGIS Pro is using 3.4, so the wrappers for ArcObjects are updated, but who knows if they will put that work back into ArcGIS Desktop...

craigds• 10 years ago

Still waiting on Ansible to make the jump. As far as I can tell it's not even on their roadmap, which is a bit crazy given that _almost_ everyone else has solved this by now.

This may not be a blocker for the rest of our stuff though, since ansible can install its own python 2 on servers automatically.

bbayer2• 10 years ago

It is a major change to ask but I am wondering if they consider to switch to asyncio instead of using Twisted. Twisted is great library but it is a huge dependency to maintain.

kmike847• 10 years ago

We're seriously considering this option (maybe for scrapy 2.0?), but no concrete plans yet. It can't be just asyncio - we'll need e.g. aiohttp and other packages.

Some links:

* POC for aiohttp as a http handler: https://github.com/scrapy/scrapy/pull/1455

* some thought about how to make async/await API for Scrapy; it is not all roses: https://github.com/scrapy/scrapy/issues/1144#issuecomment-14...

eliasdorneles• 10 years ago

There has been some discussions and a proof-of-concept with asyncio + aiohttp: https://github.com/scrapy/scrapy/pull/1455

However, as you said, it'd be a major change and it would affect the whole ecosystem (plugins and extensions), so it's complicated. We'll see what happens. :)

glyph• 10 years ago

Can you elaborate? What makes it a "huge dependency to maintain"? Is there anything that the Twisted project can do to make it easier? If this is actually a problem I'd really like to hear from users on the Twisted mailing list and bug tracker.

bbayer2• 10 years ago

Twisted is a general purpose library/framework with lots of features. This is the "huge" part. In my previous projects I have used it a lot and appreciated it.

What I was trying to tell is if Scrapy uses only small part of library, it may be possible for developers to use similar constructs from Python's standard library. In any case dependency is dependency and it is always better to minimize code footprint.

engi_nerd• 10 years ago

I've seen developers refuse to use Python 3 because of not being able to use Scrapy. Hopefully this gets some more devs to finally make the switch.

squeaky-clean• 10 years ago

I've been refusing to use Scrapy because of not being able to use Python 3 :P So this is a very exciting announcement for me.

ddebernardy• 10 years ago

Believe me, we've been hearing this too. We're super excited as well. :-)

jstoiko• 10 years ago

This is great news for Python 3. We're almost there! -> http://py3readiness.org

pen2l2• 10 years ago

Hm, why is opencv not there? I was playing around with it last week, and it seems it works well for only py2 right now (from what I could gather). OpenCV should be there in that list, a lot of people really need it.

rspeer2• 10 years ago

My guess: because OpenCV is so unlikely to be installable through pip, its download statistics on PyPI end up artificially low.

orf2• 10 years ago

It's only the 360 most downloaded packages, perhaps OpenCV is not one of them?

jritchie• 10 years ago

OpenCV 3 supports Python 3[1], but it's not distributed through PyPI.

[1] http://opencv.org/opencv-3-0.html

daveguy2• 10 years ago

Hahahaha. 7 years later. Python 3 is almost there!

rspeer2• 10 years ago

Your response to lots of people's programming effort is "Hahahaha"? Get lost.

jimmaswell• 10 years ago

I want to like Python 3 but the print statement change really kills it for me. Years later it's still frustrating on the occasions I write a python script. If I ran a project like one of those, I might not use Python just for this reason.

orf2• 10 years ago

I really don't see how the print statement can be that bigger deal. After a week at most you're used to it and `print(x)` is valid syntax on Python 2 (so don't worry about what way to write it).

eugenekolo2• 10 years ago

I write Python 2 code, and just always use `print("...")` in both. It makes a lot more sense, matches other languages syntax, and is compatible in both. I honestly don't see it in any way worse than `print "...."`.

theptip• 10 years ago

This is the most trivial of all the changes introduced in Python3.

If you'd said something about string encodings, you might have got sympathy from someone...

kbutler• 10 years ago

I empathize. Makes me want a parenthesis-optional language, though that has its own headaches.

I find switching to log() as an output function helps psychologically - it's shorter to type than print, and it's something I'm more likely to keep in production.

njharman3• 10 years ago

Really?, Really! Because every other output method was a function the statement is annoying as all hell.

Swapping between print and fh.write() log.error(), stringio.write(), sys.stderr.write(), mycustom_thing_that_suppresses_print_when_quiet_arg_supplied() in python 2.x is annoying as all fuck and simple 'ct(' with python 3.x

Keats• 10 years ago

Nice! Everytime I create a new Scrapy project I keep forgetting it was not python3 and had to recreate the virtualenv. Great news!

erroneousfunk• 10 years ago

This is great! I wrote "Web Scraping with Python" (O'Reilly) and did everything with Python 3... except for the oddball section on Scrapy. Glad to know I can update that for the second edition!

bobby_9x• 10 years ago

I'm glad Python 3 supports so many libraries now. I just made the switch a couple of months ago (from 2).

ceronman• 10 years ago

The breakup between Python 2 and 3 has been very slow and painful. Python devs know that, and that's why they won't break compatibility in such big and drastic way ever again.

I'm glad that we're starting to see light at the end of the tunnel. I find myself using Python 3 in most of my projects. Some times I still have to resort to Python 2 when some dependency is not ready but those cases are more rare every day. Also it's frustrating to use Python 2 when many cool features are now Python 3 only. It will take some more time, but I'm sure that the transition will eventually be completed.

thearn4• 10 years ago

I'm happy that we finally updated a lot of our code at work to be Python 3 compatible last year. The conversion was not that bad in the end. However, under Python 3 a number of benchmarks are slower, which we do need to look into some more. Has anyone else experienced this, specifically with scientific computing code migrating from 2 to 3?

Ultimately, we test in 2 and 3 via CI, but still primarily develop using 2.

kmike847• 10 years ago

A data point: `scrapy bench` is about 1.5x - 2x slower in Python 3.5 than in Python 2.7. This command starts a simple HTTP server which serves HTML pages linked in arbitrary depth, and crawls it using Scrapy. Profiler shows URL parsing is much slower in Python 3 for some reason, I'm not sure why. I haven't digged further.

Python 3 code also have to encode/decode strings more often, but the overhead was not significant for `scrapy bench`.

kmike847• 10 years ago

For scientific code I've seen slowdowns, but in the end they turn out to be unrelated to Python 3 - e.g. numpy compiled against different BLAS libraries in Python 2.x and 3.x.

There also was a gotcha with replacing cStringIO with io.BytesIO: the latter copied the data even for read-only cases (when you want to have a file-like interface for a binary object), so it can be much slower.

To make things worse, Python 3.x didn't have cStringIO alternative which can be used to provide file-like interface without overhead, and most porting guides suggest using io.BytesIO instead of cStringIO in Python 3. So libraries like tornado, django and flask are affected, and likely some scientific libraries as well.

This problem should be fixed in Python 3.5: io.BytesIO becomes as efficient as cStringIO for read-only cases since Python 3.5.

bootload2• 10 years ago

"The breakup between Python 2 and 3 has been very slow and painful. Python devs know that, and that's why they won't break compatibility in such big and drastic way ever again."

Fork, don't break, cf Pillow (PIL fork) ~ http://python-pillow.github.io/

What are the most commonly used Py2 packages that need to be Py3?

jacobolus2• 10 years ago

What are the most commonly used Py2 packages that need to be Py3?

https://python3wos.appspot.com http://py3readiness.org

Siecje• 10 years ago

For me it's supervisor and beanstalkc and fabric but you can run your fabric scripts with Python2 easily enough.

brownbat• 10 years ago

PaiMei was a pretty good toolkit, but it doesn't even work with anything after 2.4, at least not out of the box.

So maybe that's not even an example of the 2/3 split, it's just old.

bootload2• 10 years ago

This is great news. I often read the stick developers give to Py3, not wanting to upgrade from 2->3 citing code bases still using Py2.

njharman3• 10 years ago

Minor point of clarification re title; Python 3 has been here, it's Scrapy that has (finally and good for them) come to Python 3.

kmike847• 10 years ago

that's an example of https://en.wikipedia.org/wiki/Galilean_invariance

penetrarthur• 10 years ago

Not related to Scrapy, but what are some things you scrape the web for?

kevin_thibedeau2• 10 years ago

I once scraped every October posting from Slashdot to see long term trends. Short story: it's dying. I project the active userbase will be gone by 2020. Curiously the bulk of the posters were in the 100k to 300k UID range. There was also evidence of shenanigans with UID assignment where they were skipping even numbers and odd numbers at various times possibly to inflate their numbers.

This will get you IP banned BTW but I did get a full data set before their script caught me.

gst• 10 years ago

I wonder if Netcraft already confirmed that Slashdot is dying ;)

hatchoo• 10 years ago

ScrapingHub (the guys behind Scrapy) offers Crawlera which provides some sort of automatic proxying and throttling so you can scrape away avoiding getting banned.

BuckRogers• 10 years ago

I've been holding out for a while on moving to 3. I have tried every 3 release and always found issues or performance regressions. Like others have said here it's a concern about the performance issues, Python is already slow. It's an even harder sell when

- all major (and equally important, the long tail of minor) libraries support 2

- CPython2 has the performance advantage in most (if not all) applications

- virtually every 3rd party implementation supports it very well (PyPy in particular)

- Python2 already supported unicode, so that gets old to hear about

- Most of the new features are available as backports

- Some new features are absurd, like the new 4th string formatting method in 3.6

People shouldn't openly wonder why someone uses 2 instead of 3 if you really look at it.

I just started a new 3.5 project because while I gave 3.0-3.4 a shot, 3.5 hasn't had its runthrough yet. Most people in my shoes have more than likely moved on from Python to Go. I'd like to have this be the one and stop going back to 2.7. Admittedly patience is running on fumes after ~8 years of testing CPython3 releases.

It wasn't just a bad break, it seems like it was a sloppy break. Instead of feature bloat, I'd like to see Python3 focus on performance.

Chris20485• 10 years ago

Also, Jython py3 dev lagged far behind py2.

Chris20485• 10 years ago

I mentioned some of the issues I had with scrapy ages ago on reddit: http://www.reddit.com/r/Python/comments/g112q/installing_and...

Never got a reply, but I'll reproduce here; I wonder how much of this is still true?

" The docs where pretty good, but it was unclear sometimes how to proceed; There was a lot of structure to understand in order to get started.

When I used it, I wanted to scrape a site until certain conditions were met; when the last page scraped returned no objects. I wanted all results initially returned from a page to be dropped if they were older than a certain date; Thus I wanted Scrapy to keep scraping until no new items were found. Also, I wanted the latest date of the items returned so I could use this the next time I scrape.

I created the 'DropElderMiddleware' middleware to do this. I couldn't see any other way of making calculations based on items returned from a particular page.

I could never figure out what the difference between input and output processors where, or when I should use one or the other.

The MapCompose function flattens object by default, So I had to be careful sometimes when returning lists that represented structure I wanted to retain.

The way the html match object worked was sometimes confusing; If I wanted to match multiple items, then match items within each of those, I wanted a list of lists (group matches together base based on what matches they were found in). I can't remember the details of why I found this hard, but I can try to come up with an example if you like?

In the end I figured I was having to learn the structure of Scrapy for everything that I wanted it to do, but many of Scrapy's features I didn't need e.g. I didn't want command-line control (I would actually prefer not to use the interface, though didn't discover how I could write a python script to apply the spider directly).

Now I prefer to use mechanize + PyQuery; PyQuery is at least as good as processing web pages as Scrapy's object, and if I need something more for opening a page e.g. complicated login, I can use mechanize. I find this a more modular approach, and think that I better understand what's going on in my scripts. "

kmike847• 10 years ago

The docs where pretty good, but it was unclear sometimes how to proceed; There was a lot of structure to understand in order to get started.

Yeah, docs used to be a problem; they improved a lot in 1.0 and 1.1 releases though.

When I used it, I wanted to scrape a site until certain conditions were met; when the last page scraped returned no objects. I wanted all results initially returned from a page to be dropped if they were older than a certain date; Thus I wanted Scrapy to keep scraping until no new items were found. Also, I wanted the latest date of the items returned so I could use this the next time I scrape.

The easies way is to raise CloseSpider exception in a callback if no new items are scraped - see http://doc.scrapy.org/en/1.0/topics/exceptions.html#closespi...

I could never figure out what the difference between input and output processors where, or when I should use one or the other.
The MapCompose function flattens object by default, So I had to be careful sometimes when returning lists that represented structure I wanted to retain.

I also have troubles understanding ItemLoader details. They are totally optional though, and they are no longer in Scrapy tutorial (http://doc.scrapy.org/en/latest/intro/tutorial.html). Item loaders provide features very similar to https://github.com/Suor/funcy or https://github.com/kachayev/fn.py.

The way the html match object worked was sometimes confusing; If I wanted to match multiple items, then match items within each of those, I wanted a list of lists (group matches together base based on what matches they were found in). I can't remember the details of why I found this hard, but I can try to come up with an example if you like?

I'm not sure what problems did you have. Scrapy selectors library (https://github.com/scrapy/parsel) is quite similar to PyQuery (esp. when CSS selectors are used), and nothing prevents you from using PyQuery with Scrapy. In future we may add PyQuery (and BeautifulSoup?) support to parsel and provide PyQuery selectors as response.pq (like response.css and response.xpath), +1 to do that.

In the end I figured I was having to learn the structure of Scrapy for everything that I wanted it to do, but many of Scrapy's features I didn't need e.g. I didn't want command-line control (I would actually prefer not to use the interface, though didn't discover how I could write a python script to apply the spider directly).

Yeah, library interface used to be a problem. It was improved in 1.0 release (there is an official API for integrating Scrapy with Twisted apps and running spiders from user scripts), but there is still more to go. See http://doc.scrapy.org/en/1.0/topics/practices.html#run-scrap....

It probably won't be as easy to integrate with regular Python scripts as mechanize because Scrapy is async. On the other hand, Scrapy is easier to integrate with async servers like Twisted or Tornado.

Now I prefer to use mechanize + PyQuery; PyQuery is at least as good as processing web pages as Scrapy's object, and if I need something more for opening a page e.g. complicated login, I can use mechanize. I find this a more modular approach, and think that I better understand what's going on in my scripts.

You may want to check the new 'Scrapy at glance' page (http://doc.scrapy.org/en/latest/intro/overview.html). The main advantage of Scrapy over mechanize is that it handles parallel downloads and have a wide range of built-in extensions you won't have to implement yourselves.

Chris20485• 10 years ago

Hmm, the matching issue might have been something like wanting to do "[i.match(tag='foo') for i in body.match(tag='bar')]" and getting a list-of-lists back, but this was a long time ago :-)

Incidentally, I've since gone off pyQuery as it doesn't always keep up with jquery. I now prefer lxml or BS4..

BTW, I love ScrapingHub. I bashed out a few Spiders with portia, but ultimately, I'll prob start scripting instead. Do you know if portia actually generates script code? Might be easier for fast scraping to get 60% of the ways with portia, then manually write the rest of the script.

One last thing - looking at this page

http://stackoverflow.com/questions/6261714/inferring-templat...

there is mention of a "wrapper induction library"; I can't find anymore mention of it though, does the class/functionality still exist?

darkrho• 10 years ago

Try it out via conda!

  conda install -c scrapinghub/label/dev scrapy

Python 3 comes to Scrapy

Related Stories