Accelerating Python Libraries with Numba (Part 2)

http://continuum.io/blog/numba_performance

by corinna • 13 years ago

70 23 13 years ago

paddy_m2• 13 years ago

I work on the http://wakari.io where the notebook is hosted. You can run the notebook in your own (free) wakari account.

Here is the direct link to the notebook: https://www.wakari.io/sharing/bundle/aron/Accelerating_Pytho...

Demiurge3• 13 years ago

I would have liked to see PyPy test for the sum function in the comparison.

Update:

Well this is interesting ;D

time python test.py

real 0m11.169s

user 0m11.141s

sys 0m0.022s

time pypy test.py

real 0m0.259s

user 0m0.239s

sys 0m0.018s

cat test.py def python_sum(y): N = len(y) x = y[0] for i in xrange(1,N): x += y[i] return x

python_sum(xrange(100000000))

paddy_m2• 13 years ago

To be clear, you are comparing pure pypy vs pure python?

Demiurge3• 13 years ago

yes

freyrs32• 13 years ago

Yes, it is well known that allocating 100000000 PyObjects is slow.

onalark3• 13 years ago

Hi. I was reluctant to publish any results with PyPy because I am not an expert in working with their code base. If you'd like to look at the GrowCut benchmark being run on my Mac (Intel iCore 7 2.7 GHz) on PyPy vs. Anaconda Python (or even better, download and run it yourself), the gist is [here](https://gist.github.com/ahmadia/5638980).

Ihmahr3• 13 years ago

WHY didn't they use the Python sum and reduce built-ins for sum??

jofer• 13 years ago

Because it wouldn't be fair to test python's built-in sum on a numpy array (it would be very slow compared to numpy.sum). Numba is mostly about optimizing operations on numpy arrays.

kingkilr3• 13 years ago

Why would it be slow?

hogu• 13 years ago

the blog has been updated with some pypy benchmarks added (just as a gist)

from the blog:

https://gist.github.com/ahmadia/5638980

Update:

At the request of several commenters, here is a test script and benchmarks that we ran on PyPy and Anaconda Python (with Numba). The results are not tuned (I am not a PyPy expert!) so we did not post them in the blog. We’d be happy to look deeper into this with the PyPy developers. While PyPy is not currently installed on Wakari, we are looking at a number of ways we can install and support the PyPy community.

skierscott• 13 years ago

Numba and Cython are faster than C? In my experience, C is the fastest, and the baseline. Was the C compiled with the right optimization flags?

It'd be interesting to see a comparison to Matlab's JIT. Is Numba competitive?

onalark3• 13 years ago

Numba and Cython are faster (for very small kernels) than Python interfacing into a C kernel via ctypes. This is not the same thing as a C kernel as part of a C application. I hope I didn't falsely give that impression! You can see that for medium-size and above kernels, the call overhead makes very little impact on the code execution speed.

The C was compiled with the same flags used to compile Python. In this case: -O2 -g.

I don't have access to a MATLAB license to compare, but I would love to see this comparison done. Let me know if you need any help putting it together.

Accelerating Python Libraries with Numba (Part 2)

Related Stories