Pythran: a Python to C++ compiler with a focus on scientific computing

by vmorgulis • 10 years ago

107 64 10 years ago

m_mueller6• 10 years ago

What is the advantage of doing this compared to writing Fortran (90+) kernels, use F2Py bindings and combining them with NumPy? For HPC you generally need to be able to have full control over compiler flags. Plus, Fortran already has way better multi-dim array handling built-in - NumPy is basically just a wrapper around that.

greglindahl• 10 years ago

I'm a huge Fortran fan, but even I can see that reducing the number of languages required to get high performance can be a good thing. Not all of my science colleagues are interested in learning a ton of languages.

m_mueller6• 10 years ago

Well, most of the scientific computing in Python is happening on NumPy. Which uses Fortran compatible data structures. I doubt that a C++ backed python extension will be compatible with that. For that reason, if anything, I'd advocate to transpile to Fortran rather than C++ if you want more comfort for scientific users.

shoyer5• 10 years ago

Plus, Fortran already has way better multi-dim array handling built-in - NumPy is basically just a wrapper around that.

I don't think either of these statements is true.

Fortran is hard to beat for performance, but from a usability perspective NumPy (and other Python libraries) has some nice features for array-oriented computation that you can't find even in modern Fortran. The most obvious examples are broadcasting [1] and advanced indexing [2].

NumPy itself contains only C, though some SciPy routines wrap Fortran code. Yes, NumPy drew heavy inspiration from Fortran, but it's hardly a wrapper.

[1] http://docs.scipy.org/doc/numpy/user/basics.broadcasting.htm... [2] http://docs.scipy.org/doc/numpy/reference/arrays.indexing.ht...

m_mueller6• 10 years ago

I guess by "indexing" you mean what NumPy calls "advanced indexing". All the basic stuff seems lifted straight from Fortran. But yes, I should probably have been more exact with my statement - NumPy sure has some more helpful syntax that saves some LOC.

vonnik• 10 years ago

Reminds me of JavaCPP: https://github.com/bytedeco/javacpp

akandiah• 10 years ago

The problem with a lot of these optimizing tools for Python is that they only support Python 2.7 not Python 3.

njbooher• 10 years ago

Not really a problem here as most of science still seems to be using Python 2.

edit: Cython was always great for Python 2 and apparently supports 3. http://cython.org

ericfrederich• 10 years ago

Yeah. I came across C library that used Cython to create Python bindings. It was funny, they could easily support both Python 2 and Python 3 at the same time since all their binding code was in Cython, it was only their installation script witch used an incompatibility (print statement)

shoyer5• 10 years ago

Numba also supports Python 3.

Demiurge• 10 years ago

As a happy Python user for 10 years, I didn't even wonder whether it supports python3. Maybe the problem is somewhere else, and not with the library I'm looking forward to trying?

nerdponx5• 10 years ago

Are you insinuating that the problem is with Python 3? What is actually so much more amazing about Python 2 that you would rather freeze the language than upgrade?

bmer3• 10 years ago

I use Python for my research work...and this has never been a problem for me, I suppose because I use Python 2.7, and I guess I am "part of the problem"?

frozenport• 10 years ago

Yes you are.

chestervonwinch• 10 years ago

Same - grad student. I use python / numpy / scipy / matplotlib / sklearn daily for one-offs, proofs-of-concept, etc. There hasn't been any real incentive for me to switch.

mrweasel• 10 years ago

I never looked into, but I wonder if there's something in Python 3 that makes it hard to do these kind of things to, compared to Python 2.7.

Python 3 is the platform I use for almost everything, going back to 2.7 is always a major pain. 10 minutes into some 2.7 work and you have to deal with encoding and you start wondering if it would be quicker to just upgrade to 3.5.

Of cause if you mainly do numbers, there's perhaps little gained from Python 3.

Arkanosis• 10 years ago

Can't talk for Pythran devs and others, but in the case of shedskin, one of the issues is the removal of the “compiler” module in Python 3, on which shedskin relies to, well, parse the Python code. This, plus some syntax changes, that are not really bigger that some that were introduced between, say, 2.6 and 2.7, but that are mandatory and breaking backward compatibility.

So, nothing “hard”, really, you just have to do the work again (and prepare to support two versions for a long time, instead of virtually only one).

nerdponx5• 10 years ago

What does this do that I can't already do with some combination of Numpy, Cython, and Numba?

bmer3• 10 years ago

I use numba quite a bit, and with every new version, the developers have notes like:

PR #1970: Optimize the power operator with a static exponent. PR #1961: Support printing constant strings. PR #1927: Support np.linalg.lstsq.

My point being that the developers have to go about re-implementing (in LLVM IR?) features that are very mature in C libraries (e.g. numpy) and similarly C++ libraries.

Could it be that one gain of going from Python to C++ will be that one could take advantage of all the mature libraries that C++ has access to, without having to do this re-inventing of the wheel?

I am not sure, and regardless, I'll be using numba for a long time coming, because...I like the devs, they work hard on it, and numba works like a charm for what I need to do, so I'm not looking for an overkill.

nerdponx5• 10 years ago

That makes sense.

Numba aside, though, it's strange to me that I see no mention of Cython on their project page. They ought to at least acknowledge that other fairly mature products exist in this space.

Also, maybe it's more of a curiosity for you than a practical tool, but Stan (which is becoming the de-facto standard DSL for Bayesian statistics) compiles to C++, where it takes advantage of libraries like Eigen and Boost.

_ZeD_• 10 years ago

What's the difference between this and nuitka?

dagw3• 10 years ago

The big one seems to be that pythran is targeting only a subset of python with the goal of getting faster performance, while Nuitka is aims at being able to compile all valid python programs (currently only up to python 3.4), but in doing so it has fewer opportunities for optimization and thus gives you slower performance than pythran. Pythran also introduces its own type annotation system you have to use if you want to get good performance.

So If you write a python program that is compatible with pythran, then pythran will almost certainly be much faster than nuitka. If however you take a random python program it will almost certainly compile with Nuitka and almost certainly not compile with pythran.

jsjsjsjsjsjs• 10 years ago

No idea why they could not contribute some patches to nuitka to make it do exactly that. Because further down the road it is the goal of nuitka - to natively compile as much as possible.

guyzmo• 10 years ago

Wel, too bad they did not implement this with python 3, and actually gave some meaning to the type annotations… I mean

def dprod(l0: list[int], l1: list[int]): return sum(x * y for x, y in zip(l0, l1))

Wouldn't it be much nicer?

srean3• 10 years ago

To be brutally honest, they probably couldn't give a flying fuck, I couldn't when I tried. (sorry)

To the number crunchers, Python 3.* offers no compelling reason to move (except for threats, "you are on your own now, 2.7.* will not get updates"). I see two compelling advantages that 3.* offers (i) better abstractions for asynchrony (ii) UTF strings (that exacts a cost even if you don't need them). If you don't need these two there I see no technical reason to move (political/social reasons yes, technical not really).

module0000• 10 years ago

For what it's worth, Python 3 does offer at least 1 compelling reason to move. Multiprocessing lib in python 3 is significantly faster than python 2. My use case is spawning 30-35 worker processes managed by a pool, and recycling them after 5 workloads pass through. The switch from 2.x to 3.x without any code changes, increased my throughput ~150%. Your mileage may vary.

edit: typo

leaningtower• 10 years ago

As I once discussed with a friend of mine, once you remove semicolons, and give spacing a syntactic and semantic value, you almost have FORTRAN... this is just the icing on the cake :¬)

nhamausi• 10 years ago

Wondering how does it compare to Cython?

hairy_man674• 10 years ago

So we have CPython which has an ffi and a native implementation of the source interpreter.

We have PyPy which is JIT for Python that emits assembly at runtime though not as successful Cpython for its support.

The complexity class of this problem, without type annotations, namely completely transforming an arbitary Dynamic Python program to a static, partially inferred typed language (C++) is O(N): https://en.m.wikipedia.org/wiki/Hindley–Milner_type_system

The transformations are also not guaranteed to improve runtime performance and might even do the opposite or have side effects for precision and a program that is slower due to generalizations that an algorithm makes.

With type annotations... why, just why? Use C++ or C if you want types and performance gains thereof..nThe potential for bugs and failures is less than relying on an algorithm if you realise the fear and laziness of compiled languages is umfounded.

toolslive2• 10 years ago

(updated, see comment below) There used to be another python to c++ compiler called "shed skin" it did quite a literal translation, fe python classes would turn into c++ classes and it was quite readable.....

but utterly useless: a python integer would be translated into a c++ int. Where python automatically starts using a wider type this one would just wrap.

dagw3• 10 years ago

Are you sure you're not thinking of some other project? As far as I recall unladen swallow always targeted LLVM and was a JIT compiler rather than a AoT compiler.

toolslive2• 10 years ago

you are right! it was "shed skin".

sitkack• 10 years ago

Compare to https://github.com/shedskin/shedskin which does whole program type flow analysis. Shedskin offers excellent performance on numerical workloads.

madsohm• 10 years ago

We do something similar with Bohrium[1]. Bohrium intercepts your NumPy calls and compile a kernel for either CPU, Multicore CPU, or GPU and run that instead. It will also fuse together all NumPy calls until one has a side-effect, that the run-time needs.

We are currently building smart filters into it as well, so that we can do byte-code optimizations.

[1] https://github.com/bh107/bohrium

Pythran: a Python to C++ compiler with a focus on scientific computing

Related Stories