Pybind11 — Seamless operability between C++11 and Python

by tony • 9 years ago

135 117 9 years ago

fulafel8• 9 years ago

I like how the first basic example invokes undefined behaviour without mention:

  int add(int i, int j) {
      return i + j;
  }

devrandomguy2• 9 years ago

Looks fine to me. Are you supposed to check for integer overflow before doing the addition, or something?

nikbackm• 9 years ago

Probably. Signed integer overflow is undefined behaviour in C and C++.

pjmlp2• 9 years ago

Yep,

    add(INT64_MAX, 1)

Is undefined.

If using Ada, similar kind of code would produce "raised CONSTRAINT_ERROR : program.adb:7 overflow check failed"

To be fair, Ada's behavior is easy to reproduce with a checked integer class like https://accu.org/index.php/journals/324

coldtea4• 9 years ago

I like it how contrived complaints can get...

aoloe• 9 years ago

what about opening a ticket / doing a pull request with a better trivial example that does not "teach" bad habits?

geofft16• 9 years ago

A simple fix is using unsigned integers, as unsigned overflow is well-defined.

(There isn't a way in C/C++ to request an exception or fault on integer overflow, which is what you'd want here, since you're running inside Python. A better conceptual fix is just doing the addition in Python, which is a more robust language than C/C++ for signed integer addition, but that kind of defeats the point of the example.)

louiz• 9 years ago

Just like “operator+(int, int)” invokes undefined behaviour without mention: you can write “a + b” and it will break if the addition overflows!

Or std::vector<T>::operator[] invokes undefined behaviour if the argument is invalid (out of band).

Or printf() invokes undefined behaviour if an argument is of the wrong type (for example "%s" with a corresponding int).

Or strcpy invokes undefined behaviour if you give it invalid pointers (or non-null terminated strings, etc).

So, yeah, many functions can invoke undefined behaviour when they are called with bad arguments, that doesn’t make them bad functions.

fulafel8• 9 years ago

The difference is that this was a function exposed to Python. The expectation of Python programmers is that calling into the module won't explode.

The Python vs C++ joke here is that the minimal a + b example turns out to be unsafe to call with random numbers.

e12e• 9 years ago

I for one am grateful for this comment, I program rarely enough in C/c++ that I wouldn't have assumed signed overflow was undefined. As the example below with "int a; a+5>a" illustrates, that's potentially pretty serious.

I wonder if the "best" fix here, would be to use assembler intrinsics to maintain performance, possibly with a fallback to a type cast/promotion to a bigger int for the sum, and some form of defined behaviour for overflow on the python side? (reinventing the python number tower with full automatic promotion seems counter-productive here, as presumably the idea is to work with fast/simple/native (signed) integer math...)

Or just redefine the function to use unsigned ints.

ivan_ah• 9 years ago

other projects in that space are: cython (need to manually do wrapping), xdress, SWIG, SIP, clif.

Does anyone have experience with using these and able to compare/contrast?

I'm afraid of C++, but if I can wrap it and learn how to use from a python REPL, I think I can handle... any recommendations/tuotirals/howtos would be much appreciated. (The library I want to wrap is https://github.com/openzim/libzim )

first_amendment19• 9 years ago

I've used cython, swig, and boost.python. I prefer pybind11 to all of them. It requires the least extra tooling and allows you to seamlessly transition between C++ and Python using the idioms native to each language.

ThePhysicist4• 9 years ago

I built bindings for various C++ libraries in SWIG (not only for Python but also Lua and Ruby) and I was always very impressed with the result. What's amazing about SWIG is its ability to easily port very complex and rich code. I successfully exported very complex class hierarchies and code relying heavily on pointers and STL to Python and/or Lua. The process consists of writing several header-like configuration files that SWIG ingests and uses to generate bindings, which is often pretty straightforward. Often you can simply import the normal C/C++ header files in those configs and add some glue code / extra hints to help SWIG in cases where it can't figure out what to do with a particular type.

In my experience, the most difficult aspect of binding generation is not writing the glue code, but doing so in a scalable way. Manually writing bindings for any real-world C++ codebase would therefore be extremely tedious (IMHO), so having an automated system that does this work for you is a huge time saver.

Some larger libraries / frameworks have their own wrapping generators Btw, PyQt (which provides extremely good bindings to QT) for example has SIP which is worth looking at (it's open source).

piokuc• 9 years ago

What's also important is to have a simple, neat API, ideally C API. If you do have it, then writing wrappers for any language, even manually, without SWIG (which is great, but does add complexity to your project's build process), is simple.

brational2• 9 years ago

Have you used pybind? I've used SWIG extensively so I'm familiar with that - you talked about SWIG but not how it compares to this new one.

Reading through some docs, it looks like pybind suits better for a python codebase that occasionally needs C++ pieces added in as you have to setup the C++ code to handle the py needs.

SWIG I was always impressed with exactly your point. You could sometimes just make 1 big header file in a facade pattern to expose some "start" buttons needed to run a huge c++ application and nothing else needed to be setup. No going back and forth.

Example usage here was a single heavy duty image processing application but using python has the distributor and work manager. C++ code was a "worker" and python and zeroMQ did everything else. So the only "connection" we needed was a start button and some metadata to the c++. Maybe 1 header file with 100-200 lines of code to make the bridge for SWIG.

(to anyone else reading) does that sound trivially simple to the do the same with pybind?

BeetleB• 9 years ago

Swig is great when it works, a nightmare when it doesn't. A coworker in a former job used Swig to enable a C library to be used in Python.

Initially, all was great. But then we discovered a memory leak. He tried for weeks to identify it, but never could.

Cython would be my first choice.

tehsauce2• 9 years ago

I used pybind11 for my open source asteroid detection code. It's very slick, especially it's ability to seamlessly convert between stl and python lists- for example a vector of strings converts directly to a python list of strings. Also the interoperability with numpy. A class containing an stl of primitive types and the dimensions can easily be turned into a numpy array. Very neat

tehsauce2• 9 years ago

https://github.com/DiracInstitute/kbmod

The project I was referring to :)

unkown-unknowns• 9 years ago

Also, Boost.Python. http://www.boost.org/doc/libs/1_64_0/libs/python/doc/html/in...

A proprietary piece of software we use at work has a Python API for which they are using Boost.Python.

I've written a piece of software to get some data from the API and to analyze the retrieved data. The Python API has a bit of a "C++ smell" to it and doesn't feel completely pythonic but it's good enough. The bigger problem is that the vendor's idea of documentation is a single PDF generated from the class definitions and everything is either very sparesly or not at all commented, heh.

aldanor5• 9 years ago

In fact, pybind11 was heavily influenced by Boost.Python, to the point where some of the syntax may look quite similar. It has one obvious advantage though, in that it doesn't depend on Boost :)

jcelerier• 9 years ago

Here's how the pybind11 bindings of a c++ lib I work on look:

https://github.com/OSSIA/libossia/blob/master/OSSIA/ossia-py...

as you can see, it's just a matter of declaring the classes and methods you want to wrap. It was very straightforward and works extremely well.

frumiousirc• 9 years ago

ROOT https://root.cern.ch/ provides Python bindings through its PyROOT functionality. It builds the bindings (ROOT calls them "dictionaries") by parsing your C++ using their "cling" interpreter based on LLVM. As far as usability, ROOT is a big package but once a given it's rather easy and non-intrusive to create Python bindings for your C++.

autopoiesis4• 9 years ago

Also as teajunky notes below, cppyy (which was born out of PyROOT) is quite nice: http://cppyy.readthedocs.io/en/latest/

xioxox• 9 years ago

I've used SIP, which wasn't too painful. The code you write to do the interfacing looks quite like the C++ definitions. I mostly used it as was an application which used PyQt. It probably is harder to use as it's not the most popular option.

teajunky• 9 years ago

Has anyone looked at https://pypi.python.org/pypi/cppyy ? This seems to be even easier than pybind11

aldanor5• 9 years ago

Some may find it useful: I've written a Jupyter notebook extension for pybind11 a while ago, so you can write C++ in the notebook and it gets automagically compiled and imported into the kernel (plus you get C++ syntax highlighting and a few other goodies):

https://github.com/aldanor/ipybind

(still not on pypi but the plan is to release it soon)

nmalaguti• 9 years ago

At what point should I start thinking about rewriting computationally expensive parts of my Python application in C++ and making bindings?

Does anyone have experience taking one expensive method and replacing it with C/C++? Were the trade offs worth it?

aldanor5• 9 years ago

A typical use case is pushing loops in the hot path of your code to the C++ space. A simple loop that does nothing like `for i in range(int(1e9)): pass` takes about 20 seconds to execute in Python on my machine, whereas in C++ the overhead would be thousands times smaller.

Because there's also overhead for transferring objects to/from pybind11 (it has to keep track of object lifetimes, figure out the conversions, etc), it's generally more beneficial to wrap big chunks of logic in C++ rather than every single method.

aldanor5• 9 years ago

For the sake of illustration, there's a simple but realistic example at the very end of my talk at this year's EuroPython:

https://speakerdeck.com/aldanor/pybind11-seamless-operabilit...

(reimplementing rolling_stats from pandas that ends up being faster)

chombier• 9 years ago

The easiest would be to first give cython a try, as it can provide a nice performance boost at a minimum effort in some cases.

If you just need to bind simple number crunching functions, you may also write a small library exposing them with c linkage, then access them using ctypes.

For more advanced usage, you will probably benefit from using a binding framework instead of using the python c api.

sametmax2• 9 years ago

First, try to optimize your Python. It's unexpected what you can do with it. E.G: slicing assignment is crazy fast, removing calls and lookups goes a long way, using built-ins + generators + @lru_cache + slicing wins a lot, etc. Also Python 3.6 is faster, so upgrading is nice.

Then, you try pypy. It may very well run your code 10 times faster with no work on your part.

If you can't or the result is not as good as expected, you can start using EXISTING compiled extensions. numpy, uvloop, ujson, etc.

After that, and only now, should you think about a rewrite. Numba for number related code, cython for classic hot paths or nuikta for the entire app: they can turn Python code into compiled code, and will prevent a rewrite in a low level tech.

If all of that failed, congratulation, you are part of the 0.00001%.

So rewriting bottlenecks in a compiled language is a good option. C or C++ will do. But remember you can do it in Rust too!

yig• 9 years ago

I love Python, but this is one of the pain points. Working through a sequence of domain specific languages when you could have just written it in a fast one to begin with (e.g. Julia or C++).

dr_zoidberg2• 9 years ago

I'd remove PyPy. It's not 100% compatible and there are antipatterns in between them. For some time I've treated CPython and PyPy as two very similar, but different languages. PyPy is more C-ish if you want to call it that way, what's fast in it is more direct, similar to the C mindset. In Python a good abstraction usually will give you better performance. If you mix them, it's not quite one thing or the other.

w8rbt• 9 years ago

I wrote a demuxer for CAN Bus data in C++. Then, I built a Python module using Boost.python so that our Python programmers could simply 'import demux' and write scripts to see and manipulate individual variables, load data into the DB, etc.

The productivity gains of Python scripts that could demux were huge. It made demuxing fast and easily accessible to all of the Python coders. Also, the underlying C++ code could be used in our C# environment too, although I'm not sure if they ever did that. So, we had the ability to use the exact same underlying C++ code in multiple dev environments to ensure consistency.

The downside, IMO, was maintaining the modules. The C++ code itself was short and easy to test (maybe 100 or 200 LOC), but you need good documentation and the ability to build the modules for various versions of Python, with various toolchains, on various systems, etc.

I would do it again if in the same or similar situation.

geofft16• 9 years ago

This doesn't directly answer your question, but I think a better use case for pybind11 is when you have an existing C++ library with a fairly rich typesystem and you want to expose it to Python.

If you just want to reimplement some parts of your code in C for performance (I'd argue you neither need nor want C++, you shouldn't bother with C++'s object system and you should keep using Python)'s, CFFI might be simpler:

https://cffi.readthedocs.io/en/latest/overview.html#purely-f...

KKKKkkkk1• 9 years ago

I've been using boost::python, and the experience is what I'd call cargo-cult programming. You copy and paste stuff off the tutorial, and when you get a compiler error, you just try other stuff at random, because the inner workings of the library are inscrutable. I vastly prefer cython to this. Is Pybind11 any better?

autopoiesis4• 9 years ago

Pybind is derived from boost::python, so it's not much different. I wrote a large Python binding in the past for a moving C++ library target, using boost::python, and keeping the C++ binding code up to date was a nightmare akin to maintaining a fork of any fast-moving project.

Try cppyy [1]. It's very nice, though quite fresh. It's used extensively (as far as I can tell) at CERN, and derives from their cling C++ JIT interpreter. Plus it does lots of nice automatic transformations to make a C++ API more pythonic.

[1] http://cppyy.readthedocs.io/en/latest/index.html

wjakob2• 9 years ago

Not quite -- while pybind11 was originally inspired by boost::python, it is an entirely different project that is intended to address many of its problems.

zwieback• 9 years ago

I'm curious - are you more of a Python or C++ programmer? C++ compiler errors can be famously cryptic so I'm wondering if these types of binding-generators are mostly suitable for C++ programmers that want to expose their libs to Python and don't mind decoding template errors in their own code.

I've done both C++ and Python and never attempted to generate bindings, in the near future I may be faced with this though.

wjakob2• 9 years ago

pybind11 shares some of the motivation of boost::python, but it is designed to be considerably easier to use. Out of the box, it is aware of a large set of C++ types that are automatically translated to their Python equivalents and vice versa (STL data structures, C++17 types like optional<> and variant<>, std::function<> for callbacks, and even sparse and dense matrices for numerical linear algebra).

@aldanor recently gave a nice overview talk of what pybind11 can do: https://speakerdeck.com/aldanor/pybind11-seamless-operabilit...

Also, take a look at our documentation (http://pybind11.readthedocs.io) -- it contains detailed examples of many tricky use cases.

arximboldi• 9 years ago

I have used boost::python a lot and later tried pybind11. A lot of people prise it because they consider boost bloated, but in my experiments boost::python is actually 2X faster at runtime (not at compile time). I think I'd rather use boost::python.

Pybind11 — Seamless operability between C++11 and Python

Related Stories