Want to call C from Python? Use D

https://atilaoncode.blog/2020/02/19/want-to-call-c-from-python-use-d/

by atilaneves • 6 years ago

193 108 6 years ago

michelpp• 6 years ago

Interesting article, but I wouldn't say introducing a 3rd language is the easiest way to call C code from Python, that distinction goes to the CFFI library by Armin Rigo et al. I've used it to wrap several C libraries and it's amazingly powerful. Bonus it works with CPython and PyPy.

https://cffi.readthedocs.io/en/latest/

oefrha7• 6 years ago

Don’t forget the builtin options.

First there’s PSL ctypes,[1] which is supposedly inferior to cffi to some extent, but hey it comes with your CPython installation, no third-party dep required.

Or you can write a C/C++ extension module.[2] A bit more heavy-handed and there’s a learning curve, but once you’ve done it once or twice it’s really not hard, and it affords you great flexibility.

[1] https://docs.python.org/3/library/ctypes.html

[2] https://docs.python.org/3/extending/extending.html

atilaneves20• 6 years ago

The solution in the blog post is an extension module. Written for you, by the compiler. Instead of a learning curve and "it's really not hard", it's "no learning curve" and "I struggle to find how this could be easier".

SQueeeeeL• 6 years ago

My problem with these is that in my use case, it completely broke on Windows...

yellowapple2• 6 years ago

I'm curious about what ctypes' supposed inferiorities are. Last time I used it was to implement native printing in Windows (long story short: needed to print to a ZPL printer, and Qt doesn't seem to offer any ability to just send raw data to a printer, so I implemented it myself with the Winspool API), and it was a breeze (comparatively speaking, i.e. as much of a breeze as Windows API programming can be).

If ctypes can handle the Windows APIs, then I'm convinced it can handle just about anything.

0xff00ffee• 6 years ago

I don't understand how the OP could write this article and not talk about FFI? What a glaring oversight. (FWIW: The FFI module works extremely well in NodeJS as well.)

atilaneves20• 6 years ago

After a week, I've finally had time to look at cffi. It's not even remotely as easy. I tried using it just now to call nanomsg and gave up immediately. I can't just give it the headers, I have to pass it a string which is valid C code. I can't read the headers (separately!) and pass that in either because it can't handle preprocessor directives. It's a mess.

In the article, I do this:

``` #include "nanomsg/nn.h" #include "nanomsg/pipeline.h" ```

That's it. I don't think the approaches can be compared.

hatsunearu• 6 years ago

Or you know, learn any of the native options that let you call C things relatively painlessly.

atilaneves20• 6 years ago

I describe how easy it is to call C code from Python by using D as the glue layer. AMA.

cannam• 6 years ago

So behind the scenes is a utility that scans the #included headers, runs the C preprocessor on them, then uses Clang code to pick out the declarations, wraps them for calling from D, and redefines any C macro values as D constants? Do I understand that correctly?

(The other half of the puzzle being a tool called autowrap that wraps the resulting D for consumption by Python.)

I'm not sure whether I personally would ever use this, but it's a very interesting exploration of the subject.

atilaneves20• 6 years ago

Do I understand that correctly?

Yep!

(The other half of the puzzle being a tool called autowrap that wraps the resulting D for consumption by Python.)

Correct.

I'm not sure whether I personally would ever use this, but it's a very interesting exploration of the subject.

Python was an example. Want to call it from C# instead? Same code. Excel? Same code.

jabwork• 6 years ago

Have you used pythons cffi and if so how would you differentiate this from that?

Does this work for all revisions of C? iirc cffi works for all of c89 and most of c99

atilaneves20• 6 years ago

No, but I used the equivalent in Lua. I was under the impression that no solution handled preprocessor macros, but this thread corrected me. I might do a follow-up post on this.

klyrs• 6 years ago

I've been using Cython since it was called Pyrex, and I'm perpetually annoyed with rewriting c/c++ headers in Cython. This seems neat. But the reason that I use Cython is that I can wrap complex data structures (including templated c++ classes) and do processing in a fuzzy boundary between Python and c/c++.

The "code-free" approach here seems to forbid such a fuzzy boundary. Am I wrong?

atilaneves20• 6 years ago

I can't really comment without an example. I don't think this approach stops you from doing anything. It can (and has) allowed Python to call into templated D code, as long as you give the particular instantiation you're interested in an alias.

skocznymroczny• 6 years ago

"Envious of C++’s and Objective C’s credible claim to be the only languages that can seamlessly interoperate with C (due to header inclusion and compatible syntax)"

Wouldn't Nim and Zig also fit the bill?

atilaneves20• 6 years ago

The last time I looked at the Nim documentation, one had to manually declare C and C++ functions. I don't see how that's different from pretty much any other language that can call C (i.e. pretty much all of them). Then there's preprocessor macros.

I don't know how Zig does it, I'd have to look.

I expand on why dpp is the way it is in my DConf 2019 talk: https://www.youtube.com/watch?v=79COPHF3TnE&t=8s

nimmer3• 6 years ago

Nim has an automated generator for C/C++ wrappers or translating from C.

Also there are very easy ways to create Python modules in Nim:

https://robert-mcdermott.gitlab.io/posts/speeding-up-python-...

https://github.com/yglukhov/nimpy

mratsim2• 6 years ago

It's easy to call preprocessor macros from Nim, no difference from wrapping C:

Example on wrapping a simple benchmark macro:

- https://github.com/mratsim/weave/blob/052ae40a/experiments/e...

steveklabnik• 6 years ago

Zig basically includes clang inside of it: https://twitter.com/andy_kelley/status/1099485306783440896

jaked893• 6 years ago

D & Nim (especially Nim) are very appealing from a language perspective, but the lack an IDE undermines the advantages they bring.

I personally won't consider using a language for a meaningful project unless it has a decent debugger, and reasonable refactoring; ie. at least find references & rename.

Beef is building on an IDE, which is a refreshing take, but it's still too early.

atilaneves20• 6 years ago

I'm not an IDE user, but I know that certain D programmers use Visual Studio (with VisualD) and seem happy. Did you try that option?

EvenThisAcronym• 6 years ago

Visual Studio with Visual D is a top notch D experience.

giancarlostoro2• 6 years ago

D works fantastic with VS Code. See here:

https://wiki.dlang.org/Editors

Also had really good luck doing D with Spacemacs, but Spacemacs is a hit or miss to setup for me. Other times I just use Sublime Text for the syntax highlighting.

kbd• 6 years ago

Nim also works very well in vscode. Inline compiler errors as you edit, and so on.

vmchale2• 6 years ago

What C IDE do you use?

sigstoat• 6 years ago

FWIW, I did some embedded development in C (STM32) using Eclipse, and I was astonished by how nice the whole environment was.

beagle3• 6 years ago

Nim works very well with Emacs and Visual Studio Code at the very least, also Vim (from reports by others) - including debug support. What are you missing?

jaked893• 6 years ago

Debugging and (semantic) renaming. As said, this is the minimum requirements I pose on any language for production use. Nim in VS Code lacks both.

There's some limited debugging support at the generated C code, but this is not sufficient for me.

JNRowe3• 6 years ago

My own pet peeve about D is the lack of local documentation. If I install my system's compiler packages¹ I receive basically no documentation at all. That is a huge difference to the experience with nim/rustc/go/$others which all provide a nice fat $lang-doc package in their Suggests field, and all of those contain full standard library docs/tutorials/etc.

1. gdc or ldc on Debian. The experience /may/ be different with the dmd repositories, but being external they're of less interest to me.

alphaglosined• 6 years ago

It is provided by dmd as part of the download.

You can also get soft copies (i.e. mobi, pdf) and the docs as part of the d-apt repository for debian.

Depending on the nature of distribution it is probably just in a different package (or the package maintainer needs yelling at).

https://d-apt.sourceforge.io

dguaraglia• 6 years ago

I'd recommend using Dash (on Os X) or Zeal (Linux and Windows) if you want to have an easily browseable local version of the docs.

WalterBright3• 6 years ago

D comes with an interesting utility called `dman`. Run it from the command line and as an argument give it a D topic and it'll open a browser on the topic.

    dman dman

will, of course, open the browser on a page about dman.

ktm5j• 6 years ago

You're spot on that D is lacking a batteries-included IDE, but there are some good options available.

D in emacs works very well, but you'll have to put some work into it. Spacemacs (https://www.spacemacs.org/) makes the work a little easier, here's my .spacemacs file if anyone is interested https://gitlab.com/snippets/1942930

Using that .spacemacs file you should be able to launch a fully prepped for D! The only other dependencies are a D compiler and DCD (https://github.com/dlang-community/DCD) built and in your PATH. I'd be happy to answer any questions if something doesn't work.

Other IDEs with a languageserver implementation can make use of DLS https://github.com/d-language-server/dls. VSCode and sublime are two I've had good success with.. but I always end up back with emacs ;)

As for a debugger, you can debug any D program with GDB/LLDB (depending on if you compile with dmd/lcd). Then you can use any graphical debugger that uses gdb. Nemiver is my favorite, then there's a few web based ones that are nice: https://www.gdbgui.com/

mhh__• 6 years ago

VisualD is pretty damn good, visual code has a few D plugins that are ok to excellent when they work (which is most of the time)

There is also a D specific IDE which I can't remember the name of which is probably best if you just want something that works (it's pretty lightweight)

jononor• 6 years ago

I often use C++ and pybind11 to do this. pybind11 takes an approach very similar to boost::python (template magic). It works rather well.

Though most of the C code I write can compile as C++ also. Why write in C then? It's still the lingua franca in microcontroller land... Just want to reuse the code on PC and Embedded Linux, and Python makes high-level testing much nicer.

atilaneves20• 6 years ago

Both pybind11 and boost::python involve writing code. This solution doesn't, save for listing the names of the files to be made available to Python (or C#, or Excel, or...).

oefrha7• 6 years ago

Neither ctypes nor cffi involves writing glue code. In fact you don’t even need another source file, let alone a whole other language toolchain (wait, you actually need two other source files plus your third-party dpp library).

aldanor• 6 years ago

Pretty much this. Add to that pybind11's exceptional numpy compatibility (including easily mapping C types to structured numpy dtypes that can be e.g. converted to a dataframe), and there's not many choices left if you're not planning on reinventing a wheel or two.

(disclaimer: pybind11 contributor)

pletnes• 6 years ago

Is pybind numpy aware? This is interesting. Having never used pybind, where can I learn more?

jakogut3• 6 years ago

I'm currently working on a project using Cython to wrap and expose a large and complex C API (CEF) to Python. One of the most time-consuming, tedious, and error prone tasks is translating functions and structures to Cython syntax, including managing the GIL. Also, keeping the translated API up to date with new releases.

I'm curious to see how this D approach compares, both in workload required to expose and use an API, and the flexibility of the end result. There are a lot of quirks and special syntaxes in Cython to allow for not just interop between C and Python, but also writing C in Python-like syntax mixed with actual Python. This is an area I assume this approach would be lacking.

Regardless, as someone who loves mixing C and Python, this is an encouraging reason to look more at D.

jessermeyer2• 6 years ago

Is there a reason you chose not to build / refactor the project as a library to call from Python?

jakogut3• 6 years ago

Can you clarify your question? I'm using Cython to write extension modules that directly interface with CEF's C API. CEF is about 40k lines of C++ that programmatically controls Chromium, which itself is written in C and C++.

loeg4• 6 years ago

This D approach is essentially cffi. With another compiler thrown in for fun.

epicgiga• 6 years ago

How many languages do we need again? The amount of time we spend playing around with special case languages and frameworks. Would be nice if machine languages followed the lead of natural languages, with a single one slowly pushing the rest to the sidelines.

abrax31413• 6 years ago

Indeed lisp is slowly (very very slowly) doing this. Witness how it seems to get reinvented or embedded in every other language.

mratsim2• 6 years ago

There were some studies about the impact of the loss of some natural languages. Some represent a complete different ways of handling the world that are worth keeping.

And there is no one size fits all.

macintux• 6 years ago

What a terrible world that would be. For a while, 20 years ago, it looked like Java was going to be the one language to rule them all; dodged that bullet.

Kitchen sink languages are my least favorite. Give me something with a tight focus on a problem space.

terminaljunkid• 6 years ago

with a single one slowly pushing the rest to the sidelines.

That didn't quite happen.

How many languages do we need again? Till you eliminate all tradeoffs in a single language, there would be new ones.

abrax31413• 6 years ago

Okay, but what if I want to call python libraries from C (or any other language)? What we need is some sort of .net-like global interop in the Unix world.

2fast4you• 6 years ago

If I want to make an application composed of C, C#, and Visual Basic, there nothing stopping me in Windows. You try to pull that shit in Unix though, and you get a whole mess of problems. This is why Unix is clearly just so inferior. /s

dralley• 6 years ago

You can't really do that without including the entire Python runtime in your application, and cpython is (so I've heard) absolutely horrid at being embedded in things.

abrax31413• 6 years ago

Why can’t we create a --server mode standard where every language can be started up in that mode and serves a standard API with standard linker types (where possible)?

srean• 6 years ago

Extending is ok, embedding is indeed quite horrid. Tcl got that right. Yu want to embed multiple interpreters in the same address space with no sharing -- no problem. The mantle has been passed to Lua I think, but Guile is pretty good and not as minimalistic as Lua (that's bit of an understatement though).

vmchale2• 6 years ago

Linker types? I think I recall Amal Ahmed gave a talk on them. http://www.ccs.neu.edu/home/amal/papers/linking-types.pdf

bepvte• 6 years ago

You might be interested in GraalVM, which can do this between certain languages implemented on top of it.

int_19h• 6 years ago

.NET-like interop relies on having a specific runtime.

WinRT is a better model. Basically a higher-level ABI.

andrewprock2• 6 years ago

Is swig no longer a thing in the 2020s?

I've always done this using the bundled Python/C integration path, or when there was too much stuff, use swig.

http://swig.org/

atilaneves20• 6 years ago

I used swig once. Never again.

andrewprock2• 6 years ago

I'm curious, what was your negative experience?

dekhn• 6 years ago

It's really crazy how many ways there are to implement FFI in Python. I started in the "old days" (Makefile, a bunch of CPython C code to implement argument parsing, etc), switched to swig (experienced both the good and bad sides), hoped that GCC-XML would make it easier to convert complicated C++ classes to Python interfaces (big mistake), then hoped that LLVM 'would solve the problem', saw Google release CLIF (still can't compile it from scratch on a modern linux system), and the large number of "late-binding" approaches. Of all of them, I was most skeptical of Cython but have slowly come to appreciate its sublime cleverness.

WalterBright3• 6 years ago

I wrote a little project called dpp because I’m lazy

I can confirm that Atila is a very lazy man. He's always figuring out ways to have his computer do the work for him. He's so successful at avoiding work, people keep giving him more work to do, making him a very hard working lazy man.

canada_dry• 6 years ago

always figuring out ways to have his computer do the work for him

One of my earliest programming mentors mused that the reason I might well become a decent programmer was because I was both extremely stubborn and horrifically lazy. He'd observed that I would re-write code until it was the most efficient possible to automate a needed task that could save me time/effort in the future.

ugexe• 6 years ago

It’s a common observation, and indeed known as one of the Three Virtues of a great programmer according to Larry Wall

http://threevirtues.com

marmaduke• 6 years ago

https://github.com/davidjamesca/ctypesgen/wiki

does this without needing a compiler, by parsing the header files and generating the ctypes wrappers

atilaneves20• 6 years ago

There's no way to parse headers without a compiler.

justlexi93• 6 years ago

Scripting languages by nature will always be slower than C. There's a tradeoff between program speed and development speed

fithisux• 6 years ago

Hopefully the dream to compile a D library to another language (something like swigd) it will be interesting

atilaneves20• 6 years ago

That already works with Python, Excel and C# now. There's a link in the blog post to another blog post to that effect.

Want to call C from Python? Use D

Related Stories