GoPy – Extend CPython by writing Go

http://gopy.qur.me/extensions/examples.html

by spdy • 13 years ago

209 36 13 years ago

gopy.qur.me

Goranek• 13 years ago

I would love to see some benchmarks here! Great idea!

microtonal3• 13 years ago

Nice! I assume this only works with gccgo (to create a shared library)?

levesque• 13 years ago

Nice! It looks pretty simple, but it makes me wonder. Why bother interfacing your Python code with Go? Why not go straight to C for more performance?

bvdbijl• 13 years ago

Because Go is way more pleasant to write and is still very performant

pavanky• 13 years ago

1. Because it can be done.

2. The native way of doing this in C (without using SWIG or cython or any other such)is more verbose.

3. It may be easier (not necessarily faster) to write certain code in Go as opposed to both python or C.

rattray2• 13 years ago

Would Go's static binaries be an added advantage?

Laremere• 13 years ago

Python doesn't do any parallel computation. Go does it very well. I honestly don't know how to do parallel computation in C or C++, but I doubt it's very easy. Slower code can run faster overall if you have multiple processors, so pure processing speed has the potential to matter less.

microtonal3• 13 years ago

I honestly don't know how to do parallel computation in C or C++, but I doubt it's very easy.

Actually, with OpenMP parellelization is often easy in C and C++. In embarrassingly parellel problems, it is often a matter of adding an OpenMP pragma before the for-loop. To give one example, in some machine learning code I wrote, it was simply a matter of replacing

  for (int i = 0; i < static_cast<int>(contexts.size()); ++i)

by:

  #pragma omp parallel for
  for (int i = 0; i < static_cast<int>(contexts.size()); ++i)

Obviously, you have to ensure that shared mutable variables are properly protected. But, in many cases it's fairly simple. I have tried to emulate some of OpenMP's functionality in the par package:

http://github.com/danieldk/par

I think the more convincing argument is that many Python programmers will feel far more comfortable writing Go than C or C++. It offers good performance and some control over memory layout, while being substantially higher level than C and not the enormous language C++ has become.

d0mine2• 13 years ago

You can do parallel computations in Python e.g., many numpy functions can utilize more than one CPU:

  import numpy as np

  N = 5000
  # create a couple NxN matrices with random elements
  a = np.random.rand(N, N) 
  b = np.random.rand(N, N)
  # perform matrix multiplication
  c = np.dot(a, b)

Cython makes it easy to write C extensions for Python if necessary.

You can also do a compute-intensive work in parallel using multiple processes:

  import random
  from multiprocessing import Pool
  
  def fire(nshots, rand=random.random):
      return sum(1 for _ in range(nshots) if (rand()**2 + rand()**2) < 1)
  
  def main():
      pool = Pool() # use all available CPUs
      nshots, nslices = 10**6, 10
      nhits = sum(pool.imap(fire, [nshots // nslices] * nslices))
      print("pi = {pi:.5}".format(pi=4.0 * nhits / nshots))
  
  if __name__ == '__main__':
      main()

The code calculates Pi using Monte Carlo method http://demonstrations.wolfram.com/MonteCarloEstimateForPi/

dvirsky• 13 years ago

outsourcing parallel work to goroutines? yes please!

jff• 13 years ago

Nice! Does this let you get around the GIL?

mmariani• 13 years ago

I believe so, at least you can in C. You have to release the GIL in your extension, do your business, and cleanup before giving the control back to Pyhton.

http://docs.python.org/3.3/c-api/init.html

Edit: Even though you can do it in C, Go is a no go. Under item four in the caveats [1] is stated:

  | When using goroutines py.Lock must be used. No other 
  | GIL or threading interface functions have been 
  | tested - and they most likely will not work.

[1] http://gopy.qur.me/extensions/

tiredofcareer2• 13 years ago

> Calling Go code from a second thread will currently cause the code to raise an exception (i.e. all calls into Go code must be made from the same thread)

That's ... interesting.

d0mine2• 13 years ago

yes. See lock.UnblockThreads(); in the parallel example from the article (the sleep is executed without holding GIL).

rattray2• 13 years ago

This looks fantastic. Having only worked extensively with Python and Javascript, I haven't been looking forward to learning C or C++ to write performant bits of an application, but Go really excites me.

What are the limitations? Are there situations where GoPy won't work that (eg) ctypes would?

toyg• 13 years ago

The main limitation atm is that it looks like a very experimental project, nowhere near production-quality.

pjmlp3• 13 years ago

For new code, I would rather use Julia.

nnq• 13 years ago

...was thinking of the same thing. Julia seems much nicer for the use case of writing scicomp code, but Go overall will be a much more popular language. How would you choose one of them for writing performance tuned Python extensions?

pjmlp3• 13 years ago

If it is for writing extensions, then PyPy would be a better option if it is to stay in Python land.

For writing new code from scratch, Julia is a much better option. The language is not yet 1.0 and already achieves parity to C compilers in many algorithms.

Go compilers are yet to match JVM and .NET speeds, let alone C.

seunosewa• 13 years ago

When can we expect a Windows port?

apendleton2• 13 years ago

This looks really slick, but it's a bit unfortunate that this was implemented in a way that's tied to the CPython API specifically, given that there are Python-VM-neutral methods of low-level interop available (ctypes, CFFI, etc.). My general sense (and certainly my own experience) has been that many people doing speed-sensitive Python work are already running on PyPy now, and that writing lower-level extensions has become a way to squeeze out an extra level of performance beyond that, so having to go back to CPython to take advantage of this infrastructure would be kind of a bummer, and almost definitely a non-starter in my projects, which leaves me stuck with C.

masklinn• 13 years ago

> My general sense (and certainly my own experience) has been that many people doing speed-sensitive Python work are already running on PyPy now

Depends, an other category drops down to Cython (which in many cases of low type dynamism consists of just running Python code through Cython).

apendleton2• 13 years ago

Right, I definitely don't think PyPy+[ctypes|cffi] is universal ("many," not "all"). Lots of people are doing Cython. Also, anyone who has a performance-sensitive workload that depends on SciPy is out of luck on PyPy at least for the time being. All I'm saying is that PyPy as a performant-Python strategy is common enough now that it's worth supporting in new performant-Python projects.

fixxer2• 13 years ago

My thoughts exactly. Great concept and I will definitely follow it as it develops, but I'd love to see an abstraction layer that takes the pain out of the a$$ that is the Python C-api. I left that years ago and I'm not going back.

dman• 13 years ago

Ive been using the Python C api recently and I find it the most straight forward option of getting performance out of Python. If you write a few helper classes in C++ using RAII - the python c-api becomes actually pretty pleasant to use.

SatyajitSarangi• 13 years ago

This looks brilliant. My only concern being, consdering an extra layer comes in, how much is the performance affected by this?

GoPy – Extend CPython by writing Go

Related Stories