PyPy 1.6 Released - Full Python 2.7.1 Implementation

http://morepypy.blogspot.com/2011/08/pypy-16-kickass-panda.html

by jparise • 15 years ago

244 54 15 years ago

morepypy.blogspot.com

voyvf2• 15 years ago

> has beta level support for loading CPython C extensions.

Is this via ctypes, or "real" support in much the same as how CPython would behave?

I ask because this is one of the features that I've been waiting (impatiently) for - I've run some Flask projects using PyPy and gunicorn, and love how fast it goes, but really want to be able to use the rest of my codebase, which unfortunately does rely on some C (and Cython) extensions. (:

vgnet4• 15 years ago

It's "real" support, using your criteria. However, it's slower in PyPy than in CPython and than what the same thing based on ctypes would be on PyPy.

If there's a pure-Python version of the C extension, it might be faster on PyPy than the C extension support that cpyext (PyPy C-API compatibility module) provides.

Cython-based code is currently incompatible (it goes well beyond the public C-API), but a GSoC project to generate ctypes-based pure-Python code from Cython is (was?) going on.

voyvf2• 15 years ago

> It's "real" support, using your criteria. However, it's slower in PyPy than in CPython and than what the same thing based on ctypes would be on PyPy. > If there's a pure-Python version of the C extension, it might be faster on PyPy than the C extension support that cpyext (PyPy C-API compatibility module) provides.

That's fine, the mere ability to run them will be nice. (:

> Cython-based code is currently incompatible (it goes well beyond the public C-API), but a GSoC project to generate ctypes-based pure-Python code from Cython is (was?) going on.

I remember that GSoC! Any idea as to what happened to it, if it was scrapped, or whatever?

pjenvey• 15 years ago

Not via ctypes, via CPyExt, a layer emulating the CPython C extension API (it actually implements a reference counting GC and everything)

sylvinus• 15 years ago

I'm always blown away by the consistent performance gains they reach with each new version. Congrats!

kristofferR• 15 years ago

In general I'm very happy with my choice of Ruby/Rails instead of Python/Django, but PyPy is one of the few things I envy Python developers for.

I wish something similar could be developed for Ruby.

postfuturist• 15 years ago

http://rubini.us/

lobo_tuerto2• 15 years ago

Maybe rubinius will fill this need? http://rubini.us/

Also, this could be an interesting read: http://www.engineyard.com/blog/2010/making-ruby-fast-the-rub...

bryanwb• 15 years ago

but what is the status of rubinius? is it nearly as far along as pypy?

I would think that the metaprogramming features of ruby which make it so much fun would also make it n times harder to build a JIT for.

irahul2• 15 years ago

> but PyPy is one of the few things I envy Python developers for.

Depending on your needs, there are others - numpy, scipy, matplotlib, nltk, gevent.

> I wish something similar could be developed for Ruby.

Isn't http://rubini.us/ supposed to be the PyPy for Ruby? It's not complete, but then neither is PyPy.

alphakappa• 15 years ago

matplotlib and numpy make Python really awesome. If only there was a way to get matplotlib to work on Lion.

stephenjudkins2• 15 years ago

Rubinius and PyPy take quite different approaches to solving the same general problem. It's inaccurate to analogize them.

Rubinius specifies a bytecode (see http://rubini.us/doc/en/virtual-machine/instructions/) and implements a VM that executes this bytecode. The VM is written in C++. The main difference between Rubinius and other Ruby implementations is that nearly everything else is written in Ruby. The lexer, parser, and parts that compile the AST to bytecode is all written in pure Ruby. Further, large parts of the standard library and Ruby language that are implemented in lower-level languages in other implementations are instead written as runtime-level Ruby. However, large and important parts of the infrastructure (including GC and memory allocation) are written in C++.

PyPy takes an altogether different approach. It specifies a restricted subset of Python known as "RPython". Basically, it's statically-typed Python. There is also a "compiler" (also called a "JIT generator"), written in pure Python, that compiles an arbitrary interpreter written in RPython into a fast JIT. (See http://morepypy.blogspot.com/2011/04/tutorial-writing-interp... for an example of writing a very simple, small interpreter in RPython.) A cool feature of RPython is that while there must a statically RPython AST to compile, during the "eval" stage you can leverage the full dynamic capabilities of Python.

The second part of PyPy is a full Python interpreter written in RPython. Since RPython is a subset of Python, you can actually run this interpreter sandboxed inside any other Python implementation, albeit very slowly. The PyPy toolchain compiles this down to a very fast Python interpreter that features JIT compilation, among other cool features.

The neat part of PyPy (in contrast to Rubinius) is that the toolchain part is largely decoupled and generalized away from Python-the-language. Conceivably, one could write ANY language in RPython (or what's compiled down to RPython) and have a fast JIT interpreter. Experimental language features can be added and changed (see the source code for the different garbage collectors in PyPy) with little mucking with the low-level VM.

Rubinius is a neat, pragmatic project that could turn into one of the best Ruby implementations around. However, it's not breaking new ground like PyPy is.

luismgz• 15 years ago

> I wish something similar could be developed for Ruby.

There's something like Pypy for Ruby already: It is Pypy itself. Pypy is a framework for implementing jitted dynamic languages (any language, not just python). You can generate interpreters for any language you want, as long as you write them in Rpython (restricted python). So, if you want a pypy for Ruby, just write a ruby interpreter in Rpython, and then use the pypy translation toolchain to compile it down to C, while automatically generating a just in time compiler for free.

sirn• 15 years ago

What about Rubinius?

Can you also elaborate on why you're happier with Ruby and Rails than Python and Django. Since I'm completely the other way around, while Ruby paid my bills, if there's a Python job offering I would have jumped onto it without much thought.

One of few things I envy Ruby developers for is the HAML, SCSS, Jammit stuff. Apart from that, not so much, especially for documentations for Ruby modules. It might be from the fact that I prefers Flask[1] to Django, but I also like Python's explicitness than Ruby's implicit and found Python language design to better suits my taste (e.g. the Python class[1]).

[1] http://lucumr.pocoo.org/2011/7/9/python-and-pola/

[2] http://flask.pocoo.org/

mattdeboard• 15 years ago

I use SCSS with Django all the time.

rglullis• 15 years ago

As an alternative for scss and jammit, you can try django-css: https://github.com/dziegler/django-css

Freaky• 15 years ago

If it's mainly the performance you're after, have a look at JRuby. It's already a decent chunk faster than MRI/YARV, and with Java 7 and invokedynamic there's a lot more to look forward to and play with now.

Last week one of the JRuby core devs tweeted: "Ok, another set of tweaks in JRuby and fib(35) is another 10% faster (and only about 10% slower than Java. Do I stop now?"

lliiffee5• 15 years ago

Can I take advantage of this thread to ask the HN crowd a technical question? Some time ago, I implemented an automatic differentiation tool. Using operator overloading on a special "autodouble" type the tool would trace the execution of a block of numerical code. Then, some calculus would automatically happen, and it would output and compile fast c-code that would compute the original function and derivatives in pure c. This was great, except the c-code that was output was freaking gigantic (like hundreds or thousands of megabytes) albeit very simple, and so the c compiler would take forever to run. Sigh.

My question is: could I leverage pypy somehow to avoid this? Can I output RPython? Can I output whatever RPython is compiled down to instead? Can I do this with no more than, say, a 3x penalty compared to c?

(I apologize for asking a question only marginally related to the particular article here...)

carterschonwald2• 15 years ago

Off hand and guessing about the problem:

You might want to look at approaches that use dual numbers. Likewise, in instead of inlining the procedures, generate a differentiated version of each procedure with a new name. If those don't cover your problems, perhaps look at how other autodiff tools for C do it?

lliiffee5• 15 years ago

For being offhand, those are very good guesses! Dual numbers won't be efficient, as I want reverse-mode autodiff.

As to the multiple procedures: Well, as I was doing it, even a single procedure can be many hundreds of megabytes large. Even for very very simple code, however, I noticed that GCC was superlinear in code size. I suppose I could somewhat arbitrarily break up the code into arbitrary functions. I wonder if that would speed things up?

Other autodiff tools: Well, they basically trace execution, but then run an interpreter instead of trying to actual generate compiled code. I wanted to both be faster than that and have the wonderful experience of writing pure python...

sjs3• 15 years ago

LLVM? I don't have much experience with runtime code generation but you might find it useful.

lliiffee5• 15 years ago

LLVM would seem to (maybe? possibly!?) be a good solution, but I was never able to convincingly determine if it was likely to solve my problems, and the documentation seemed to be quite sparse for doing what I wanted to do. (This was a couple years ago, however, and I've love to be proven wrong.)

beambot• 15 years ago

Theano (http://deeplearning.net/software/theano/introduction.html) might be worth a shot if you have closed-form expressions.

'''Theano is a Python library that lets you to define, optimize, and evaluate mathematical expressions, especially ones with multi-dimensional arrays (numpy.ndarray). Using Theano it is possible to attain speeds rivaling hand-crafted C implementations for problems involving large amounts of data. It can also surpass C on a CPU by many orders of magnitude by taking advantage of recent GPUs.'''

ori_b• 15 years ago

RPython compiles down to C. Also, do you have an example of the generated code? I'm not sure what sorts of patterns would need it to be hundreds of megabytes.

(Also, if Pypy's compilation time is any guide, it takes massive amounts of time to compile rpython)

socratic3• 15 years ago

Are there production users of PyPy?

I feel like PyPy has always been the most academically interesting Python implementation. But has it taken away mindshare from CPython?

vgnet4• 15 years ago

Yes, there are.

Recently Quora announced it was running on PyPy[1]. Some other disclosures were made (a Django project[2], LWN internal processing[3], tweets about speedups in production, etc.), but the PyPy team is thinking about officially asking for success stories in the near future[4].

[1] http://www.quora.com/Alex-Gaynor/Quora-product/Quora-is-now-...

[2] https://convore.com/python/whos-using-pypy-in-production/

[3] http://lwn.net/Articles/442268/

[4] https://bitbucket.org/pypy/extradoc/src/tip/blog/draft/succe...

socratic3• 15 years ago

To me, this list seems to suggest the opposite actually.

[1] is the company where one of the main developers of PyPy works. [2] appears to be a discussion where they are looking for anyone using PyPy in production. [3] seems to be an article about someone experimenting with PyPy for git processing. [4] is the developers looking for non-toy examples of production use so that they can get more funding.

sho_hn• 15 years ago

I'm still bummed at being stuck with the dilemma of having to chose between CPython 3.x and PyPy. PyPy with Py3k support would rock.

PyPy 1.6 Released - Full Python 2.7.1 Implementation

Related Stories