Speed improvements in Python 3.7

https://hackernoon.com/5-speed-improvements-in-python-3-7-1b39d1581d86

by orf • 8 years ago

78 47 8 years ago

singularity20016• 8 years ago

did they further improve backwards compatibility?

Neither trolling nor beating dead horses:

I recently wanted to do some screen capture and needed tkinter, which of course has different imports for py2 and py3. adding library aliases would help. I honestly wished people would use transitional deprecation warnings more often to educate about changed apis.

As long as 50% of tensorflow projects are still in Python 2 it is always a pain to encounter Python 3.

``` try: from urllib2 import urlopen from urllib import urlretrieve except ImportError: from urllib.request import urlopen, urlretrieve ```

orf5• 8 years ago

Have you tried the 'six' library? It does what you describe, as well as paving over many py2/py3 differences

singularity20016• 8 years ago

Tangentially heard of it. Thanks, will try.

Does it add py2 conventions to py3 or Does it add py3 conventions to py2 ?

kevin_thibedeau• 8 years ago

Is 15 years to prepare still not long enough to migrate your code incrementally? You can get Py2 into a nearly-ready-for-Py3 state with just a little effort stretched over whatever horizon you desire. Beating a dead horse doesn't do anything about your stagnant code base.

singularity20016• 8 years ago

it's not about migrating old code bases, it's about encountering other peoples tensorflow/ubuntu code and often ending up in unnecessary(!) hickups.

orf5• 8 years ago

I used to be quite militant about this, but you have to get that often business goals don't align. You can often do it on the side, during other changes, but convincing your manager that it's worth it is a hard problem.

mjevans6• 8 years ago

No, because they did a bunch of things that make automated porting not work well and provides a ton of developer friction.

I'm willing to live with Python's space related insanity. Mandating that indenting is meaningful. Mandating a best practice of using spaces instead of tabs for indent* (which makes tab width matter). Mandating that a tab equals 4 spaces instead of 8. (Honestly, I REALLY hate this, most of the Windows developers are happy with the space usage koolaid, let us have a default of tab equals 8 spaces of indent.) Those are ALL, REALLY ANNOYING, but lint tools can save me.

What will bite me for as long as Python 3 exists, and maybe Python 4, is how they handle Unicode support and encoded outputs. Pretty much the ENTIRE rest of the computing world does one of two things.

1) Doesn't care what the encoding is, and flings around non-validated 8 bit sequences of data.

2) Does care, and defaults to assuming well formed UTF-8 data-streams for text data, BUT still doesn't do anything to enforce it UNLESS ASKED. Most importantly invalid data streams just continue flowing through the processing tools until they reach a point where a human has designated they're willing to care and /do/ something about it.

If Python 4 wants to 'fix' Unicode support:

    * Use a binary sequence base storage class with metadata tags.
    > Length (in bytes, in 'display length', and maybe in 'codepoint length')
    > encoding* (including if normalized and how so)
    > If the encoding has been validated.
    * Support automatic coercion between standard encodings
    * Allow user defined helpers for custom conversions
    * Make all input/output default to UTF-8**

but permit raw output of such types without validation or re-encoding, just like every other UNIX tool that knows what an encoding is.

viraptor8• 8 years ago

For the new 3.7 there's also gc.freeze (https://bugs.python.org/issue31558) which will be great for many web projects.

orf5• 8 years ago

https://docs.python.org/3.7/library/gc.html#gc.freeze

Documentation link for anyone that is interested

rojoca• 8 years ago

Also the source(?) of this change from instagram: https://engineering.instagram.com/copy-on-write-friendly-pyt...

wyldfire• 8 years ago

Also collection before a POSIX fork() call may free pages for future allocation which can cause copy-on-write too so it’s advised to disable gc in master process and freeze before fork and enable gc in child process.

I guess it would be nice to know if optimizations like this are already covered in the high-level libraries like `multiprocessing`.

hartator• 8 years ago

Is it faster than 2.7 now?

ubernostrum5• 8 years ago

Python 3.0 was slower than Python 2.7, largely because some new modules were initially implemented in pure Python in order to get the APIs out there for people to experiment with. They're since been replaced with implementations in C.

Python 3 passed Python 2 on performance years ago.

icegreentea2• 8 years ago

Except at interpreter startup.

jxub• 8 years ago

Python is the best (as in most productive and happy) language for me bar none.

It feels great to know that speed, the only major drawback, is being seriously improved in each release. Here's to the future!

billsix• 8 years ago

If your looking for a nice gui framework for python, check out my pyNuklear!

https://github.com/billsix/pyNuklear

btown2• 8 years ago

Since one of the mentioned improvements is performance of the builtin regular expression library... for anyone curious, there's a drop-in replacement, re2, that leverages a Google library to provide linear time guarantees for arbitrarily complex or user-provided regexes. (You lose lookahead assertions, but most use cases for them can just be handled in pre- or post-processing.) It's an incredibly handy tool that lets us confidently use complex regexes in production.

https://pypi.python.org/pypi/re2/

https://github.com/google/re2/wiki/WhyRE2

RE2 was designed and implemented with an explicit goal of being able to handle regular expressions from untrusted users without risk. One of its primary guarantees is that the match time is linear in the length of the input string. It was also written with production concerns in mind: the parser, the compiler and the execution engines limit their memory usage by working within a configurable budget – failing gracefully when exhausted – and they avoid stack overflow by eschewing recursion.

Really interesting articles in the second link detailing how this works!

tyingq2• 8 years ago

Great observation, but re2 does have tradeoffs to gain that performance. Backreferences, for example: https://github.com/google/re2/issues/101

Blackthorn• 8 years ago

Backreferences are one of the items that makes regexes non-linear, which defeats the point of being able to trust arbitrary regexes to have good performance.

btown2• 8 years ago

Absolutely. But for most common uses of backreferences, this is largely mitigated by having a post-regex step that compares capturing groups for equality, which induces overhead also linear in the length of the string. Of course this falls short of true backreference support, but for tasks where we do not expect potential matches to overlap, it gets the job done.

Speed improvements in Python 3.7

Related Stories