What’s New in Python 3.5

https://docs.python.org/3.5/whatsnew/3.5.html

by calpaterson • 11 years ago

288 131 11 years ago

docs.python.org

Lambdanaut• 11 years ago

Oh my god they finally removed the GIL!

lambda3• 11 years ago

Hmm? Where does it say that?

dalke2• 11 years ago

Umm, no? https://docs.python.org/3.5/glossary.html#term-global-interp... - "The mechanism used by the CPython interpreter to assure that only one thread executes Python bytecode at a time"

jdlyga• 11 years ago

I still use Python 2.7. I guess I should upgrade.

theseatoms2• 11 years ago

I've found this website useful for checking which packages have made the jump.

https://python3wos.appspot.com/

rspeer2• 11 years ago

That site uses a very outdated list of popular packages. Use http://py3readiness.org/ instead.

inglor4• 11 years ago

I'm extremely enthusiastic about `async/await` semantics, and in particular async iterators and the `for async` loop.

It's astonishing to see how fast Python is moving in this direction. It is truly a powerful language for async computations now - surpassing in this ability both JavaScript and C# (at least for now). For example Python got `async with` but JS isn't even close (userland solutions like Bluebird's using exist) and C# is only starting to work on `IAsyncDisposable`

rayalez• 11 years ago

This is awesome!

I want to ask:

- Does it mean that it will be possible to create realtime apps with python? Will it be possible to use it instead of node.js? Do you think Django will implement that functionality?

- Now that WebAssembly is coming - doesn't it mean that it could be possible to create both backend and frontend for realtime apps in python?

That would be so cool....

inglor4• 11 years ago

Does it mean that it will be possible to create realtime apps with python?

I'm not sure what you mean by realtime but usually real time means something else and requires more deterministic behavior.

If you mean applications that use non-blocking IO to great extent then yes, but this depends a lot on the ecosystem to provide libraries for async file io, database io and so on. The Python language left good infrastructure to use though.

Now that WebAssembly is coming - doesn't it mean that it could be possible to create both backend and frontend for realtime apps in python?

Not anymore than we could before with asm.js unfortunately as seen in http://repl.it/languages/python3 the problem is wrapping and the library, I hope a company "picks up the glove" and makes "Python for the frontend", sharing Python code between backend and frontend could be amazing.

bdarnell3• 11 years ago

It's always been possible to create realtime apps with Python. Callbacks have been available since the beginning, and frameworks like Tornado and Twisted have been doing something very similar to the new async/await functionality with generators since Python 2.5 (or maybe even older). The async/await keywords are a nice performance improvement and work in a few places that "yield" doesn't, but they're not adding fundamentally new capabilities.

hamiltonkibbe2• 11 years ago

I don't know that it's possible to have a realtime application running in an interpreter (which makes no realtime guarantees as far as I know) running on top of a soft-at-best-realtime OS.

Scarblac• 11 years ago

As for Django, I think this is an exciting idea: https://gist.github.com/andrewgodwin/b3f826a879eb84a70625

jerf• 11 years ago

"Will it be possible to use it instead of node.js?"

It's been possible to use Python instead of node.js since before node.js existed. And by "possible" I mean "tons of people have been shipping code", not just "it's theoretically possible but nobody does it".

Node ported a well-established existing technique to Javascript, it didn't invent it.

epberry• 11 years ago

Me too. There will be less of a switch in mindset for me after working in JS for a while then switching to Python for the backend.

muraiki• 11 years ago

I admit to not fully understanding the async PEP (partially since I don't do much Python programming anymore). Are there any downsides to having to explicitly mark an async function using the "async def" syntax?

I'm thinking of how in Go I can launch a goroutine with "go foo()", where foo can be any function and not one that happened to be declared explicitly as async.

Or in Perl 6, I can "my $p100 = start { (1..Inf).grep(*.is-prime)[99] }; say await $p100;" This provides the similar blocking "await" keyword, but I'm still free to execute any arbitrary code inside the block denoted by "start".

I have a feeling that these questions are both answered by whatever coroutine functionality existed prior to the introduction of "async def", but that's an area of Python that I have no experience with. I'd appreciate any clarification that could be offered. Thanks!

Edit: re: the P6 example, it looks like it works somewhat like Python, where a "Future-like object" (from the PEP) is provided to await; in P6 start returns a Promise. However it looks like in Python I still need to wrap the enclosing function with "await def". https://www.python.org/dev/peps/pep-0492/#await-expression

rdtsc5• 11 years ago

Are there any downsides to having to explicitly mark an async function using the "async def" syntax?

Yes. One main one is splintering the library ecosystem. So some libraries are using Twisted, some are using base threading system, some run in Tornado, some will use async and so on. They are usually not mixable. One day in the distant future they will all support async, but that means updating all of them.

Special markers ("async" waits points/objects) for IO co-routines tend to propagate vertically through all the API layers. If, say, a low level library you found is returning a Deferred/Promise/Future or generates values, then top level has to handle that as well. Then its parent also has to handle it.

It is infecting the ecosystem with low levels details about how the IO needs special casing because of how the GIL works. Yes, it is more elegant in Go, Rust, Erlang, C#, Java etc.

BTW Python has something like it based on the greenlet library (eventlet and gevent libraries are based on). Those monkeypatch system libraries so it manages to hide this complexity, but there are other costs, unsupported or untested side-effects being one.

ceronman• 11 years ago

Think of async/await as syntactic sugar for doing the kind of asynchronous programming when you pass callbacks/continuations and have an explicit event loop. For example, assume that you have an http_request function that looks like this:

    http_request(url, callback)

If you want to make a request, get a value, and then make another request depending on that value, you would need something like:

    def callback(result):
        url2 = get_from_value(result)

        def inner_callback(result):
            print('Final result:', result)

        http_request(url2, print)

    http_request(url1, callback)

This is very complicated, specially because in Python it's not easy to inline callbacks like in JavaScript. And even then, it's sometimes messy and becomes a callback hell.

With async/await you could do something like:

    result1 = await http_request(url1)
    result2 = await http_request(get_from_value(result1))
    print('Final result:', result2)

The await keyword doesn't block like in Perl 6. It suspends the execution and returns the control to the even loop. Then the event loop calls other functions in response to other events. All this happens in a single process and a single thread.

This model is more explicit than the Goroutines of Go or the multiple threads of Perl 6. But it has the advantage that the user has more control over when the execution control flows between different parts of the program. For example, if I have this code:

    await request1()
    # some synchronous code
    await request2()

I am completely sure that nothing will run while the code in the middle of the two request is running. This simplifies a lot of synchronization problems.

bdarnell3• 11 years ago

Goroutines are, for all practical purposes, threads. They're cheaper than real threads but they still have all the subtle complexity that comes with shared-memory preemptive threading (everything shared needs to be appropriately locked, scheduling is non-deterministic and can happen any time, etc). Async/await is different; context switches can only happen at an `await` expression. This greatly reduces the number of possibilities that must be considered and simplifies the task of concurrent programming (for more, see https://glyph.twistedmatrix.com/2014/02/unyielding.html).

In Python, the closest equivalent to `go foo()` is `threading.Thread(target=foo).start()`. Or you can use gevent to get goroutine-like lightweight threads, but I find this is works better in go than in python because the entire language is designed around it. In python it's common to find libraries that are not compatible with gevent's monkey-patching and so you lose concurrency in a way that is difficult to detect or guard against.

rdtsc5• 11 years ago

Python could do green threads (IO-base coroutines) for a long time -- see eventlet, gevent, even Twisted's inlineCallback pattern. There are probably others, there was even one based on "yields" as well.

duaneb• 11 years ago

Stackless python pairs well with green threads. Very, very well. I believe pypy based a lot of their initial work on the design of stackless python.

Spiritus2• 11 years ago

One thing I don't really get, why is the "async" keyword necessary at all? Couldn't just the mere presence of "await" in the function body implicitly make the function "async"?

balls187• 11 years ago

Self documenting code for one.

I suppose you could follow the .NET Naming convention of appending "Async" to all routines.

skrebbel• 11 years ago

I guess the reason is that it changes the type of the return value. Compare:

    def moo():
        return 5

    async def oink():
        return 5

moo returns 5, oink returns an awaitable.

For a human, without the 'async' marker, you'd have to scan the entire function source to know. I guess also type checkers and documentation tools like to have an idea about the return type.

svieira• 11 years ago

This is directly addressed in the PEP [1] - the short of it is, without the `async` modifier if you have two functions:

   def important():
       await something()

   def something():
       # ... stuff ...
       await something_else()
       # ... more stuff ...

and you refactor `await something_else()` into a new method then you've changed the return type of `something` from `Awaitable[ReturnType]` to just `ReturnType` and `important` will break at run-time when it tries to `await something()`.

  [1]: https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword

Ultimatt2• 11 years ago

C# has the reactive extensions API https://rx.codeplex.com/ which is the inspiration for the Perl 6 concurrency model which is also quite advanced https://www.youtube.com/watch?v=JBHsdc0IVIg So not sure why you think Python is somehow unique in doing all this.

chc• 11 years ago

I thought % formatting was the "old way" and we're supposed to use format() in Python 3. Strange that they're adding % support to bytes now.

lambda3• 11 years ago

As long as % formatting still works, it is in many ways more convenient that format. '"Foo: %s" % bar' is a lot shorter to write than '"Foo: {}".format(bar)'. And % formatting is still built into logger, so if you use logger heavily you will probably be used to it.

While the format() mini-language is nicer in a lot of ways, the convenience of % and its ubiquity mean that I generally use it more often. The main time when I reach for format() is when I need more detailed control of column widths and the like, which you can do with things like "{var:{width}}".format(var=var, width=width)

kbd2• 11 years ago

Personally I dislike % formatting even if it's a few characters shorter to type. .format is superior in every other way and has consistent-with-everything-else method call syntax.

Edit: I think it's largely the special flower overloading of the % operator that bugs me the most. If % was spelled .sprintf I'd like it more.

nhumrich• 11 years ago

If you read the PEP, it explains how they are adding it for easier python2 -> python3 migration

kbd2• 11 years ago

Thanks for bringing this up. I asked about this on reddit yesterday[1] and was linked the thread where they originally discussed this[2]. Basically, the goal is to make porting from 2 to 3 easier, and there's also a notion that printf-style formatting may be more natural than format-style formatting for bytes.

I still wish they were consistent and provided a .format method for bytes, especially since, as you point out, .format was supposed to be the "new way" to do formatting and this feels like a step backwards. And what about backward compatibility for porting Python 2 code that used .format to do formatting on what are semantically byte strings?

[1] https://www.reddit.com/r/Python/comments/3c7lne/python_350b3...

[2] http://thread.gmane.org/gmane.comp.python.devel/144712/focus...

jakob223• 11 years ago

How much of this is planning on being backported to python 2?

ubernostrum• 11 years ago

Python 2 is feature frozen. 2.7 gets bugfixes and security improvements and that's it; new language syntax/features only happen in 3.x.

ngoldbaum6• 11 years ago

Except large chunks of the 3.5 standard library has been backported and are available via the cheeseshop.

lambda3• 11 years ago

Lots of good stuff in here.

Python 3.5 may actually be what finally convinces me to move to Python 3 (that, and the end of Python 2.7 support coming up in a few years).

The new async stuff, finally fixing some of the problems with byte strings that made them not really an adequate replacement for Python 2.x strings, and os.scandir is a substantial performance improvement over os.listdir plus calls to stat (though os.scandir, at least, is also available in Python 2.7 via PyPI).

inglor4• 11 years ago

Let's not forget https://www.python.org/dev/peps/pep-0484/ which looks really promising

ngoldbaum6• 11 years ago

For what it's worth, even if the core Python developers discontinue support for 2.7 (and they might not, it was originally supposed to EOL this year), there is enough Python 2.7 code running in the wild, making people money, that some entity will certainly take over maintenance responsibilities. Python 2.7 is here to stay for a very long time if only because of the huge amount of legacy code written in it.

Animats2• 11 years ago

"Python 2.7 is here to stay for a very long time if only because of the huge amount of legacy code written in it."

True. It might even outlive Python 3, just as Perl 5 looks like it will outlive Perl 6.

jason_s• 11 years ago

I hate the new @ operator for matrix multiplication, it just looks ugly. (And I use numpy matrix math all the time.) But whatever.

inglor4• 11 years ago

As a fellow numpy user - why? I would love having sugar for this, I use it all the time. Introducing a single operator to overload seems odd but it's definitely a pragmatic solution to making code more readable. What would you do instead?

Sanddancer• 11 years ago

Personally, I'd have made it a diglyph of some sort, like :* or ^*, to make it obvious to someone looking at python math code for the first time that this is a multiplication operator. @ just leaves a weird taste here.

noreasonw• 11 years ago

In R %*% is matrix multiplication.

eliben• 11 years ago

What alternative would you propose?

Given that having an operator for this was deemed important for more readable numeric code, and that * was already taken, there weren't too many fabulous choices left.

perlgeek• 11 years ago

Also it feels like wasting a character on something that only a small number of users are likely going to need.

sherjilozair• 11 years ago

Not necessarily. You can overload @ to do some other kind of object combination, which fits your paradigm better. You're not tied to using it for matrix multiplication.

mcherm• 11 years ago

https://www.python.org/dev/peps/pep-0465/#choice-of-operator

sdenton42• 11 years ago

Python 3.5: In which your email address became syntactically correct.

wldcordeiro2• 11 years ago

These kinds of advancements in the language are why I'm glad I've switched to Python 3, Python 2 needs to die already.

Walkman• 11 years ago

Python 2 will never die. Ask Kenneth Reitz for example. He says he will use Python 2 forever. There are a lot of companies with huge codebases who never will port to Python 3.

rudolf03• 11 years ago

This is the unfortunate truth. My company still uses Python 2 solely, not a single line of Python 3.

Unless there is some major breakthrough in porting tools (in combination with more and more libraries supporting 3), Python 2 will still probably be the only Python version in many enterprises for the next 10 years.

dragonwriter• 11 years ago

Python 2 will never die.

Everything dies.

Ask Kenneth Reitz for example. He says he will use Python 2 forever.

So, assuming this currently-stated intent never changes and Reitz is immortal, Python 2 will always be in use.

The first of those premises is somewhat suspect, the second even moreso.

There are a lot of companies with huge codebases who never will port to Python 3.

That's probably true. Its less likely true that those companies will never port off Python 2.

dr_zoidberg2• 11 years ago

In my job we have some big projects on Python 2 and Cython gives us access to some of the "new Py3 features": types (real types, not just hints) and async IO (through C primitives). We eventually chose Cython because at least we can get big speedups (depending on the code and how much can be properly cythonized, of course). With Py3 it was a change to the codebase (which we also had with Cython, of course), to get approximately the same performance, maybe slightly less, maybe slightly more.

kozukumi2• 11 years ago

While there are some Python experts around I have a question; in the world of Python 3 does it matter between 32-bit and 64-bit anymore? I remember a while back the only real option was 32-bit as most libraries were 32-bit only, especially on Windows where Python wasn't fantastically supported compared to Linux/OSX.

tdicola• 11 years ago

If you're downloading pre-built binaries like on Windows then unfortunately yes it does matter if you're using 32-bit or 64-bit libraries. Windows in general has always been a bit of a second class citizen with Python library support, although it will hopefully improve with easier access to compilers (there's a free version of visual studio tools for python library compilation apparently now). If you just want a lot of libraries that 'just work' you're probably best served using a pre-built distribution like Anaconda. If you're more adventurous this page has all kinds of pre-built libraries for Windows in both 32 and 64 bit: http://www.lfd.uci.edu/~gohlke/pythonlibs/

Alternatively it's insanely easy to spin up a Linux VM with vagrant and just compile and run python scripts from there. Everything will pip install without any issues.

wldcordeiro2• 11 years ago

Windows is a second class citizen for most language library support, that being said though Python does a great job of having the most recent versions available for Windows. The same can't be said of Ruby unfortunately. :(

ambivalence• 11 years ago

Well, there's also type hints that so far we haven't added to "What's New" :)

https://www.python.org/dev/peps/pep-0484/

benhoyt• 11 years ago

Yes. I'm not entirely sure why this was posted now, as it's a couple of months from 3.5's release date yet, and What's New is not up to date. On python-dev, David Mertz just volunteered to write/update this "What's New" page, somewhat sponsored by the company he works for, Continuum: https://mail.python.org/pipermail/python-dev/2015-July/14065...

Animats2• 11 years ago

Type hints are not Pythonic. The syntax is ugly. It adds "stub files", like C's ".h" files, something newer languages such as Go and Rust don't need. Some of the proposed syntax is even in comments, because it didn't fit the language. But something had to be done to meet competition.

The problem Python faces is that Python's Little Tin God painted the language into a corner. He insisted for years that everything in Python had to be fully dynamic. This limited performance.

Then came competition in Python's space. Go came along, Javascript got traction on the server, and Python faced a real threat. Both of those can beat Python on performance, sometimes by huge margins, and they're useable by Python-type programmers used to the freedoms of scripting languages.

The Python 3 debacle had already thrown the Python community into disarray - most of the production work is still on Python 2, which was supposed to be abandoned but refused to die because the cost of conversion was much higher than expected. Meanwhile, Python 3 takeup was far less than expected; numbers like 10-20% come from downloads, and production use is probably less. Another incompatible change in the syntax would be rejected by the market. An incompatible change to a "typed Python" wouldn't fly.

So we get PEP 484, which is ugly but might help Python survive. Might.

hyperpape• 11 years ago

As far as I know type hints don't affect Python's performance, because they're optional and not yet used by the runtime.

In any case, JavaScript is dynamically/unityped, so its performance is unlikely to provide a reason for type annotations. The real problem here is that PyPy is still maturing, and can't be used by many people because of legacy C extensions.

rpcope12• 11 years ago

So we get PEP 484, which is ugly but might help Python survive. Might.

This seems really hyperbolic. I'm not sure Go really competes in the same space as Python; also Python is so well entrenched as the successor to Fortran in the scientific computing space, that this would seem to guarantee continued relevance far into the future. It's probably like a lot of things, it may not be as sexy or break neck fast (though PyPy is staged to change that, and maybe we can also feasible get a non-sucky version of IronPython or Jython), but it's pretty much everywhere now, kind of like Perl, C, PHP, Java....

silverpikezero• 11 years ago

Your comment is grossly overstating the issue, and unnecessarily inflammatory. I have reviewed the type hints syntax, and it's not so bad. It's doubtful that anything nicer can be implemented on top of what you have clearly pointed out is a dynamically typed language.

Your criticism of Python being dynamically typed is also misguided. There are many benefits of having a dynamically typed language, which I won't bother to enumerate here since this subject gets beaten to death regularly on HN. It is a good choice to make this design decision up front and honor it as the language grows. Guido is not a moron; he knew there would be performance implications. Nobody today chooses Python for it's native performance anyways (although Cython and PyPi have made great strides for common cases).

The value of Python is not in performance, it's in the language simplicity, large ecosystem, and highly developed libraries. There are some disciplines such as machine learning and quantitative finance which are all but predicated upon Python, with excellent results. Comparisons to Go and JS are incongruous; those languages have other benefits which would make them good choices if things like concurrency (Go) and very high level abstractions (JS) are important.

The Python 3 transition was indeed rough, but in no way is it a "debacle". The community is not in "disarray"; that's absurd. The transition to 3 will happen eventually, and indeed this lethargy was caused by deliberate breakage in language features. Maybe not the best decision in hindsight, but far from this cataclysmic fantasy you seem to be depicting.

kevin_thibedeau3• 11 years ago

How's this for a debacle: I just ported a 2.6 app to 3 and it only required fixing one line due to a change in library behavior. 2to3 took care of the rest.

zeckalpha• 11 years ago

That's not wholly true. While stub files are proposed, they're not required to use this feature.

Ultimatt2• 11 years ago

Perl 6 has full gradual typing YAY https://www.youtube.com/watch?v=id4pDstMu1s

craigyk• 11 years ago

Looking good. Soon I'll just be waiting for the announcement that they have sped things up 10X, and I might finally be able to let go of my Python ennui.

berkerpeksag• 11 years ago

Note: The "What's New" document is not up-to-date yet. See https://hg.python.org/cpython/file/3.5/Misc/NEWS (warning - it takes time to load) for a more complete list.

bite_victim• 11 years ago

I am a bit reluctant about asking this but where do we stand in terms of real world performance? This is such a broad topic, I know, but surely there has to be some in-house metric system about this (perhaps popular web and scientific frameworks, game dev frameworks etc).

dalke2• 11 years ago

The answer is "all over the place". It really does depend on your benchmarks and use case. There is no aggregate real-world benchmark. Such a thing is very hard to do, and not all that portable.

For example, in one performance intensive code that I use, while mostly written in Python, has a C extension, which itself uses inline assembly for the core routine. Similarly, most numeric calculations that use NumPy will end up doing BLAS / LAPACK / ATLAS calls, so the performance limitations aren't due to C. The switch from 2 to 3 isn't really going to affect these.

If it was seriously and consistently worse, eg, for Django, then it's likely we would heard about it by now.

Instead, there are reports like https://www.youtube.com/watch?v=f_6vDi7ywuA (slides at https://speakerdeck.com/pyconslides/python-3-dot-3-trust-me-... ) from PyCon 2013. At minute 30:54 / slide 16 the speaker evaluates the two and finds that the median performance was effectively identical for microbenchmarks, and ~5% slower on macrobenchmarks, though even then it varies from 27% slower to 13% faster depending on the specifics, and as the speaker points out, the tested packages aren't doing exactly the same things on both versions.

In any case, that was 2 years ago.

88e282102ae2e5b• 11 years ago

An image-reading tool I maintain works 14 times faster in Python 3.4 compared to Python 2.7. I have no way of accounting for the difference though.

dr_zoidberg2• 11 years ago

This is probably the first time I heard of such a speedup when going from 2.7 to 3.x, and the fact that you can't account for it makes it worse because we can't evaluate what's happening and if such speedup could be achievable on other projects :(

davesque2• 11 years ago

How come `bytes` and `bytearray` got old-style formatting? Aren't we trying to do away with that in favor of a `format` method?

theVirginian• 11 years ago

I think the general opinion is "use where appropriate but be consistent" which I tend to agree with. There are times where the simplicity of old-style formatting just gets the job done but it certainly has it's limitations and new style can be extremely powerful.

ngoldbaum6• 11 years ago

It's much more convenient to decode bytestreams (e.g. for a wire protocol) using % formatting. The lack of % formatting for bytes was a huge impediment for porting mercurial to python3, just as an example.

davesque2• 11 years ago

Could you point me to a code example? It's hard to see how old-style formatting would facilitate decoding. Typically, I think of decoding as being a process which deconstructs something; not like formatting, which is really more about constructing.

kelvin0• 11 years ago

Am I the only one surprised by Zipapp (since 2.6?). Wish I had known about this before ... https://docs.python.org/3.5/whatsnew/3.5.html#whatsnew-zipap...

uxcn• 11 years ago

This is nice. I actually used to manually create zip archives and prepend a shebang.

gambogi• 11 years ago

Can that handle statically linking dependencies with FFI calls?

talaketu• 11 years ago

does PEX still have value?

https://pex.readthedocs.org/en/latest/

mackwerk• 11 years ago

It's going to be hard to get used to having the function body 'less' indented than `def`.

ergl• 11 years ago

Maybe this is obvious, but what would be the difference between the new async / await def and the current @asyncio_coroutine decorator?

Or between the new async / await and the current asyncio?

bcheung• 11 years ago

Does it really matter if nobody is going to upgrade past 2.7?

a3n• 11 years ago

Does anyone retain their legacy Python2, write new in Python3, and communicate between processes, read/write from a common database, and just in general keep 2 and write new in 3?

timetraveller• 11 years ago

Someone somewhere is.

mkempe• 11 years ago

We should appreciate the contributions of Serhiy Storchaka, who appears to be in the Ukraine (or Kazakhstan, or Tajikistan).

What’s New in Python 3.5

Related Stories