Python 3 Q&A by a prolific core developer

http://ncoghlan_devs-python-notes.readthedocs.org/en/latest/python3/questions_and_answers.html

by hynek • 14 years ago

145 94 14 years ago

ncoghlan_devs-python-notes.readthedocs.org

toyg• 14 years ago

Unreadable on iPhone, what's up with its stylesheet? Safari won't even zoom...?

hynek13• 14 years ago

Yeah, it seems to be the default RTFD style…try: http://www.instapaper.com/text?u=http%3A%2F%2Fncoghlan_devs-...

thu• 14 years ago

The following situation happens with other data than just unicode. This seems an argument for static typing in general.

  The reason this approach is problematic is that it means the traceback for an unexpected UnicodeDecodeError or UnicodeEncodeError in a large Python 2.x code base almost never points you to the code that is broken. Instead, you have to trace the origins of the data in the failing operation, and try to figure out where the unexpected 8-bit or Unicode code string was introduced. By contrast, Python 3 is designed to fail fast in most situations[...]

qznc3• 14 years ago

I agree that static typing would helps for such bugs.

However, not "in general". The general problem is that bogus data leads to an irreparable situation. To fix that bug, you have to find out who broke the data. Consider for example, a circle in a linked list. You would need quite an expressive type system to prevent that statically (dependent types might suffice? Theorem provers are the ultimate hammer).

zanny2• 14 years ago

Different implementations for different use cases in different situations :P

DannoHung2• 14 years ago

Uhh, how would you make a linked list with a circle in it with the Haskell List type?

gardentheory2• 14 years ago

Well the lighter solution and more Pythonic solution is strong typing. If a Unicode string and a byte string could not be combined then the problem would not exist AFAIK. For example "a" + 1 is an error, why is u"a" + "s" not an error?

xerula• 14 years ago

But b"s" + "a" is an error in Py3.

qznc3• 14 years ago

This just gives a different error message. It does not help to find the error source.

The programmer would probably just try to fix it via str(u"a")+"s" or u"a" + unicode("s").

masklinn2• 14 years ago

> For example "a" + 1 is an error, why is u"a" + "s" not an error?

"a" + b"s" is an error in Python 3.

slurgfest10• 14 years ago

It is an argument for failing fast in general. Static typing isn't guaranteed to pick up every mistake at its origin, because far from every mistake is of the "int rather than string" variety.

Since you have brought up the general case, the origins of many bugs involve something sophisticated going wrong which would require you to encode much of your program (hopefully not too redundantly) into a Turing-complete type system. Rather than stuffing square pegs into round holes, you could just write the appropriate error handling code (or even just asserts).

earl• 14 years ago

Please:

   alias hncomment='fold -w 77 -s | sed "s/^/   /" | pbcopy'

   use:
   echo "blah blah blah" | hncomment
   command-v

Thus:

     The reason this approach is problematic is that it means the traceback for 
   an unexpected UnicodeDecodeError or UnicodeEncodeError in a large Python 2.x 
   code base almost never points you to the code that is broken. Instead, you 
   have to trace the origins of the data in the failing operation, and try to 
   figure out where the unexpected 8-bit or Unicode code string was introduced. 
   By contrast, Python 3 is designed to fail fast in most situations[...]

zanny2• 14 years ago

I think all the outrage is that Python 3 behaves like a new language in how everything needs to be ported (albeit at a magnitude less complexity than going up the language chain or across the playing field, like C to Ruby or C# to Java) but it still incurs the new language cost, while being the new version of the old language.

If people just thought of Python 3 as the replacement to broken text model Python, like how XHTML tried to supplant HTML and failed, and then HTML5 supplanted XHTML with some quirks, the same thing happens here.

Also, I adore the comments about the GIL. I understand the benefits of using a single language in every circumstance, but (even) a JIT scripting language is not sufficient for all problems. I guess it depends on if a polyglot code base is more complex than handling multithreading and the corresponding memory complexity and gimmicks, and that is developer centric.

Still, I would much rather just stick processor bound code in openMP C functions and call those from Python when need be. It seems like the "right" answer to the performance problem, without losing much productivity.

smeg• 14 years ago

Python3 and Perl6 can both fuck off.

I look forward to the day PyPy is considered the real Python. Look at PyPy's homepage (http://pypy.org/), doesn't even mention Unicode as a significant feature. Instead it talks about speed, security, concurrency, and compatibility with the current real Python (2.7.2) - all the things real Python programmers care about and expect the Python developers to focus on.

PyPy may be some way off but I want to find the developers and hug them for setting the right vision and trying.

I just donated $50 and if you hate Python3 you should too.

sho_hn2• 14 years ago

Or if you really like Python 3 (as I do!) you can donate those $50 to Python 3 support in PyPy: http://pypy.org/py3donate.html

Deestan• 14 years ago

> all the things real Python programmers care about

Guess I'm not a real programmer, then, since I find proper Unicode handling to be a welcome feature, and find the implementation in Python2 is painful.

Marwy• 14 years ago

I think you should take a walk, and then come back and read this ridiculous stuff you wrote.

lclarkmichalek• 14 years ago

Any reason they can "fuck off"? Anyway, you're incorrect about pypy. "Real" python developers will likely not run their code in a sandboxed environment, so PyPy's security advantages are minimal, added to the fact that your sandbox is only as good as its configuration. The concurrency is interesting, but there are numerous other solutions available, and that alone makes it unlikely that the de facto implementation of the language would include one. Claiming that PyPy is better on compatibility than CPython is ridiculous; last time I looked, anything using a C extension had to be rewritten to use ctypes. On speed, yes PyPy is impressive. But in python, anything that is particularly CPU intensive should probably be written as a C extension and PyPy takes a hit when using ctypes iirc. In general, I don't see the point of people trying to optimize generic python code. Most code never gets profiled, and most code isn't a problem. At least, not a problem enough to switch interpreter. Definitely not a problem enough to ignore the massive problem that is python 2.x's unicode implementation.

dbaupp2• 14 years ago

It's easy to dismiss nice builtin Unicode support if you are an English speaker in an English speaking country where all applications you write can conveniently ignore other alphabets without meeting any complaints or problems.

sutro• 14 years ago

Unconvinced.

ash3• 14 years ago

Why can't I use Python 2 modules from Python 3 program? Is there any effort to make it possible?

hynek13• 14 years ago

There is no such thing as "Python 2" or "Python 3" modules. There are just modules that will go belly up because they are incompatible with one or another.

I presume you mean that you can’t access your old Python 2 modules from a new Python 3 installation. That’s just because Python installations usually don’t share their modules (i.e. site-packages). You can try to install them using Python 3 (usually just python3 setup.py install) and see if they work or not.

lrem3• 14 years ago

There's a deeper question here: why doesn't the interpreter contain a compatibility layer for the old code? There's no really good reason for the old modules to go belly up in the new interpreter. At least no other than the man months needed to implement that layer. But then, that probably would still be orders of magnitude less than migrating all the libraries that keep people from migrating...

dmbaggett7• 14 years ago

From the article:

Back in reality, though, complaining about the GIL as though its a serious barrier to adoption amongst developers that know what they’re doing often says more about the person doing the complaining than it does about CPython.

It was a solid, well-argued piece up to this point. You do yourself and the Python community a disservice by writing off your critics as ignorant. It sounds petulant and childish, and is wrong.

There are valid arguments on both sides of the GIL argument, but neither side's advocates are ignorant or bad programmers.

sho_hn2• 14 years ago

I have to agree here, despite the two of us rating the importance of the GIL problem quite differently in yesterday's discussion: A better strategy against the GIL FUD is plain old education, not trying to make it a taboo topic. Especially programmers who do know what they're doing won't take too kindly to a community where that becomes prevalent.

hynek13• 14 years ago

I find Nick arguing pretty much to the point and the quote above was ripped from it’s context which is a huge section balancing the arguments against each other.

tldr He argues there are better ways for scaling out than threads and removing the GIL would have enormous consequence _throughout_ the whole code base so it’s removal cannot be warranted.

Reducing 10 paragraphs and 11 bullet points to “only stupid people use threads” is just poor style.

slurgfest10• 14 years ago

Let's be honest: you are upset because you have sometime committed to saying the GIL is a huge deal and you feel insulted by anyone saying it isn't, which is why you are using emotionally loaded words like 'petulant' and 'childish' and 'ignorant' and 'bad'.

The GIL is the single biggest target for language-advocacy FUD against Python by advocates of other languages. I would be a rich person if I got a dollar for every time I saw someone trashing Python as a toy language because of the GIL, without significant knowledge of how to use Python. It's just a much huger issue to someone without real Python experience than it is to people with Python experience.

Setting that aside, there are a vast number of use cases where someone might try to use threads when actually that's not a good solution. It actually isn't incredibly easy to come up with cases where the GIL is this huge fatal flaw. It's really sad how many times I have seen people ranting about the GIL and totally unaware of multiprocessing, unable to give a technical reason why they can't try processes, unaware of greenlets, wholly unaware that the GIL does not exist in Jython/IronPython, etc.

I'm sure that doesn't apply to you, which would put you in the minority. Thus, "often says more about the person doing the complaining".

Threads should be used judiciously, because they dramatically increase the complexity of a program and the difficulty of debugging and reasoning about it. Shared-everything is a great way to blow your foot off if you don't specifically need it. So if you throw threads at every problem indiscriminately, then that really is a weak point in your programming. (Note: I am not saying that using threads is always a bad idea)

If you are a good programmer then you should already know, and not be offended to hear, that threads are a tool of specific applicability, not a panacea to scale up everything.

dmbaggett7• 14 years ago

Thank you for telling me what was inside my head. I was simply unaware of my emotional state and motivations until you helpfully pointed them out to me.

But seriously, "good programmers don't use threads (much)" is your counter-argument?

bwood7• 14 years ago

In my experience with Python, the issue of the GIL comes up far too frequently. I use Python first as a prototyping environment, and once I have a proof of concept I like to optimize for performance (without re-writing the entire program in a more appropriate language). Most of the stuff I do is CPU-bound, but also involves large amounts of data. So, without writing C code as an extension, I can either suck it up and deal with single-core performance, or I can use the multiprocessing module and hope that inter-process communication isn't too expensive or difficult to code (and no, it's not always as simple as just using multiprocessing.Queue). I am under no delusion that the GIL will ever be "fixed", but I am disappointed that I spent years working with Python before I got into more CPU-intensive tasks and eventually hit the brick wall that is the GIL. And of course, comments like you mentioned, which attack my competence as a coder, are not constructive.

pwang2• 14 years ago

I'm interested to hear about your CPU-bound Python programs that also involve large amounts of data. Normally such problems are I/O bound and are better solved in a data parallel way. (Graph problems can sometimes be an exception, but those are still generally I/O bound.)

For writing extensions, have you considered Cython?

seunosewa6• 14 years ago

From the article: "The only downsides of this approach are that it means that CPU bound Python code can’t scale to multiple cores within a single machine using threads, and that IO operations can incur unexpected additional latency in the presence of a CPU bound thread."

Response: The only downside? As a Python user suffering from JVM envy, I have to say that that's a SERIOUS downside!

Here's why: (1) Python is slow. Almost any real life program in pure Python will have CPU-bound components. (example: BBCode parsing on my forum).

(2) Most programs that need to scale won't need to scale beyond a single machine (the most active web forum on my continent runs on a single quad core server)

Therefore the need to be able to scale CPU-bound python programs to multiple cores on is a very real need. Even though we accept that removing the GIL is hard, let's not insult real-life Python users by suggesting that their needs are not real.

hynek13• 14 years ago

The article doesn’t say multi-CPU scaling isn’t necessary. It says that threads are usually the wrong answer anyway.

There are great process based ways to scale out – look no further than Erlang to see it’s true.

seunosewa6• 14 years ago

I use Python for my day job. I've experimented extensively with Java and Scala. I must say this: Threads are AWESOME. Threading is the most flexible model of concurrency because you can can build programs that EFFICIENTLY implement "alternative concurrency models" like message parsing and STM with threads. Threading is supported natively by every OS. And fast.

Erlang is hyped as the ideal model for concurrency, but in practice is a niche product that's primarily useful for programs that are almost pure IO - chat servers, routing components like proxy servers and packet switches.

The Erlang model does NOT apply to python, anyway, since Python processes are nothing like Erlang processes. Unlike Erlang processes, Python processes are very heavyweight and message passing between them is costly.

MetaCosm• 14 years ago

SMP Erlang DOES use threads, it then uses green-processes and a custom scheduler to schedule processes fairly and preemptively across those threads. It doesn't use like 1 thread per process, but threads are a key component of how it works.

It is absolutely nothing like spawning OS level processes. They are micro-processes, green-processes than live inside the Erlang VM.

lrem3• 14 years ago

Yeah, but do most programs that need to scale also need shared context? Otherwise, what are the big downsides of processes vs threads?

seunosewa6• 14 years ago

Many programs can benefit from shared context. You can push all the shared context to your database, but it's often helpful to keep around some shared data structures for performance reasons. For example, you can cache pure Python functions using functools.lrucache, and share such caches between threads, but such caches can't be shared between processes. In-process data structures like dicts and lists are much, much faster than alternatives like memcached and redis because they avoid the overhead of IPC and deserialization, and they are also easier to use since they are built-in.

slurgfest10• 14 years ago

You are writing a forum which includes BBcode parsing, and Python's CPU usage is your bottleneck? Really?

You are devoting multiple cores to parsing an individual user's BBcode?

This is your example of a real need to remove the GIL?

seunosewa6• 14 years ago

No, that was just an example of a unexpectedly CPU-bound operation. My point was that you can't assume your Python program is not CPU-bound. Every time you have to do something non-trivial in Python, performance could become an issue.

dmbaggett7• 14 years ago

In the last few hours, this has been silently added to the post, and I cannot post a reply on that site:

Jumping on the internet to say that “they” (specifically, the people you’re not paying a cent to and who aren’t bothered by the GIL because it only penalises a programming style many of us consider ill advised in the first place) should “just fix it” (despite the serious risk of breaking user code that currently only works due to the coarse grained locking around each bytecode) is also always an option. Generally speaking, such pieces come across as “I have no idea how much engineering effort, now and in the future, is required to make this happen”.

Fine, then don't bother. But don't insult us or call foul for our pointing out the downside impact on us of this decision. Your assumption that we are naive rubes who don't know how to code is really, really wrong.

I have a very good idea how much engineering effort would be involved in fixing the GIL, and I am well aware that Python has involved many person-millennia of gratis work, and am appreciative of both. However, I still disagree with the Python devs' obviously entrenched position that fixing the GIL isn't worth the effort, and I will continue -- even when shouted down by the likes of you -- to advocate for the GIL's removal or some equivalently good solution. (As I said earlier, I am not opposed to STM solutions, but the current one performs unacceptably without special-purpose hardware.)

Why? Because I have a single, selfish interest in this. I depend heavily on Python now, and want the language to be better. I have written many lines of Python 2 code that rely on the threading primitives in the standard library. Perhaps it was foolish of me to expect that the threading model offered by the standard library, modeled on Java's threading primitives, would some day work in the same way as Java threads do in practice. Nonetheless, I am left with a real world problem: my CPU-bound threaded Python code does not scale well to multiple cores. I need the GIL fixed, or to rewrite my Python code, or to migrate to another language that supports the standard model of threading programming that real-world programmers have been using for several decades, and which has built-in support from all major operating systems. Or, sure, wait for STM to be ready for prime time and migrate my thread-based semantics to the new STM-based semantics.

The best path right for me right now is migration to Jython or IronPython. But then we are still unsupported orphans, living in the third world of Python 2.X.

I guess it comes down to: do you want people to actually use this language to write programs they want to write, or do you want Python to be an advocacy platform for "correct" programming? Python's pragmatism has always appealed to me, so the ivory tower reaction to the practical concerns around the GIL really seem dissonant. (And this is coming from an MIT AI Lab Lisp guy who would rather write everything in Lisp. But Lisp lacks Python's wonderful, high-quality third party libraries and developer-centric pragmatism regarding unit testing, documentation, etc.)

I know you are tired of hearing people bitch about the GIL, but, really: people write multithreaded programs. They should work as advertised, using native OS threading primitives and taking advantage of the native OS thread scheduler. Why does Python offer threading primitives if the language is not meant to support, from a practical standpoint, multithreaded programs?

Flimm2• 14 years ago

"They should work as advertised, using native OS threading primitives and taking advantage of the native OS thread scheduler."

Just to be clear, Python does use native OS threading primitives, and it does make use of the native OS thread scheduler. Also, Python does support, from a practical standpoint, multithreaded programs, but only if the program is not CPU-bound. I think you could rewrite that paragraph to make the role of the GIL clearer.

dmbaggett7• 14 years ago

Fair enough;the distinction is subtle. Python uses native system threads and of course such threads are scheduled by the OS. But in practice, the GIL allows only one thread to run at a time unless the GIL is subverted via C extensions or similar. I get why this is, given Python's architecture and the legacy of the GIL (which dates to the late 90s, when multicore machines were relatively rare and expensive).

But it's still not the case that multiple threads "normally" run in parallel, with the OS ensuring fairness, which is what (I think) most programmers would expect threads to do in a general-purpose threaded language.

old-gregg• 14 years ago

...the standard model of threading programming that real-world programmers have been using for several decades...

What standard? And whose "real world"? The need for threads has always been controversial even among OS kernel devs. UNIX/Linux/BSDs have twisted and non-trivial threading histories peppered with religious wars similar to this one. And which "several decades" are you talking about?

There is no such thing as "standard threading model". To some, a thread is just a flavor of fork() with a wrong parameter and plenty of "real world" programmers continue to believe that kernel-level threads is a hack. And please, do not make it sound like Python threads are useless. Far from it.

Python threads are not what you are used to. That's pretty much TL;DR of your comment.

...Python's pragmatism has always appealed to me, so the ivory tower reaction to the practical concerns around the GIL really seem dissonant...

I feel like they are being dragged into it though. The original motivation behind GIL support has always been a pragmatic one: removing GIL will make the entire codebase more complex, harder to hack on and will complicate and slow down the development/maintenance of the libraries. That's pretty pragmatic.

But a fairly vocal groups of users started to claim, similarly to you, that programming with threads is supposed to work like they expect it to work according to make-believe "threading standard", to which GIL supporters (correctly, IMO) replied that shared memory + locks is not the only/best approach to concurrency. It is easy to be offended by this answer but it doesn't invalidate their point.

dmbaggett7• 14 years ago

Python threads are not what you are used to. That's pretty much TL;DR of your comment.

Correct. I am used to programming language threads that work the way computer scientists and programmers have typically described them -- for example, as in this (I hope uncontroversial) Wikipedia article:

http://en.wikipedia.org/wiki/Thread_(computer_science)

When I say "standard model of threading", I am not talking about nuances of call conventions to the underlying OS thread primitives. I am talking simply about running multiple streams of instructions, bytecodes, or other units of computation in parallel, within a single OS process.

cpeterso• 14 years ago

Python could avoid GIL problems with atomicity by implementing actor-based concurrency. Like Erlang or Rust (and unlike Go), actors could send deep copies of objects to each other using channels. Each actor would have its own GIL. To maintain compatibility, C extensions would only run on the primordial actor thread. After some testing, C extensions could opt-in to be multi-actor safe.

If Guido doesn't want to implement actor concurrency within the Python interpreter, someone could write their own Python host that allows multiple, independent instances of the Python embedded interpreter. The host would implement a C extension that allows interpreter instances to send deep copies of objects to each other.

gvalkov• 14 years ago

It's just one of the little things, but I'm still very conflicted about print as a function. The reasoning is sound, but I just can't seem to get used to it. It feels especially cluttered and clumsy with new-style string templates:

    print('ver: {}'.format(', '.join(str(i) for i in sys.version_info)).upper())

versus:

    print 'ver: {}'.format(', '.join(str(i) for i in sys.version_info)).upper()

Despite the more consistent naming (ConfigParser -> configparser), simplified api (iteritems() -> items()) and all other syntactic improvements, I somehow still find 2.x code more enjoyable. Writing small scripts in Python has kind of lost its charm for me.

This, of course, is all very subjective and I'll probably grow over it in a few thousand lines of code. I hope you're all less sensitive to the little things that annoy you.

wololo2• 14 years ago

I wish python 3 would make default parameters immutable. It seems to go against the "simple", "explicit" python philosophy, not to mention that (it seems) most people tend to learn about the feature through debugging.

description: http://effbot.org/zone/default-values.htm

hynek13• 14 years ago

Yeah sometimes I wish there were a warning whenever someone defines a function with [] as def arg.

wololo2• 14 years ago

I know pylint and pychecker do this, but not pep8. Opinions on those?

Python 3 Q&A by a prolific core developer

Related Stories