Curing Python's Neglect

http://zedshaw.com/blog/2009-05-29.html

by twampss • 17 years ago

149 91 17 years ago

jnoller3• 17 years ago

Patches, PEPs and bug reports welcome. If you see a problem, it's polite to at least assist in fixing it.

req22• 17 years ago

Though I agree these things are best addressed patch in hand, it's worthwhile pointing out that there are design issues that people do need to be aware of to prevent future occurrences of the idiosyncrasies he is mentioning.

jnoller3• 17 years ago

I concur; which is why I mentioned a PEP to solidify some of the idiosyncrasies "out" - akin to PEP8, propose an API style guide.

pbiggar• 17 years ago

I hate that when criticised, people demand that the criticizer fix the problem they're pointing out. I realize your request is less strong than that, but fixing "your" software is not "my" problem. Especially when you consider that nearly every piece of software I've ever used has bugs, and most of the bug reports I submit are ignored.

jnoller3• 17 years ago

I'm sorry; No. I'm not demanding he fix it; I'm asking he do something more constructive than nothing. Python isn't "my" software - it's "our" software. Zed's smart enough to contribute and help fix things - I hold him to a higher standard than someone just walking in off the street.

Python-core does not ignore bug reports, especially ones which really fix things, and come with a patch. Asking him to help isn't out of bounds.

tow212• 17 years ago

Complaints around infelicities in the Python stdlib are fair enough - I don't think anyone would defend them, but what can you do? You can't change published APIs.

Fos most of the rest of his issues, I think Zed's problem is that he thinks there's one obvious answer, but lots of other people would disagree with his one answer. Python tends to be either very opinionated, or very agnostic. Where there's One Way To Do Things, that's what you must do; except where there's obvious disagreement in which case it doesn't bless a single answer.

API doc autogeneration; I can guarantee that if anyone came up with a default tool with default output such that "doctool module html_directory" was all you needed (and indeed people have) then there would be widespread, perfectly legitimate, disagreement on any number of choices in the design of it.

(Prsonally, I think API doc autogeneration is entirely pointless, and indeed fundamentally the Wrong Thing To Do for most Python libraries. When I come across pages of JavaDoc-ish API documentation, I groan internally and just look at the source instead.)

arohner• 17 years ago

>Complaints around infelicities in the Python stdlib are fair enough - I don't think anyone would defend them, but what can you do? You can't change published APIs.

You're absolutely right, but that doesn't mean Zed isn't on to something. How many different ways are there to open a subshell in python? There's popen, popen2, popen3, popen4 (I think), plus more (subprocess?). Yes, the past can't be changed, but it's still a good idea to stop and ask why there are so many mistakes like this.

Why is there a time module, and a date module, and a datetime module, and yet all three are broken?

The main question is What factors allowed the stdlib to get so quirky, and what can be done to fix that? Pointing out the stdlib is quirky is the first step in fixing the problem.

aston2• 17 years ago

Python 3 intends to fix a lot of the stdlib issues. For example, the subprocess module should obsolete all of the various popens.

http://docs.python.org/library/subprocess.html

stcredzero2• 17 years ago

If you have a quirky stdlib, this can be a big problem if its source is visible. This can be a tremendous source of bad examples to the community. This happened with the various Smalltalk images -- there was some rather good code, and then there was a lot of procedural/spaghetti code by junior programmers. A lot of it happened to get into places where it would be very visible to other junior programmers. (Database/OR tools.)

I think that a lot of this quirky stuff is due to the "uncool" factor of cleaning up such infrastructure level stuff. It's much cooler to be working on a proxy framework that talks to some new protocol. Making sure dates and times work well seems pedestrian. But it is actually tremendously important.

pmjordan• 17 years ago

Complaints around infelicities in the Python stdlib are fair enough - I don't think anyone would defend them, but what can you do? You can't change published APIs.

I was under the impression that Python 3 broke a lot of backwards compatibility, so could they not introduce new APIs with that? Or is that in fact what they did, and Zed is just complaining about 2.x?

I apologise if I'm mistaken on this, I've hardly used python at all. (though I keep meaning to learn it so I can finally stop writing bash scripts)

csbrooks2• 17 years ago

I apologise if I'm mistaken on this, I've hardly used python at all. (though I keep meaning to learn it so I can finally stop writing bash scripts)

Get crackin'. It's a breeze to learn, and you'll be writing more robust scripts than in bash in no time.

jmillikin2• 17 years ago

Python 3 removed old, deprecated APIs, but did not change any existing APIs. Probably the most extreme re-organization was the urllib/urllib2 merger, which was at its core just removing old stuff. Because Python is dynamically typed, the "protocol" of a library is extremely important, and the semantics of existing functions/classes are almost never changed.

Of Zed's complaints, I feel that two are misguided (time handling seems quite reasonable to me, and I've never seen an easier documentation system than Sphinx).

I think applying the "del" keyword to objects in a container is a mistake in the language, but that's how it was designed, so it probably wouldn't have been changed in the 2 -> 3 transition. It certainly won't be changed now -- the best that can be hoped for is for its use to be discouraged in the documentation.

Install/Uninstall of modules is fairly easy, but requires all concerned parties to work together. Otherwise you end up with half-installed, broken modules. There are PEPs in progress to work on this, but it's a social/political problem rather than a technical one.

Python's file-manipulation APIs are an ongoing calamity.

rabidgnat• 17 years ago

"API doc autogeneration; I can guarantee that if anyone came up with a default tool with default output such that "doctool module html_directory" was all you needed (and indeed people have) then there would be widespread, perfectly legitimate, disagreement on any number of choices in the design of it."

In my experience, the opposite actually happens. When something is written, people either use it and customize it over time, or they just find something else and say it didn't suit their needs.

It's when an extremely useful/general library is PROPOSED that it becomes a huge disagreement. This is, for instance, why C++'s Boost still doesn't have logging or a unified XML library, and won't likely have a garbage collection library before a C++ TR defines the interface: everyone argues over the color of the bikeshed, and the people who try to write the libraries get caught up in infinite microdiscussions over details.

Conversely, someone like Linus Torvald's success is that he just makes the things he needs/wants (ignoring bikeshed discussions), and people generally end up using his creations. When it comes to software design, a dictatorship is often more productive than a democracy!

"When I come across pages of JavaDoc-ish API documentation, I groan internally and just look at the source instead."

Amen :)

derefr2• 17 years ago

> a dictatorship is often more productive than a democracy!

"The only thing democracy has ever really been good for is making sure that things people don't like get vetoed. Democracy is no way to plan; it is merely a way to restrict one's choices, possibly to none at all."

aston2• 17 years ago

Hey Zed, it's mystuff.pop(4).

batasrki• 17 years ago

Is that intuitively the orthogonal method to append()? Not in my view. If I appended something, I'd want to remove it or delete it, not pop it. If I push'ed, then popping would be obvious.

randallsquared• 17 years ago

Well, it exists, which is what he was complaining about. One step at a time. :)

smanek3• 17 years ago

Just to be pedantic, orthogonal means perpendicular (i.e., 'at a right angle to'). It seems like you meant opposite/complementary/inverse, etc.

Sorry - people misappropriating math jargon into the mainstream is a little pet peeve of mine. I agree with your main point though.

dtf• 17 years ago

pop(i) or del a[i] is remove the item at i; insert(i, x) inserts x before the item at i; append(x) adds x after the last item (which can't be done with insert). remove(x) is unordered as it operates on the item not the index, so the opposite of remove should be an unordered add (which exists in an unordered set but not in a list). The rest of his points are more valid.

shiranaihito2• 17 years ago

Coming from a Java-background, I've often found myself trying to call list.add(). I think that would be an even more natural complement to list.remove().

rbanffy• 17 years ago

I think the del statement as in "get rid of this" is perfectly fine. There is no need to pollute the namespace of lists (or anything, BTW) with a method to duplicate an already existing, and very fundamental, language feature.

grandalf4• 17 years ago

I've been exploring Python over the past few weeks after a few years of intense Ruby immersion.

There are definitely some very nice things about Python. I highly recommend that all rubyists do a nontrivial project in it asap.

tptacek• 17 years ago

What nice things would those be? I'm struggling for an obvious example of something I miss from my Python days (which ended in '06).

grandalf4• 17 years ago

I've found that code is way more readable, even though I've only been using the language for a short time.

I wonder what a whitespace sensitive ruby would be like :)

  %w(a b c d e f g).each do |i|:
       puts i

  foo = lambda do |x|:
       x + 2

  foo 3

erlanger• 17 years ago

"A normal person will eat everything in front of them, but a person with neglect will happily eat only the things on the left side of their body (emphasis added)."

It appears that this condition is more serious than we originally thought.

andres• 17 years ago

the symmetric add/delete operations are actually:

mystuff[4] = 'apple'

del mystuff[4]

i'm not saying it's very elegant, but it makes sense.

calambrac• 17 years ago

mystuff[4] = 'apple' doesn't add an 'apple', it overwrites an existing element in a list of at least 5 elements, and del mystuff[4] actually does change the length of the list. They aren't symmetric at all.

shadytrees• 17 years ago

Yet, here’s what you have to do for Sphinx which is an insane amount of work for something that JavaDoc, POD, Doxygen, RubyDoc

Nit: Sphinx is a system for long-form, separately written documentation. Like manuals and tutorials; think stuff that requires indices. That's why autodoc is an extension, why you usually set up a separate directory, why you have the flexible build system, and so on. (Although all you have to do is create a file and type `automodule` and then the module name.)

triplefox• 17 years ago

The one that always bugged me but seems to be true for most dynamic languages is immutable types being passed by value and everything else passed by reference.

brendano• 17 years ago

For an immutable type, there's no distinction between pass-by-value and pass-by-reference. How would you ever know the difference?

tow212• 17 years ago

Speed? Not sure if this is actually what the OP was thinking of, but pass-by-value usually implies copying, while pass by reference just involves handing around a pointer.

ie, if you had an enormous immutable value (in Python, a tuple with lots of entries), then in a hypothetical Python implementation which did call-by-value, it might be significantly slower than call-by-reference, on account of having to copy the entire tuple.

But this falls squarely into 'implementation detail' - if you're so inclined, you could implement either CBV or CBR in hundreds of other ways. Certainly from the perspective of time-independent program behaviour you shouldn't be able to tell the difference.

callahad• 17 years ago

  > Then when you are told about it, you’d make up excuses trying to explain
  > why it is totally normal.

That's a pretty incredible rhetorical device. "If you disagree, you're fundamentally incapable of reasoned thought."

gchpaco• 17 years ago

Agreed. Sadly, Shaw (ab)uses it fairly often.

ellyagg• 17 years ago

Except, do you doubt its truth? Humans are rationalization engines. Communities will go to great lengths to prove that everything about their culture is good.

jjames3• 17 years ago

This is exactly why people who learn multiple programming languages/paradigms are so valuable to these types of conversations. The problem is that reasoned and experienced thought gets thrown out with the speculative garbage.

Confusion• 17 years ago

I not only doubt its truth: I know for sure it's false.

If I disagree with someone, then I often still grant that their opinion is the result of reasoned thought. In such a case it's the underlying assumptions that we disagree on and those assumptions are usually not amenable to reasoning.

A programming example is bracing styles: I have my preferred one and a colleague has his preferred one. I have my arguments and he has his arguments. In the end, it comes down to what each of us considers to be 'best readable', which is an entirely irrational consideration. We get along very well, despite our differences (and of course, for each project, we settle upon a style, based on exterior considerations like: what would be consistent with this clients' codebase).

Another example is religion: I'm a staunch atheist and my girlfriend is a Christian. 'nuff said.

nailer• 17 years ago

  >> Then when you are told about it, you’d make up excuses trying to explain
  >> why it is totally normal.
  > That's a pretty incredible rhetorical device. "If you
  > disagree, you're fundamentally incapable of reasoned 
  > thought."

No, that's not what Zed's saying at all. He's saying that people often respond to valid criticism of things they know well by justifying the behavior being criticised.

Ie, people mold their way of thinking to suit their existing tools. Since they're familiar with how things are, they have problems seeing what could be.

It's not an ad-hominem attack at all. Think of the academic who has no idea of how to teach because he knows his subject so well he can no longer see it from the perspective of an outsider.

ghshephard• 17 years ago

Python is relatively consistent with regards to appending/popping/inserting/removing:

a=[] dir(a) 'append', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort'

help(a.remove) remove(...) L.remove(value) -- remove first occurrence of value

help(a.pop) pop(...) L.pop([index]) -> item -- remove and return item at index (default last)

append(...) L.append(object) -- append object to end

insert(...) L.insert(index, object) -- insert object before index

rw• 17 years ago

From the OP: "For those of you who refuse to read the article, "

Maybe some of us already know what hemispatial neglect is, Zed.

petercooper2• 17 years ago

  >>> exit
  Use exit() or Ctrl-D (i.e. EOF) to exit

Enough said. If you bloody well knew what I typed, just exit FFS.

Python's design is full of shortsighted, dogmatic decisions like this - don't even get me started with functions versus methods and __len__-esque line noise, argggg! And there should only be one obvious way to do something? Give us a break - that's just not how life, mathematics, or anything should work.

j2d2• 17 years ago

What makes you think programming languages should mimic life?

tvon• 17 years ago

Seriously, that sounds like an absolutely horrible idea.

petercooper2• 17 years ago

Life was just the first item on my list. I also said mathematics. There's plenty of precedent for programming languages to "mimic" mathematics due to being a branch of mathematics, after all. Having only one obvious way to do something is religious dogma unmatched by any similarly practical activity I can think of right now.

martey• 17 years ago

The only reason I could think of this warning (which I have run across entirely too many times) was in case someone had a variable called 'exit'. Yet:

  >>> exit = True
  >>> exit
  True
  >>> exit()
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: 'bool' object is not callable

The only way to then exit is Ctrl-D, but the shell will not prompt you to enter it!

ii4• 17 years ago

> The only way to then exit is Ctrl-D

Nope.

  >>> __builtins__.exit()

ciupicri• 17 years ago

Or Ctrl-Z (EOF) on Windows.

jjames3• 17 years ago

Explicit function calling via () is one of my favorite things in the language. To each their own.

"exit" merely references a function. It doesn't call by name alone. It can thus be passed to a function (a callback for example), saved to a variable, etc for later calling. Maybe not so useful for the exit function in the interactive shell but very useful for functional programming. This is an example of consistency, sometimes at the expense of convenience; not great for hacking, awesome for collaboration and large projects (imo).

>>> dir(exit)

['__call__', '__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'name']

(pardon the dunderware)

Here's another way to exit the interactive shell ;)

>>> exit.__call__()

derefr2• 17 years ago

But how does "exit" by itself print that pretty little message telling you that, although it's obvious you wanted to exit, it's not going to let you? It's not just a function reference if it shows up that way.

shiranaihito2• 17 years ago

"exit" merely references a function. It doesn't call by name alone.

His point was that the user already typed in "exit", and the interpreter recognized the user's intent to exit, so it should just shut up and exit already, instead of telling the user to exit in a different way.

ii4• 17 years ago

You can monkey-patch your interactive interpreter BTW:

  >>> type(exit).__repr__ = lambda self: exit()

It will exit then the way you want :)

Nobody do it for the sake of overall consistency. Yet you are free to change it with the one line of code.

thristian• 17 years ago

Some responses to various parts of the article:

"No easy_UNinstall" - I'm one of the lucky few who has access to an extensive and sensible archive of easily installable and uninstallable Python library packages; I call it "Ubuntu". From what I can see, distutils is a good metadata format that helps real packagers like Debian and Ubuntu create real packages - but trying to write a cross-platform installer is just asking for trouble. Just unpack the package somewhere and set $PYTHONPATH in your startup scripts; that's what your startup scripts are for, anyway.

"rm -rf" - my understanding is that the contents of the "os" stdlib package are a thin wrapper around POSIX, and rmdir(2) won't remove a non-empty directory either (although Zed says Python's rmdir will remove a directory with subdirectories, but not files... that's odd). "shutil.rmtree" isn't in POSIX, so it isn't in "os" either.

"Time Converstion" - Zed says "If all they did was give me the exact same POSIX C API I’d be happy.", but so far as I can tell, the 'time' stdlib module basically is the POSIX C API, with braindead awkwardness fully intact. I'll grant that "calendar" is stupid and "datetime" is crippled, although the third-party "dateutil" module fills a lot of the holes in "datetime". (also, for as long as I've known of it, mx.DateTime has been freely available)

"API Documentation Generation" - It's not in the standard library, but I've been quite happy with the third-party tool "epydoc" for Python API doc generation, and it pretty much is as easy as "epydoc path/to/package" (and predates Sphinx and a lot of the other tools).

I have to agree with the rest of his examples, though - Python's had a long, rocky road from "procedural Unix scripting language" to "Object-oriented, Internet service-providing language", and although Python 3.0 has cleaned up a lot of the cruft there's still some oddness that remains (like the len function, or the del keyword). A lot of the standard-library crud has come from people saying 'here, I've found/written an 80% solution to this particular problem, let's put it in the standard library since it's better than the solution that's in there at the moment' rather than 'here's a problem, let's design a 90% or 95% solution for the standard library'.

stcredzero2• 17 years ago

A lot of the standard-library crud has come from people saying 'here, I've found/written an 80% solution to this particular problem, let's put it in the standard library since it's better than the solution that's in there at the moment' rather than 'here's a problem, let's design a 90% or 95% solution for the standard library'.

This is always a cultural/community issue. It would need a cultural/community fix. And, given what I've seen in other communities, it should be fixed, as there's a lot to be gained by doing so.

fauigerzigerk• 17 years ago

Yes the python APIs are very inconsistent, but I disagree about the del statement. That's a different matter and the way Zed looks at it is very much the kind of hemispatial neglect that tends to befall people who are immersed too deeply in the OO paradigm.

del x unbinds x from whatever namespace it resolves to. How would you say c.remove(x) if you don't know which collection c represents? Now, of course you could argue that del x is too different from removing something from a collection to have it use the same syntax. But that's not about neglect. It's just a different opinion about what is consistent.

The reasoning is probably that removing a variable should always be done in one way and that is del x no matter if x is inside a collection we know or not. It's a very conscious attempt to make it consistent and that's not neglect. Whether it's a good idea I don't know.

Maybe a global del function would be better. If x is inside c and we know what c is we would say c.del(x) and if we want c to be resolved we say del(x). But is that really so different from what we have now?

More generally, my opinion is that the way functions are used in OO is in itself very inconsistent. If you have a function f and variables x, y, and z, it's mostly a consideration of implementation details that leads to a decision of whether it should be x.f(y, z), y.f(x, z) or z.f(x, y). Why does the user of an API have to think about or remember which one it is?

sandGorgon• 17 years ago

Ruby's Gems ownzors easy_install - with versioned libraries, updates, etc..

And apt-get, pacman, rpm are not excuses, because I like to use Slitaz as my OS

If virtualenv (which I love) is something that plays havoc with package management - they should be merged, rather than not do anything about it.

llimllib3• 17 years ago

what I love is when I want to install a single ruby gem and I get to sit around waiting for a half hour while it fetches seventeen unrelated things from github. For extra special sauce, let github be down.

Get pip and be enlightened.

carbon8• 17 years ago

The situation you describe simply doesn't exist.

holygoat• 17 years ago

I've ranted on this subject to my coworkers many times in the past. Python's API is full of missing inverses, missing analogues, and similar things with oddly different shapes.

Curing Python's Neglect

Related Stories