Going beyond the idiomatic Python

https://hackernoon.com/going-beyond-the-idiomatic-python-a321b6c6a5e6

by okaleniuk • 9 years ago

60 86 9 years ago

falcor84• 9 years ago

I'd argue that actually all of the suggestions in this post are more idiomatically pythonic than the original ones. So the trick is not to avoid idioms but just to write clearer code :)

agumonkey• 9 years ago

it's also a quality that takes times, you have to weight a lot of things together. shorter names sometimes, longer names sometimes... Something you appreciate with times, no matter the language.

pwaivers• 9 years ago

I like the rules that lays out in this article. However, I don't see his argument against Writing Idiomatic Python. He cherry picked two methods that (correctly) could have been written better. However, this does not mean that all idioms are bad, and he doesn't provide any examples where using an idiom is sometimes bad and sometimes good.

Additionally, it seems like the word idiom is just re-skinned as a "rule" in this article.

Finnucane• 9 years ago

Yes, I think the problem here is not that the idea of idiomatic Python is not good, but that this book about it is not good.

dcdanko• 9 years ago

He provides three examples where an idiom is sometimes bad and sometimes good...

The point of the article is just the classic 'foolish consistency is the hobgoblin...'

llimllib• 9 years ago

    def inverse_of(matrix):
        return adjugate_of(matrix) / determinant_of(matrix)

aiiieeee no, don't do that

aaronblohowiak• 9 years ago

care to elaborate?

jacob019• 9 years ago

Why? Don't like self-documenting code?

scienceman• 9 years ago

In the context of matrices, its pretty clear what inv, adj, and det mean.

knappa• 9 years ago

It's numerically unstable and O(n^3) (at least, probably worse) when computing via row reduction is O(n^2).

BeetleB• 9 years ago

No doubt, the article was written by a physicist.

graton• 9 years ago

On a tangent. I'm still kind of pissed off at Jeff Knupp. I bought the e-book which claims it is updated frequently and you get free updates for life. I have emailed him over three times asking about updates and so far have received zero replies :(

crdoconnor5• 9 years ago

If you're aiming for idiomatic on 6 you don't do:

    def contains_duplicate(array):
        return sum([sum([1 if a_i-rot_ij else 0 
                    for a_i, rot_ij in zip(array, rot_i)]) 
                    for rot_i in [array[i:] + array[:i] 
                    for i in range(1, len(array))]]
                   ) != len(array) * (len(array) - 1)

OR:

    def contains_duplicate(array):
        for i in range(len(array)):
            for j in range(i):
                if array[i] == array[j]:
                    return True
        return False

You do:

    def contains_duplicate(array)
        return len(set(array)) < len(array)

Or, if you can't hash the items:

    def contains_duplicate(array):
        any(len([y for y in items if x == y]) > 1 for x in items)

FWIW, I don't think I've ever seen an example where the cleanest (list | generator) comprehension is less clear than the cleanest equivalent for loop. I don't believe one exists.

So I think the examples are contrived and the 'rule' is bullshit.

LyndsySimon2• 9 years ago

FWIW, I don't think I've ever seen an example where the cleanest (list | generator) comprehension is less clear than the cleanest equivalent for loop. I don't believe one exists.

Interesting. I'm not sure if I agree or disagree.

I think the primary benefit of a comprehension is that it can be considered as a "unit", and you don't have to think about the iteration as fully to understand what's happening. As soon as "what's happening" gets more than minimally complex, that advantage is lost. Generator comprehensions should be much more clear and concise than their equivalent implementations using generator functions, though.

If I remember this evening, I'm going to go through my code library and see if I can find something that will satisfy your challenge :)

crdoconnor5• 9 years ago

I consider it an instance of https://en.wikipedia.org/wiki/Rule_of_least_power

List comprehensions are less expressive than for loops hence there is less to reason about.

Hence, if what you want is achievable with a list comprehension, its intent will naturally be clearer than the for loop equivalent.

...provided you don't make it intentionally obtuse like the OP did.

pvg2• 9 years ago

That's not equivalent code. Your version only works with a list of hashable objects. Try it with a list of lists.

pvg2• 9 years ago

You deleted your other comment but you've now written the comprehension version twice (the second version added in an edit when it was pointed out the first one isn't equivalent to the loop) and neither of them actually work. So perhaps writing the comprehension version is neither as clear or simple as you suggest.

crdoconnor5• 9 years ago

It was sloppy copying and pasting of a working expression. I think it's pretty clear I meant to write this:

    def contains_duplicate(array):
        return any(len([y for y in array if x == y]) > 1 for x in array)

Or were you saying that this ^^ doesn't work?

wcarss• 9 years ago

In my opinion, the non-hashable version which you moved here from a now-deleted response below is a perfect example of code that is harder to read, harder to hold in one's head, and most importantly harder to modify.

The most pythonic version (your set solution) has a bug in that it does not apply to the general case. The second-most pythonic version (your one liner double for-loop slam dunk) is as clever as we can make it, and thereby fails Kernighan's debugging test[1].

I feel that one for loop inside another is fine. Everyone in every language can see it and know instantly what the run-time characteristics likely are, and what at each point it is doing. What's most important: if it ever seems to be broken, I can drop a debugger or a logger (maybe with a condition!) in at any point, with ease. There is a reason that pattern is nearly universally known, and imo it's that it is a good one.

1 - "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?", https://en.m.wikiquote.org/wiki/Brian_Kernighan

crdoconnor5• 9 years ago

is a perfect example of code that is harder to read, harder to hold in one's head, and most importantly harder to modify.

It's both shorter and less expressive than the equivalent for loop. If you find it harder to read that says more about your familiarity with list and generator comprehensions.

The most pythonic version (your set solution) has a bug

Not in lieu of a hard requirement for counting non-hashable items, it isn't.

It's faster as well as simpler: O(n) instead of O(n2). Unless you're counting lists of lists or something it makes no sense not to use it.

I feel that one for loop inside another is fine. Everyone in every language can see it and know instantly what the run-time characteristics likely are, and what at each point it is doing. What's most important: if it ever seems to be broken, I can drop a debugger or a logger (maybe with a condition!) in at any point, with ease

Writing your code in a more verbose and complex way in order to satisfy your desire to put a debugger in is by far and away not the most important aspect of clean coding.

divs12102• 9 years ago

Let me unleash a monster that I created:

https://stackoverflow.com/a/44729275

masklinn2• 9 years ago

It seems unnecessary:

    LET(('a', 1,
           'b', 2),
          lambda o: [o.a, o.b])

alternative:

    lambda a=1, b=2: [a, b]

Thing do get worse when you need dependencies:

    LET(('a', 1,
           'b', lambda o: o.a + 1),
          lambda o: o.b)

alternative:

    (lambda a=1: lambda b=a+1: b)()

divs12102• 9 years ago

I would argue my version scales more gracefully.

How would you write this, for example?

    LET(('a', 2,
         'b', lambda o: o.a + 1),
         'c', lambda o: [i * i for i in range(o.b)]),
        lambda o: o.c)

    => [0 1 4]

LyndsySimon2• 9 years ago

I don't know what I think about this...

fhood4• 9 years ago

This is a little bit off topic, but one thing that I wish python used is ruby's indicators for the purpose of a function. i.e. "?" for boolean returns, and "!" for modifying functions in place.

In a language without explicit typing and return values having an instant indicator of the functionality of a function can be a huge help in writing readable code.

epage• 9 years ago

As someone who dislikes Ruby, I have holy envy[1] for Ruby's use of ? and !

[1] http://ancienthebrewpoetry.typepad.com/ancient_hebrew_poetry...

fhood4• 9 years ago

As someone who thinks they understand both the upsides and downsides of Ruby, you shouldn't write it off. I find ruby to be really unsuited for projects with multiple people, but genuinely fantastic for personal projects or one off scripts.

cup-of-tea3• 9 years ago

Is this enforced in Ruby or just a convention? It's a great convention and is used in Clojure and older Lisps at least with the -p suffix (for predicate). There's no reason you can't use the same convention in Python. I've often wanted to write predicates with the -p suffix but stopped myself becasue I don't think other Python people would understand it. Maybe I should just do it anyway.

masklinn2• 9 years ago

Is this enforced in Ruby or just a convention?

It's just conventional.

ralmidani3• 9 years ago

In Ruby, the "?" is optional and used to indicate you are calling a function/method which returns a boolean.

The "!" is actually part of the function/method's name. When you want to sort an array, for example, you can either say "sorted_array = array.sort" or you can sort in-place with "array.sort!". So the exclamation distinguishes between the two methods and warns you are performing a destructive operation.

brudgers11• 9 years ago

{being pedantic} In Python, you can't use the '-p' suffix because Python will treat it as subtraction.

{being unPythonic} If it makes you happy to use '_p', do it. Explain it in the documentation if it might cause confusion. Be flexible enough that you'll change before letting yourself be voted off the island because it's not that important.

{rabble rousing} I think there is room in the world for an alt-python movement...https://github.com/altpython

bakery2k• 9 years ago

"!" for modifying functions in place

I think that consistently using "!" to mean "mutates in-place" would be a good convention for a language to use, but that's not what Ruby does. From Matz himself [1]:

The bang (!) does not mean "destructive" nor lack of it mean non destructive either. The bang sign means "the bang version is more dangerous than its non bang counterpart; handle with care". Since Ruby has a lot of "destructive" methods, if bang signs follow your opinion, every Ruby program would be full of bangs, thus ugly.

[1] https://www.ruby-forum.com/topic/176830#773946

danso• 9 years ago

I teach Python to beginners, some of whom have had a CS class (e.g. in Java). I try to avoid idioms early on, such as list comprehensions and context managers, and slowly introduce them after students are confident with loops and such. It does feel painful and backwards, though, which is ironic for me to feel when I consider how confused/annoyed I was with Python initally when coming from Ruby.

brudgers11• 9 years ago

The worst idiom of Python is the idea of Pythonic. It is usually prefixed with "un" and used pejoratively. The rest of the cases it is used to rationalize arbitrary biases such as Python's lack of an identity function and the removal of map and filter [1] from the core language.

There's no such thing as "very Pythonic." It's a binary and the worst aspect of Pythonic is that people waste so much energy trying not to be unPythonic in order to meet the mores of the community.

[1]: Edit, Ok, I was confusing two experiences. It was reduce's removal that broke my code using the map->filter->reduce idiom. Map broke by returning a Map object rather than a list (but that was between Python and Python3). And so far as I know, filter has not been changed so as to break code.

dec0dedab0de3• 9 years ago

I understand why you would feel this way but I disagree. The meme of being pythonic has the benefit of most libraries being understandable. It is always very obvious when python code has been written by someone who specializes in another language, and it is usually painful to read their code.

emidln• 9 years ago

So if I have this right, things like Django's Class Based Views and sqlalchemy are Understandable. I'll take functional-style python over Understandable python any day of the week.

brudgers11• 9 years ago

Like I said, the usual use of the 'Pythonic' idiom within the Python community is prefixed by "un" and used pejoratively. One aspect of the perniciousness of 'Pythonic' is that it promotes the tribalist ideal of specializing in languages. For someone optimizing for readable libraries, Smalltalk is probably a better option.

empthought9• 9 years ago

the worst aspect of Pythonic is that people waste so much energy trying not to be unPythonic in order to meet the mores of the community

This is why Python is so successful, and yet you've never heard anything about "Python fatigue." Being "Pythonic" (which as a proxy I will define as not needlessly flouting the principles that `python -mthis` prints out) is valued by most open source projects and most heavy Python users.

With respect to your complaints, they are not "arbitrary biases," they are all the same conscious decision on the part of Guido to favor the use of comprehension syntax over higher-order functions. See https://www.artima.com/weblogs/viewpost.jsp?thread=98196

You might disagree with that decision and prefer another language, but it is not arbitrary at all.

mercer5• 9 years ago

I do minimal python programming but your comment made me re-read The Zen of Python. Such a good reminder of what I'd definitely call good practices.

brudgers11• 9 years ago

I attribute most of Python's growth to being a better general purpose lanaguage than MatLab and BASIC and Java and C++ and FORTRAN in common academic settings and more readable than Perl and more robust than TCL for moderate sized systems in commercial settings. Given JavaScript, there is evidence that the arbitrariness of a programming language's design is at best weakly correlated to its adoption.

kevinmgranger2• 9 years ago

the removal of map and filter from the core language

This never happened-- perhaps you're thinking of reduce? And reduce is still in it, it just got moved to `functools`.

empthought9• 9 years ago

Hah, I didn't even know that it never happened. Goes to show the relative utility of map/filter vs comprehensions...

amyjess• 9 years ago

It was proposed to do that with all three, and they walked it back to just reduce because there was a huge outcry.

brudgers11• 9 years ago

Thanks, I stand corrected. My high level memory is that the map->filter->reduce idiom was broken because <arbitrary>. Adding to my confusion was my conflating that <arbitary> with my memory of map breaking code by returning a map object rather than a list.

crdoconnor5• 9 years ago

Most of it ok, but the notion of truthiness and falsiness being pythonic (or even allowed!) was python's billion dollar mistake.

It's a magnet for bugs (e.g. stock items with a price of zero being treated as items with no price at all).

Walkman3• 9 years ago

Please don't ever write Python programs with this mentality.

brudgers11• 9 years ago

Like I said, Pythonic defaults to pejorative.

dec0dedab0de3• 9 years ago

Rule 6. Just because you can do anything with list comprehensions, doesn’t mean you should

I have been very guilty of this one, it's a running joke at work gatherings. In the flow I can quickly spit out comprehensions that are obtuse to everyone else. My general rule of thumb is that if I ever have to troubleshoot it I break it out into a for loop

freyr2• 9 years ago

Obviously, any falsey value triggers the "idiomatic" contains_zero function:

    [1, False, 3]
    [1, '', 3]
    [1, None, 3]
    [1, [], 3]

This seems like it could only make bugs harder to track down, yet this is often touted as the Pythonic way. What's the benefit of this approach?

empthought9• 9 years ago

The original implementation of `contains_zero` in the blog post is not idiomatic. (It's also incorrect, as you point out.)

The idiomatic implementation of `contains_zero` is the second one given, `0 in container`.

Just because someone claims something is idiomatic in a book doesn't mean they are correct about it being idiomatic.

freyr2• 9 years ago

You're right. PEP8 says:

beware of writing if x when you really mean if x is not None -- e.g. when testing whether a variable or argument that defaults to None was set to some other value. The other value might have a type (such as a container) that could be false in a boolean context!

This should apply to the contains_zero function as well. It seems a little murky when PEP8 recommends checking whether a sequence type is empty,

    Yes: if not seq:
         if seq:

    No: if len(seq):
        if not len(seq):

if seq were accidentally set to False or 0. But I guess that would be revealed as soon as you start treating seq like a sequence.

k_sze• 9 years ago

Some of the examples in the book violate the Zen of Python. So I wouldn't even call them pythonic or idiomatic in the first place.

E.g.:

`contains_zero` violates "There should be one-- and preferably only one --obvious way to do it."

`should_raise_shields` violate "Explicit is better than implicit."

`linsolve` and `contains_duplicate` violate "Readability counts."

shangxiao• 9 years ago

I was thinking exactly this as I was going through those examples.

extr• 9 years ago

To use a math analogy, I think of idioms as solutions to generalized coding problems. If you recognize your situation as a special case of the general form you can save yourself some time thinking through the implementation and make it easier to understand. Great. But also like math, you can get yourself into trouble trying to frame something in terms of a general case that doesn't actually fit, suddenly the plumbing is more complex than the original problem. The same thing plays itself out at a higher level with design patterns. The (obvious) point is not to force a solution into an abstract pattern at the expense of clarity, because that's the reason we like abstract patterns in the first place.

ralmidani3• 9 years ago

While I adore Python generally and use it all the time (including for two startups), here are a few things that I find unintuitive:

1. List comprehensions are hard to read and understand, especially for beginners. I tutor online in Python, and when I show a student a list comprehension, it is one of the most difficult parts of the language to explain. Do you read left to right, or inside to outside? Postfix if at the end makes things even more confusing.

There are better alternatives in languages like Ruby and JavaScript. Which brings us to the next point:

2. No native map, filter, reduce, etc. methods. It is far easier to explain that you are just calling another method on a list/array than it is to get a beginner to trust the hocus-pocus apparent in Python's list comprehensions.

Edit: Python has built-in functions for map and filter. See correction by ubercore below, and my response.

3. The way a class is accessed from an instance in Python is not nearly as elegant as Ruby's. Do I use "type(obj)" or make a direct call with "obj.__class__"? Neither is as clean or intuitive as "obj.class".

4. Speaking of classes, why are built-in classes like "str", "int", "float", etc. lowercase while in everything else they are conventionally CapitalizedClasses?

5. Remembering that "sorted(my-list)" returns a new sorted list while "my-list.sort()" does the sort in-place requires experience, which someone taking an Intro to Programming course will probably not accumulate over 8 or even 14 weeks.

Note: I flirted with Ruby and Rails, but never built anything real with them. I vastly prefer Python to Ruby (and Django to Rails), but think Ruby got a lot of things right.

ubercore• 9 years ago

No native map or filter? Unless I don't understand what you mean by native, don't https://docs.python.org/3/library/functions.html#map and https://docs.python.org/3/library/functions.html#filter and https://docs.python.org/3/library/functools.html#functools.r... count?

ralmidani3• 9 years ago

Thank you for correcting me. I edited my post above to reflect that.

But having to provide a lambda for the simple case makes them less useful. Also, nested function calls are less readable than chaining method calls on the same object.

mercer5• 9 years ago

Without going into the specifics, I'd argue that, in general, teaching programming using a 'multi-paradigm' language (Python, Javascript, Ruby) is inherently more confusing than picking one that focuses more exclusively on one paradigm. I tried to do as much functional programming in Javascript as I could, and yet I feel I only really learned it when I started working with languages that are 'heavily' functional.

Of course, I'm not saying that a 'multi-paradigm' language for teaching is bad, but personally I'm inclined to think that it might be better to start with something more pure.

Ruby might actually not be a bad choice in this scenario. If you avoid the 'blocks, lamdbas and procs', it seems to me like a great introduction to OO programming. Despite this, for my day to day programming I would also choose Python over Ruby. But perhaps not for teaching.

empthought9• 9 years ago

1. Try teaching your students the standard notation for set building -- that's where it comes from, and it's useful outside of Python. https://en.wikipedia.org/wiki/Set-builder_notation

2. They're all still there, but they're not as good as comprehension syntax.

3. Use type(self) unless you know why you'd want to use __class__ (Python 2 old-style classes).

4. That's legacy; probably if things were done from scratch today, they'd be Int, Str, Float. Back when the names were chosen there was a stricter line between builtins and unset-defined classes, and there was more consideration given to people migrating from C and C++.

5. Yes, luckily the .sort() method gives you at least some hint by not returning anything. It's a good application of command-query separation, but I forget it often myself.

tzs• 9 years ago

People don’t speak entirely in idioms unless they are totally off their rockers.

My first thought on reading that was to think of Darmok and Jalad at Tanagra. Then I realized that was allegory, not idiom. Then I realized I had no idea what the actual definition of "idiom" is, and looked it up: "a group of words established by usage as having a meaning not deducible from those of the individual words (e.g., rain cats and dogs, see the light)."

...and now I'm confused because nearly everything we call an idiom in programming isn't. What would be the correct term for whatever it is we are currently calling programming idioms?

For example, one from the idiomatic Python book that the article criticizes:

  def contains_zero(iterable):
    # 0 is "Falsy," so this works
    return not all(iterable)

That one was confusing to me, because I'm not a heavy Python user. I've never encountered all() before. However, after looking up the all() function in the Python documentation that code becomes immediately obvious.

It seems that most of what we call idioms in programming are just using more advanced or more obscure language features or library functions to do something in a more compact or efficient way at the cost of immediate clarity to people who are not as advanced in the language. We've decided what these do is needed often enough that we agree that these are the way people should do these things, even if they don't understand what is actually going on.

I'm not sure programming languages can have actual idioms, except perhaps in the case of language bugs or bad/missing documentation.

The combination of language plus hardware could have idioms, like nested loops iterating a multidimensional array should have the outer loop on the first dimension, the second loop on the second dimension, and so on to minimize jumping around in memory and so hopefully interact better with the cache.

mercer5• 9 years ago

That one was confusing to me, because I'm not a heavy Python user. I've never encountered all() before. However, after looking up the all() function in the Python documentation that code becomes immediately obvious.
"a group of words established by usage as having a meaning not deducible from those of the individual words (e.g., rain cats and dogs, see the light)."

Perhaps this is actually a really good illustration of 'idiom' meaning the same thing (roughly) in programming as it does in day-to-day use.

I'm not a Python dev, but I'm willing to bet that a Python dev knows the all() function, in the same way that a Javascript dev knows about something like .bind(this), or querySelectorAll, or a CSS 'dev', the horror, "margin: 0 auto;".

If that's not the case I might be wrong.

empthought9• 9 years ago

It's even worse; the book is wrong. That implementation of contains_zero is not idiomatic to anyone who writes Python.

hxtk• 9 years ago

Not only is it not idiomatic; it's functionally incorrect.

An iterable containing `[]`, `None`, or any other value that is false in a boolean context would return true. They've assumed that `(0 -> false) -> (false -> 0)`, which is simply not true.

Even if you're dealing with a context that cannot pass in an iterable with non-zero "falsy" values then it needs to be explicitly documented. At that point, why not "document" it by being explicit with the code. This is how I'd do it, but I feel like it could be made cleaner:

    def contains_zero(iterable):
        return any([i is 0 for i in iterable])

Edit: I forgot how to python. It's `return 0 in iterable`.

Veedrac• 9 years ago

The Zing of Python

(Or the well intentioned but ultimately misguided attempt to enlighten the masses.)

Pretty is better than boring.

Comprehensions are better than loops.

Short is better than long.

Clever is better than tangled.

Nested is better with line breaks.

Compact is better than meandering.

Appearances matter.

The point of a Python program is to teach people to write better Python programs.

Though if they're Java programmers just try to make them jealous.

Exceptions should never be explicitly checked.

Except StopIteration.

If you don't know how to make it Pythonic, refuse to take the obvious approach.

There should be one-- and preferably only one --Pythonic way to do it.

Although that way may not be obvious at first unless you're Dutch.

Now is better than never.

But it's fine to take ramp it up over time as long as you get there.

If it doesn't make a good blog post, perhaps it doesn't make good code.

If it doesn't make good code, it definitely doesn't make a good blog post.

List comprehensions are one honking great idea -- let's do more of those!

zwieback• 9 years ago

Agree with the critique but it's not quite fair. The original book is about Python, maybe sometimes misguided. The critique could be leveled at any language, it basically says: "be an experienced programmer with good taste."

Walkman3• 9 years ago

    def are_all_numbers(everything):
        return all(is_float(e) for e in everything)

Walkman3• 9 years ago

    To learn to write good code you have to write a shit-metric-ton of bad code.

The problem with this is how do you know if your code is good or bad? A lot of programmers never learn. They only way is to learn from others and write. You won't necessarily will be a better programmer if you just write programs by yourself.

Going beyond the idiomatic Python

Related Stories