Code blocks in Python

http://mtomassoli.wordpress.com/2012/04/20/code-blocks-in-python/

by mtomassoli • 14 years ago

49 68 14 years ago

mtomassoli.wordpress.com

kibwen• 14 years ago

I'd like to see some before-and-after examples of Python code that's been rewritten to take advantage of blocks using this module.

bcl• 14 years ago

Seems like a good way to make your code harder to read and harder to debug.

stcredzero9• 14 years ago

Anonymous functions can generally be abused in ways that make code harder to read. Then again, so can recursion. It's a double edged sword.

Goladus4• 14 years ago

Interesting, but I'm having a hard time seeing how this syntax saves me much in terms of code size and complexity compared to just wrapping the code blocks into named functions and passing those.

Peaker5• 14 years ago

The problem with that is that it forces moving the code above where it is passed, which can greatly clutter things.

For example, consider the function "forever" in Haskell:

  forever action = do
    action
    forever action

Now you can use it as a new control structure, e.g:

  forever $ do
    (sock, addr) <- accept listener
    forkIO $ handleClient sock

The $ simply means "apply" and is low-precedence, so it removes the need to put () around the entire argument to be applied.

In Python, you could define:

  def forever(action):
    action()
    forever(action)

but then, to use it, you have to give a name to your function, so:

  def accept_once():
    (sock, addr) = listener.accept()
    fork(partial(handleClient, sock))
  forever(accept_once)

This makes "forever" much less useful as a new control structure/looping primitive.

In this sense, Python makes DSLs less usable. The built-in primitives are first-class, and can have code directly within their use. Library functions are second-class, and can only have code passed by name which must then fully appear before the use.

Another example is callbacks. The reason "twisted" is probably called "twisted", and that people hate callbacks so much, is that it forces writing the code in backwards order, precisely because of this problem. For example:

  def handle_result(result):
    print "Done:", result
  def connection_started(conn):
    conn.request(SomeRequest(), handle_result)
  def start():
    start_connecting(connection_started)

Compare this with Haskell, as an example:

  start = startConnecting $ \conn -> do
    request conn SomeRequest $ \result -> do
      putStrLn $ "Done: " ++ show result

Note, in Haskell, this would actually be worked out to be (by overloading the semicolon):

  start = do
    conn <- startConnecting
    result <- request conn SomeRequest
    putStrLn $ "Done: " ++ show result

But even the former nested representation is better than the backwards ("twisted") representation that makes people hate callbacks so much.

viraptor2• 14 years ago

This seems like a solution in search of a problem. There is a nice 'forever' construct. It's called 'while':

    while True:
      action()

I know what you mean about generalising this to other control structures, but it looks like the whole concept starts from "I want to write 'forever' that resembles something from other languages", rather than "I want to write a loop". There are existing ways to write things like that, so is do we really need to force something from other languages into Python? What about some examples which cannot be easily handled - maybe the solution for them is something completely different than porting codeblocks.

hythloday2• 14 years ago

But you don't have to write your callbacks in reverse order in python:

  >>> def fight(batman_wins):
  ...     def t(): return "pow"
  ...     def f(): return "oof"
  ...     def b(msg): print msg
  ...     # if_but not defined at this point
  ...     if_but(lambda: batman_wins, t, f, b)
  ...
  >>> def if_but(cond, true, false, but):
  ...     if cond():
  ...       v = true()
  ...     else:
  ...       v = false()
  ...     but(v)
  ...
  >>> fight(True)
  pow
  >>> fight(False)
  oof

Sure, if you want to call if_but outside a function, it has to be defined, but inside a function (as in your example, and in 99% of python development that doesn't occur in a REPL) x() is effectively globals()['x'](). Have I totally misunderstood your point?

Goladus4• 14 years ago

I understand anonymous code blocks is something that python doesn't do natively and other languages can do. What I don't understand is why it's worth the bother, given the hoops one must jump through to get it to work in Python specifically (new operators, keywords, and evaluating string literals as code).

If you really need new control structures, Python syntax isn't all that hard to hack.

andrewaylett• 14 years ago

Surely the act of enhancing a function like that, where you don't want the original code to be available, is more commonly dealt with in Python using decorators?

francoisdevlin2• 14 years ago

This is a terrible idea, and ruby used the wrong implementation. If you want this behaviour, you should just define a higher order function/decorator, and put your "block" in its own function. This is a case where there should be only one way to do it.

jamesgeck0• 14 years ago

I agree that this doesn't fit Python well, but why was it bad in Ruby?

francoisdevlin2• 14 years ago

Because of non local returns. People abuse them when instead they should be using an exception, or preprocessing their data better.

stcredzero9• 14 years ago

Blocks in other languages can help maintain encapsulation, if they can access variables from their local scope. They can also enable programmers to write their own control structures which look like natural first class control structures in the host language. I don't see this as a natural way of doing either.

sirclueless3• 14 years ago

Nested functions can access variables from local scope in Python, so this isn't a huge issue. It's a bit wonky in python (see discussion of the nonlocal keyword, Python did this wrong from the start) but it solves your encapsulation problem.

    def sorted_by(xs, attr):
        def cmp_attr(a, b):
            return cmp(getattr(a, attr), getattr(b, attr))
        return sorted(xs, cmp_attr)

It takes a little more vertical space and requires you to give a name to something that might otherwise be anonymous, but in the long term it's actually beneficial in my opinion. I've actually taken to using the same pattern in my JavaScript development, giving all of my callbacks names, because it makes maintenance so much easier later.

    var load_into = function (url, elem) {
        var handle_response = function (response) {
            $(elem).html(response);
        };

        $.get(url, handle_response);
    };

lee2• 14 years ago

Can someone give me an example where using a code block is better than using a simple function? And is the added complexity and loss of readability worth it?

wslh• 14 years ago

If the code block can run in a different thread (like in Grand Central Dispatch) it is very useful to do stuff in the background without blocking the GUI (that executes in a specific thread) and using the current scope variables. A progress bar is a clear example of this.

Doing this without code blocks adds extra code like launching another thread, connecting the scope to the new thread.

lee2• 14 years ago

Sorry, I'm not following. Why couldn't you use a function here instead of a code block to do the exact same thing?

d4nt2• 14 years ago

I'm not sure there is one. They're useful in C# as a way of defining a function within another function (which sometimes makes for more readable code). For example:

    public int CountRelevantItems(IEnumerable<Thing> things)
    {
      // Non-trivial filter that you don't want in a where lamdba
      Func<Thing, bool> isRelevant = (t) =>
      {
        ...
      }
      
      return things.Where(t => isRelevant(t)).Count()
    }

But in Python you can already define functions inside of functions so you have a better solution:

    def CountRelevantThings(things):
      def isRelevant(thing):
        ...
      return len(filter(isRelevant, things))

mhurron• 14 years ago

> They're useful in C# as a way of defining a function within another function (which sometimes makes for more readable code)

Why wouldn't you just create another function in the same class?

rbanffy3• 14 years ago

Some people may consider:

    def is_prime(n):
        if n % 2 == 0:
            return False

        sqrt_n = int(math.floor(math.sqrt(n)))
        for i in range(3, sqrt_n + 1, 2):
            if n % i == 0:
                return False
        return True

    def main():
        with concurrent.futures.ProcessPoolExecutor() as executor:
            for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
                print('%d is prime: %s' % (number, prime))

uglier than (borrowing some JavaScript syntax):

    def main():
        with concurrent.futures.ProcessPoolExecutor() as executor:
            for number, prime in zip(PRIMES, executor.map(
                    function(n) {
                        if n % 2 == 0:
                            return False

                        sqrt_n = int(math.floor(math.sqrt(n)))
                        for i in range(3, sqrt_n + 1, 2):
                            if n % i == 0:
                                return False
                        return True
                    }
                    , PRIMES)):

                print('%d is prime: %s' % (number, prime))

I don't, mostly because in the second example, I cannot unit test the anon version of is_prime in isolation. This is a rather trivial case, but I've seen much more complex and enigmatic anon functions in Node applications.

Note: the original comes from http://docs.python.org/dev/library/concurrent.futures.html#p...

aiscott• 14 years ago

"Some people may consider: [example A] uglier than: [example B]."

Do people really think that? I ask in all seriousness. I am a hardware developer by profession and don't often dive very deep in software, though I love exploring software space.

I find example A to be very clear and readable. I like to look at things in bite sized chunks.

I also like to create things in bite sized chunks. In hardware design, specifically digital chip design, I will know what my overall problem is to solve (aka the requirements), but I don't always have a clear view of what pieces I need to get there.

In the cases where I'm not sure about the smaller pieces, I'll write the larger overall structure and when I get to a spot where I'll need a block of smaller functionality, I'll just make up a function name on the spot and pretend it's already written and does the right thing.

Then I take a second pass, implementing those smaller blocks, and keep going down. I guess that's a typical top-down approach? I don't know, but it is what I do.

But I also do bottom up. Even without knowing all the steps between, I'll often have some idea that there are some basic building blocks I'm going to need, and I'll implement those.

So I iterate with some steps top-down, others bottom-up, until I meet in the middle.

All that is a long winded way for me to say that your example A fits both approaches for me. I can start with the overall structure and implement "main" first, or know what my basis will need and start with "is_prime" first.

Whereas example B seems like an all at once approach that is both more difficult to write and read. At least, that is how it seems to me.

edit: wording

mtomassoli8• 14 years ago

I think that "readability" also depends on what you're used to reading and writing. To me, my code blocks are quite readable, but that's only true because I've seen a lot of them in these last two weeks.

Code blocks let you write

  with myfunc(iter) << 'x, y':
    code which uses x and y

instead of

  def some_func_name(x, y):
    code which uses x and y
  myfunc(iter, some_func_name)

If you prefer the second form, then you don't need code blocks. I think the first form is much better. It's not just a matter of saving one single line: you don't have to define a function which you'll never use again, so the first code is actually more readable.

stcredzero9• 14 years ago

    "Here is what my DSL can look like with Blocks in Smalltalk"
    (condition)
        ifTrue: [ ... ]
        ifFalse: [ ... ]
        ifMaybe: [ ... ]

You can accomplish the same thing with named functions and function pointers, but it makes the DSL very awkward to use. Imagine if you had to write all of your conditional logic (if-then and if-then-else and switch) using the names of functions defined elsewhere. That would be much less readable than being able to write block of code inline.

GlennS• 14 years ago

Surely that depends on how well you name the functions?

rbanffy3• 14 years ago

Adding blocks to Python feels like adding classes to JavaScript... Do we really need them?

qznc• 14 years ago

No. You can always use "named blocks", but they are called functions.

lucian19003• 14 years ago

Adding function literals to Python would be nice. Something to turn "inc = lambda x: x + 1" into "inc = def(x): return x + 1".

But blocks? They're a misfeature of Ruby, badly copied from Smalltalk.

[edit: added "return"]

recursive2• 14 years ago

It looks like you changed "lambda" to "def" and added parentheses. It seems like that feature is already there...

lucian19003• 14 years ago

Perhaps my example was crap. They would also allow anything inside them.

llimllib2• 14 years ago

I did something very similar 3 years ago: http://billmill.org/multi_line_lambdas.html

mtomassoli8• 14 years ago

Oh, I know. My module is your fault! :)

llimllib2• 14 years ago

I really like the << x syntax, it's devious :)

northisup• 14 years ago

This seem like a whole series of posts on 'how to twist python syntax to save a line of code or two and make it unreadable'.

fendrak• 14 years ago

This is all well and good, but it doesn't appear to add support for the one thing I've often wanted in Python: proper closure support for functions within functions.

d0mine• 14 years ago

Do you mean 'nonlocal' keyword?

scott_s3• 14 years ago

The code examples are confusing. First, I think there is a typo in the first two examples; I think < should be <<. But he never actually explains what the semantics of << are until several examples in, and it took me a long time to figure out that the string value 'x, y=3' gets evaled into code. Which is strange, to mix strings as code, when the whole point is to have code be code.

At the very least, the semantics of this need to be explained better. When you give a new code example, you always need to say "And this is the result." Otherwise, I can't close the loop; I have new code with new and unknown semantics, and an unknown result. I need to have a known result to figure out the new semantics. (Much like you can't solve a single equation with two unknowns; you either need to have one unknown, or two equations.)

mtomassoli8• 14 years ago

Just read the docstring of the module. It's very detailed.

As for mixing strings with code, I still need to adhere to Python's syntax in order to avoid syntax errors. Anyway, syntax errors in strings are caught at "rewriting time". Using words instead of operators would also cause some problems.

f0r• 14 years ago

Should it be "<< 'x'" rather than "<< 2"?

scott_s3• 14 years ago

You're putting the carriage before the horse. People want to evaluate what it is they're getting into before they download it. If they can't understand the code examples, they won't get as far as downloading the module and looking at the docstring.

I understand why you had to use strings as code. I had two concerns about that. One, you didn't explain it - I had to figure it out on my own. And two, if a solution is supposed to make things cleaner, but ends up introducing warts like that... maybe it's not worth it. What you implemented is interesting, no doubt, but I wouldn't want to use it for that reason.

However, I think you would benefit greatly from having improved examples. Even if people don't use your module, they can still build on your ideas if it's well explained.

mtomassoli8• 14 years ago

By the way, I wish that '<' was a simple typo. Unfortunately it's Wordpress: it's driving me mad. Everytime some code has '<' and '>' in it, something unexpected happens.

optymizer2• 14 years ago

"code blocks" is why they invented functions and procedures half a century ago (and even those differ only in how they return). I think people forget (or don't know) that these are all labels to jump or branch to. So your named functions, functions, lambdas, code blocks, etc, become.. well.. equivalent. Hipsters, don't confuse the young ones with your 30 ways of jumping into a block of code. Cheers!

stcredzero9• 14 years ago

Not the whole story. Code blocks can be used to protect encapsulation and create DSLs which are first class citizens of their host language.

So your named functions, functions, lambdas, code blocks, etc, become.. well.. equivalent.

They can also become so lexically awkward as to be unusable. For example, one code a toy "fuzzy logic" system in Smalltalk with a handful of methods. The result looks like a 1st class member of the language, just like the control structures that are already there. (Same goes for the loops)

    "Here is the standard if-else"
    (condition)
        ifTrue: [ ... ]
        ifFalse: [ ... ]

    "Here is what my DSL can look like"
    (condition)
        ifTrue: [ ... ]
        ifFalse: [ ... ]
        ifMaybe: [ ... ]

Doing this with named functions is going to scatter code between different functions. It's semantically equivalent, but harder to read. Write anything involved in such a way, and it becomes untenable. Blocks can make such DSLs an order of magnitude more readable.

Hipsters, don't confuse the young ones with your 30 ways of jumping into a block of code. Cheers!

A good heaping fraction of hipsters don't understand what's good about code blocks.

optymizer2• 14 years ago

"Code blocks can be used to protect encapsulation". So can functions, which are already in the language.

"Doing this with named functions is going to scatter code between different functions". Please show an example.

"A good heaping fraction of hipsters don't understand what's good about code blocks." That's one way of phrasing it.

bicknergseng• 14 years ago

I also question the utility and reason behind this. I spend a lot of time fixing memory leaks from lousy developers who use closures and anonymous functions incorrectly in JavaScript. I'm just learning Ruby as well, and see a lot of room for similar black-boxing pitfalls in procs and blocks. It seems to me like you're reducing code size at the expense of readability and debugability. That might be the ruby way, but it seems to me like a bad way to program.

Like I said, I'm just learning... so I'd love counterexamples and reasons why I'm wrong.

Code blocks in Python

Related Stories