Reducers, transducers and core.async in Clojure

http://eli.thegreenplace.net/2017/reducers-transducers-and-coreasync-in-clojure/

by ingve • 9 years ago

163 47 9 years ago

eli.thegreenplace.net

whalesalad3• 9 years ago

Disclaimer: as a Clojure developer and functional programming evangelist, I do not want to be too critical of this article. It's a really great post about some incredibly valuable and advanced Clojure features.

That being said, it really bothers me when functional programming evangelists write out such horrible Python code examples. It makes Python look like it's not a functional language. It perpetuates the idea that Python won't let you write elegant and stateless programs that behave in a functional manner.

If the author had not mentioned Python and had instead used it as pseudocode to represent the entirely non-functional way to write something, I would have been okay with it. But calling out Python specifically is just incorrect!

I have rewritten the code in a very pythonic manner that illustrates the functional capabilities of the language:

    even = lambda x: x % 2 == 0

    # alternatively, depending on your religious beliefs:
    def even(x): return x % 2 == 0

    def process(seq):
      return sum(x + 1 for x in seq if even(x))

Note the (parens) for the comprehension instead of [brackets], which creates a generator. The above code is lazy (in Python 3)!

And if we fire up a python repl and play around:

    In [2]: process(range(10))
    Out[2]: 25

I really love Clojure and use it daily, but I find the Python version to be far more legible. It reads more like english.

Anyway, the moral of the story here is that you can do immutable/functional programming with regular ol' Python.

projectorlochsa• 9 years ago

you can remove the extra parentheses after sum.

mod• 9 years ago

I think the inner set is supposed to be brackets [].

It's a comprehension.

nimish• 9 years ago

Named lambdas are discouraged as un-Pythonic for what it's worth.

The other point in the article was that your process has now hard-coded even and sum, and composing them in python is a bit unwieldy.

I've run into a number of cases where it's just easier in python to write out the loops than string comprehensions or map/filter/reduces together because there's no threading macro or haskell's functional apply operator.

whalesalad3• 9 years ago

You can compose it all together in a relatively straightforward manner using functions in the stdlib. I agree that it is nicer with a threading macro, but is this so bad?

    def even(x):
      return x % 2 == 0

    def inc(x):
      return x + 1

    def process(seq):
      return sum(map(inc, filter(even, seq)))

Veedrac4• 9 years ago

Please use def over lambda when feasible (which is most of the time); using lambda needlessly hurts debugability and saves a total of three characters.

    def even(x): return x % 2 == 0

flavio812• 9 years ago

Yes but lambda allows some versatility, for example (excuse my Python, it is a bit rusty):

    formulas = {
        'sum': lambda x,y: x+y,
        'subst': lambda x,y: x-y,
        'mult': lambda x,y: x*y,
        'pow': lambda x,y: x**y
        }

    def apply_operator(operation_name, x, y):
        operation=formulas[operation_name]
        return operation(x,y)

Using "def"s there would kill readability and easiness of coding.

owebmaster• 9 years ago

It is not about char length. The first is an expression while the second is a statement. I find it easy to debug code with expressions rather than statements, could you clarify why the first way would be worse for debugability?

eriknstr4• 9 years ago

Furthermore I would prefer

    def even(x): return not(x % 2)

specialp• 9 years ago

I get what you are saying, but I think the author was not trying to do that.

The example was meant to show the imperative approach which is not necessarily the best approach in Python either. They said so much in the paragraph before the example. Yes, Python was used and is used often to show imperative examples. Python is used in these examples because it is also dynamically typed, and understood by a lot of programmers. The reason it is not given in Clojure is that it is pretty hard to make imperative examples in Clojure.

Also, coming from Ruby, the way the "process" function was posted is unfortunately the way many Ruby programmers would do it despite access to filter, map and blocks.

azag0• 9 years ago

Or, with a little setup, the Closure snippet

    (def s (range 0 10))
    (reduce + (map inc (filter even? s)))

can be turned into Python's

    from functools import reduce
    from operator import add

    def inc(x): return x + 1
    def even(x): return x % 2 == 0


    s = range(0, 10)
    reduce(add, map(inc, filter(even, s)))

Everything's lazy in there. It's just... not very Pythonic.

jerf• 9 years ago

"It makes Python look like it's not a functional language. It perpetuates the idea that Python won't let you write elegant and stateless programs that behave in a functional manner."

I suggest Go for this use case. It's easy enough to read for this sort of thing even if you don't know the language, easy to link to running examples on the playground, and nobody will accuse you of showing Go off in a worse light than is called for because the Go equivalent of the imperative example shown really is the Go way to do it. Go is just about the most aggressively non-FP language out there today, making even Python look friendly to FP programming, despite the fact it has closures.

eliben4• 9 years ago

Thanks for the comment. I totally agree that Python can express this with its functional primitives as well. I was simply using it to show an "imperative style", and Python is my favorite executable pseudo-code.

[P.S: I'm a Python core developer and long-time user; calling Python out is really, really not my thing!]

willtim• 9 years ago

"you can do immutable/functional programming with regular ol' Python"

One cannot easily and safely do functional programming in Python due it's imperative semantics. For example variables are captured as mutable references and even a list comprehension will mutate its bindings.

If you want to do serious functional programming, use Clojure, Haskell and/or OCaml.

Edit: to the downvoters, it's about using the right tool for the job.

didibus• 9 years ago

Python is multi-paradigm, I think everyone would agree, but I think they removed the ability to do this with function composition like map, reduce and filter in v3 no? Specifically in favor of list comprehensions and generators?

Anyways, if you do it with list comprehensions in Clojure it would look like this:

(apply + (for [x (range 10) :when (even? x)] (inc x)))

eliben4• 9 years ago

In Python 3 functions like reduce were moved to the functools module, not removed from the language.

bad_user• 9 years ago

perpetuates the idea that Python won't let you write elegant and stateless programs that behave in a functional manner

Well, in my opinion Python really does not let you write elegant and stateless programs that behave in an FP manner.

The sample you've chosen in particular is all fine, except that `lambda` expressions are almost never used, due to Python not being expression oriented, so there isn't much you can express with a `lambda`.

You might find your `for x in seq if even(x)` expression elegant, but it is preceded by a `return`. Python is a statement / side effects oriented language and even in your one liner, it shows.

Oh, and given that a Python "generator" is like Java's Iterable / Iterator, that's not FP either, although you can argue that the mutation can be localized (e.g. in your `sum` example), but you have to be careful about it.

And if you'll take a look at the whole Python ecosystem, there's not much FP in it to be honest. I mean, seriously, out of all the mainstream languages, I can't think of a worse language to do FP than Python, except for C maybe,

merlincorey• 9 years ago

Check out toolz[0] for a Clojure inspired FP toolkit in Python.

[0] http://toolz.readthedocs.io/en/latest/

blowbaybestban• 9 years ago

for full on functional style, as far as I am concerned, process() should be recursive (without all the overhead that implies in an imperative language). I'm being sarcastic because I don't like the list comprehension syntax.

raspasov3• 9 years ago

I have probably read most transducer explanatory blogs that ever existed and watched many, many talks on transducers, etc. I have read books about them (as in book chapters about them). I use the built-in transducers almost every day and I consider them an essential tool. I have written the occasional transducer myself for certain purposes.

This is the best transducer explanation and breakdown I’ve seen!

Very well structured and IMO easy to follow for anyone who understands Clojure code already. Great explanation and progression that builds towards the full transducers picture! (edit: typos)

eliben4• 9 years ago

Thanks, this is the kind of feedback I most enjoy getting :-) Glad you found it useful!

raspasov3• 9 years ago

I believe this is the type of writing that the Clojure community (both existing members and newcomers) can benefit a ton from! You should do more of it! :-)

smnplk• 9 years ago

You need to checkout ClojureTutorials channel on youtube, where Timothy Baldridge goes very deep into transducers. But the videos are not free, I think there is a small subscription.

raspasov3• 9 years ago

(I have subscription for Timothy’s tutorial videos) I have watched most of them as well some time ago, they are great also!

I really like this one because of the “first principles” approach of starting with reduce and building from there.

WaxProlix3• 9 years ago

Possibly stupid question:

At the top of the article, we see `(reduce + (map inc (filter even? s)))` -- here, `reduce` is a two-place predicate, taking a function and a collection.

Later, `reduce` is taking a function, a 'base case' [], and a collection -- it's a three-place predicate, more in line with a signature for `fold` than what I'm used to seeing for `reduce`.

Is this a clojure specific overload thing? I'm not at all an expert in FP or anything so it might be fairly standard.

dragonwriter2• 9 years ago

It's fairly common in functional programming languages for functions to be differentiated both by name and arity (reduce/2 being different from reduce/3.)

WaxProlix3• 9 years ago

I feel like this is fairly opposed to things like partial application and currying in general, no?

eliben4• 9 years ago

Take a look at https://github.com/clojure/clojure/blob/master/src/clj/cloju...

TL;DR: Clojure supports multi-arity functions (sort of like operator overloading in C++, but overloading for the number of operands, not their types)

When no default value is given, the result of reduce on the first two elements is used.

piercebot• 9 years ago

Something I found interesting when playing with the example code is that, while folding in parallel certainly speeds up the execution for realized sequences, it may not make sense to realize a sequence that you're only going to use for one operation:

  user=> (require '[clojure.core.reducers :as r])
  user=> (def s (range 9999999))

  user=> (time (r/fold + (r/map inc (r/filter even? s))))
  "Elapsed time: 450.318641 msecs"
  25000000000000

  user=> (time (r/fold + (r/map inc (r/filter even? (vec s)))))
  "Elapsed time: 463.632907 msecs"
  25000000000000

Note the `(vec s)` instead of just `s` in the second timed run above

  user=> (def sv (vec s))
  user=> (time (r/fold + (r/map inc (r/filter even? sv))))
  "Elapsed time: 156.731683 msecs"
  25000000000000

Far from a ~3x speedup (in these contrived examples), realizing the sequence in-line yields approximately the same performance as if it was operated on lazily.

Something to keep in mind if you're trying to optimize your Clojure. This is still the best resource I've read on reducers/transducers :)

[edit: code formatting]

wellpast2• 9 years ago

The performance disparity is even more dramatic on my machine. What's the reason for that - is it b/c of concurrent contention during vec/realization? Or ???

wellpast2• 9 years ago

For posterity, I just answered my own Q. Performance overhead was just the cost of the linear cost of vec realization as you pointed out. At first it seemed like I was seeing additional overhead.

Reducers, transducers and core.async in Clojure

Related Stories