Show HN: Python library for functional programming

https://github.com/EntilZha/PyFunctional

by allenleein • 9 years ago

129 95 9 years ago

Prepare for 1.0 next release

So this will soon be a stable API? Great work. The documentation is extensive.

I am personally a big fan of toolz[0], which is clojure inspired. This object method chaining Scala inspired approach is different, and interesting.

It might be fun to make sort of comparison matrix of the difference in approach between this library, toolz, and the venerable fn[1] which also has a Scala influence (and apparently already implements the '_' lambda form).

[0] http://toolz.readthedocs.io/en/latest/

[1] https://github.com/kachayev/fn.py

emmelaich• 9 years ago

... and functools/itertools for completeness?

catnaroek17• 9 years ago

From https://github.com/kachayev/fn.py :

There are theoretical and practical advantages to the functional style:
- Formal provability

This is bollocks. To prove things about your programs, you need a formal semantics for the language you're using.

bionoid• 9 years ago

This [Formal provability] is bollocks.

How precisely does modularity, composability and ease of debugging improve from this module? Not at all. It may have benefits, sure. But to claim these as general attributes is just sophistry in my personal opinion.

lmm9• 9 years ago

Functional style makes it easier to have a natural semantics that corresponds directly to the language proper: you can use a small-step semantics where evaluation is just reduction, and you then have a very simple and natural denotational semantics (the denotation of an expression is just the value it evaluates to) that obviously corresponds directly to your operational semantics. All that is much harder to do with programming styles that make use of pervasive, implicit state.

entilzha9• 9 years ago

Author here. 1.0 has been out for a while now, and I'm very committed to API stability. 1.1 (or 1.0.1) should be out as soon as I have some time with mostly minor fixes here and there.

I hadn't considered doing a comparison matrix, but I haven't done a '_' lambda since it seems redundent with the one in fn.

bryanwb2• 9 years ago

shameless plug, if you like this you may also like my library tryme which ports the Either and Maybe monads to Python http://tryme.readthedocs.io/en/latest/

catnaroek17• 9 years ago

Without exhaustiveness checks for all your case analyses, what exactly is the point?

bryanwb2• 9 years ago

The point is that you can indicate whether an operation was successful or not by wrapping that value w/ a Success or Failure, rather returning a boolean, status code, and some subset of information about the operation.

agumonkey• 9 years ago

Funny idea :)

flor1s• 9 years ago

This is quite cool, though it seems similar to RxPy (https://github.com/ReactiveX/RxPY)

vog2• 9 years ago

Curious question: Is there also something like Mobx [1] for Python?

I'd love to have a reactive state libraries like Mobx on server-side.

Mobx is also reactive, but has a different focus: It is not so much avout the low-level event dispatching, but reacting on data changes, recalculating derived values (but only those affected, and being lazy on non-observed dependent values). And Mobx determines dependencies of calculated values automatically.

Also, Vue [2] has Mobx functionality built-in, with an even better syntax, but closely tied to the framework.

Alas, both are JavaScript and targeted at client-side / user interface. Is there something similar for Python? (See also my question on SO [3].)

[1] https://mobx.js.org/

[2] https://vuejs.org/

[3] https://stackoverflow.com/q/47808562/19163

imiric• 9 years ago

I'm not deeply familiar with Mobx, but there's a few Redux implementations for Python. See pydux[0] and aioredux[1].

Not the same thing, of course, since Mobx is much more sophisticated than Redux, but in my team we're currently experimenting with aioredux for state management of client data, which drives the React+Redux frontend. It's working surprisingly well so far, although the biggest hurdle is team familiarity with functional aspects of Redux, and general structuring and responsibility of reducers.

The library is no longer maintained, but considering the simplicity of the implementation and of Redux itself, we felt it wasn't a show stopper since we could always take over maintenance ourselves.

I'd also be interested to know of Mobx Python implementations.

[0] https://github.com/usrlocalben/pydux

[1] https://github.com/ariddell/aioredux

cwyers• 9 years ago

Definitely approve of the name EntilZha for your GitHub user name.

I was interested to see that coexisting with Pandas is supported, but the documentation seems either scant or unfinished, I can't tell which:

https://github.com/EntilZha/PyFunctional/blob/master/example...

fredley• 9 years ago

There seems to be a lot of similar functionality (although in a very different paradigm) to Pandas, I'd be very interested to see performance benchmarks between the two on some common pipelines (csv -> filter -> aggregate -> SQL)

entilzha9• 9 years ago

For operations where vectorization pandas will be much faster since its based on numpy. That being said, there are things which are very awkard to do in pandas which are very easy in pyfunctional

entilzha9• 9 years ago

Thanks on name!

Fair enough on docs being a bit scarce. The main intent is to being able to easily make a pandas dataframe a sequence of tuple/namedtuples, and convert a sequence of tuple/namedtuple to a pandas dataframes.

rajaravivarma_r• 9 years ago

Python should somewhat provide a way of chaining these iterators. Coming from Ruby background, I really miss the ability to chain iterators.

And the backslashes shouldn't be mandatory for wrapping method chains.

make3• 9 years ago

Is this like what your talking about https://docs.python.org/3/library/itertools.html#itertools.c... ?

r41nbowdash• 9 years ago

Trailing backslashes are so ugly

abhsag24• 9 years ago

Is it just me or this seems too much like ruby?

wodenokoto• 9 years ago

Reminds me of dplyr. How well does this work with pandas?

entilzha9• 9 years ago

Data interchange between the two is relatively seeemless. seq(df) converts a pandas dataframe into a sequence of tuples/namedtuples. Calling .to_pandas(columns=cols) will convert a sequence of tuples/namedtuples to a dataframe with cols optionally being the column names

andybak• 9 years ago

Sorry to digress - but the comment in the first code example:

# or if you don't like backslash continuation

got me thinking. I've always felt backslash continuation is a Python wart - but using brackets in this way only feels like a minor improvement.

It's nearly always this scenario that makes me want to break a line - i.e. splitting a long chained method call on a "."

Now - starting a line of code with a "." is not valid Python as far as I'm aware and therefore allowing it would not introduce any ambiguity or significantly complicate Python's parsing rules. "If a line begins with whitespace followed by "." then consider it as a continuation of the previous line.

Can anyone comment on whether this is a terrible idea or not?

andreareina3• 9 years ago

This is a peeve of mine, and one of the reasons that I find myself preferring to write javascript these days rather than python. Never would have thought I'd say that.

I'm not up to snuff on my compiler theory, but I think at the least this would require upgrading to a two-token lookahead parser, and it might also upgrade the python grammar from context-free to context-sensitive. Accordingly, I wouldn't expect this to get implemented -- IIRC ease of parsing was/is one of the big design goals.

adrianN• 9 years ago

Line continuations could be treated by the Lexer.

Doxin• 9 years ago

In python line continuations are a pre-processing step as far as I'm aware (together with inserting INDENT and DEDENT tokens). In the same vein you could just do a simple loop over the token list and remove the appropriate NEWLINE tokens before going to the actual parsing.

yen223• 9 years ago

Whitespace-sensitive blocks aren't easily parsed either.

baldfat• 9 years ago

Now - starting a line of code with a "." is not valid Python

Absolutely agree. I really like the multi character piping at the end of lines that would get rid of the .

R has %>%

F# uses |> and for functions >>

I would love to see R and all languages just pick up and use F# |>

gaius2• 9 years ago

I think the %x% style is baked into R to differentiate built in and custom infix operators - %>% comes from Tidyverse IIRC.

FRex3• 9 years ago

You can omit the leading 0 before the dot when writing float literals so just a '.1' or any other such sub-1 float literal with no leading 0 is technically a valid but useless line of code starting with a dot (you could use it instead of pass I guess..). I can't think of a real use of a line starting with a dot off the top of my head.

bpicolo• 9 years ago

Identifiers can't start with a number, so this should still be unambiguous probably?

owl57• 9 years ago

Probably nothing remotely resembling this belongs in anything remotely resembling production code, but:

  In [1]: class Foo:
     ...:     def __rmul__(self, other):
     ...:         print("%s foos!" % other)
     ...:
  
  In [2]: .42 * Foo()
  0.42 foos!

BeetleB• 9 years ago

Pythonistas usually recommend you not use a backslash for continuation. The approach shown on the page is the recommended way, and I find it roughly about as readable as what you suggest. Does using parentheses for continuation trouble anyone?

yedpodtrzitko• 9 years ago

I'm not into functional programming in Python, but pipetools[0] looks way more handy.

[0] https://github.com/0101/pipetools

danpalmer• 9 years ago

It looks like theres not a huge amount of overlap.

Pipetools is all about implementing a "clever" syntax for data flows, whereas PyFunctional looks like it's much more about providing a library of functions for manipulating data in functional ways.

I would expect that PyFunctional could probably implement the pipe syntax if they wanted, as just a layer of syntactic sugar on top of their core, however that would break OR semantics in places so I'm not convinced that syntax is a good choice, as "clever" as it may be.

g4k• 9 years ago

Also Coconut looks interesting: http://coconut-lang.org/

andreareina3• 9 years ago

First off, well done.

My big issue with method chaining is exactly the one of requiring the line-continuation-escape, or enclosing the whole thing in parens. You also need to wrap the initial value, then possibly unwrap it at the end, and you're limited to the methods you define (mind you, this seems like a pretty comprehensive list).

For this reason I decided to emulate clojure's threading macros instead, as a pair of functions:

    pipe_{first,last}(
        init_val,
        some_unary_callable, # so far so familiar
        (some_n_ary_callable, *args), # puts the piped value first or last, depending on {first/last}
        ...
    )

Unfinished is the version that makes `(callable, ...args..., _, ...more_args...)` work, is that what you mean by the _ lambda operator?

senand• 9 years ago

Are you aware of the thread_first and thread_last functions from toolz?

andreareina3• 9 years ago

I wasn't thanks for the pointer!

entilzha9• 9 years ago

Not OP, but author. Line continuation is pretty annoying. Adding custom methods wouldn't be hard (https://github.com/EntilZha/PyFunctional/issues/113), I just don't have time right now to add it.

With respect to _, this would make these two equivalent: a = lambda x: x + 1 b = _ + 1 a(1) == b(1)

Is there an expanded example I could look at for the pipe stuff, I haven't used clojure before

valw• 9 years ago

This looks a lot like Clojure's standard library, and you could probably steal more ideas from it, e.g transducers https://clojure.org/reference/transducers

amelius• 9 years ago

Am I the only one who thinks Python's syntax is not really suited for functional programming? Creating an ad-hoc closure that spans multiple lines, for example, seems awkward.

PS: I'm not aware of any syntax changes in this area since Python 2.7, so I could be wrong.

marshray• 9 years ago

I’ll say it: Python sucks for Functional Programming. It’s just not the right tool for the job (or job for the tool, depending on your point of view.)

It’s a great language for most programmers to get things done in, but it belongs to the school of ‘limited abstraction is for your own good’. For example, no multi line lambda expressions. The argument given is that it would mess up whitespace syntax. Haskell handles this situation, but it does indeed introduce a bit more syntax.

If you need to FP in a general purpose scripting language, I would be looking at Elixir or Javascript right now.

entilzha9• 9 years ago

True, but for better or worse python has support for spark/sklearn/pandas/numpy/plotting/deep learning/web which makes it a good jack of all trades language for data/research science. Its strength is that it can do all of those at levels varying from proficiently to best-in-class.

gigatexal• 9 years ago

Thank you to the author: another day another. Wat library to tinker with.

fermigier• 9 years ago

Shameless plug: https://github.com/sfermigier/awesome-functional-python (a list of documents and projects related to FP in Python, including, already, the OP's project).

I'm personally using Toolz so far, probably because it's the most popular of the maintained ones.

Would be happy to see both some standardisation (a PEP?) and consolidation between the various projects.

entilzha9• 9 years ago

It would be great to have more native support for FP (flat map is a glaring example I run into a lot). It would help if there were 1) a less verbose anonymous closure syntax 2) support for a multi-line closure.

flavio81• 9 years ago

Very nice. I just wished Python would have a proper lambda operator. As far as i know they still have that "one-line" lambda keyword which is very restrictive for functional programming.

sevensor3• 9 years ago

I'm no expert in functional programming, but this feels a bit inside-out and backwards to me. It's writing f(g(h(x))) as x.h().g().f(), which doesn't read like function composition to me. If anything, the comprehensions already built into Python look more natural and more functional:

   (f(v) for v in (g(u) for u in (h(w) for w in x)))

The dots in the x.h().g().f() notation also suggest mutation (since we're calling methods on objects here) in a way that makes me uncomfortable.

But like I said, I'm no FP expert. My objections are basically aesthetic. Maybe if I used it, I'd learn to like it.

chasedehan• 9 years ago

The way you read x.h().g().f() is essentially the same way you read it in F# (with |>), Haskell(with . ,although it goes the opposite direction) and R(with %>%).

It is one of the most aestetically pleasing attributes of use in the functional languages

sevensor3• 9 years ago

So the thing that looks less like the function composition I know from algebra is more idiomatic in FP and more pleasing to the FP cognoscenti? I fear my familiarity with FP may be even more deficient than I thought. I could understand writing f . g . h, but what we're talking about here is written more like Unix pipelines. In which case, I could forgive overloading the bitwise or operator, which is already left-associative. You could then do x | h | g | f.

Again, this is a purely aesthetic argument -- the intent of the library seems unclear to me from its syntax. If it's beautiful and clear to you, I'm not going to object to your using it. Aesthetics aside, I don't think there's anything wrong with this library.

kccqzy2• 9 years ago

How does its parallelization feature work with lazy generators? Or does its seq function not return generators?

entilzha9• 9 years ago

Seq doesn't return a generator. At its core there is a concept of Lineage taken from Apache Spark. Essentially, when you do something like seq(data).map(func) it builds on a list of operations to execute and holds a copy of the base data. When asked for a value only then will it compute the result.

So for parallelization operations that are embarassingly parallel (map/filter etc) are paralelized with multiprocessing. If the data is heavy and serialization is expensive it might be slow, but for operations where the bulk of the work is done in parallel it can help a lot.

jjuel• 9 years ago

Is there a reason to use this over Coconut? http://coconut-lang.org/

brightsize• 9 years ago

Coconut is a compile-to-python(2/3) language, not just a library, so if you use it you have to be OK with using a transpiler in your project.

Here's one way you might write one of the examples in Coconut:

  ([1, 2, 3, 4]
      |> map$(-> _ * 2)
      |> filter$(-> _ > 4)
      |> reduce$((x, y) -> x + y)
  )

You may freely mix Coconut and Python in a project, even in the same file, but any Coconut code still needs to be run through the compiler.

Show HN: Python library for functional programming

Related Stories