Pipeline-Oriented Programming [video]

https://www.youtube.com/watch?v=ipceTuJlw-M

by ingve • 3 years ago

61 48 3 years ago

tandav• 3 years ago

shameless plug: I maintain a small library to do functional pipes. You can write:

    (
        range(10)
        | Map(lambda x: x * 10)
        | Filter(lambda x: x % 2 == 0)
        | Reduce(lambda a, b: a + b)
    )

and more. https://tandav.github.io/pipe21/

most operators are basically oneliners: https://github.com/tandav/pipe21/blob/master/pipe21.py#L17

I also maintain a list of libraries which uses pipeline/chain syntax: https://tandav.github.io/pipe21/similar-tools/

throwaway914• 3 years ago

This is very cool. I remember I got sucked into things like Ramda going down this functional programming rabbit hole :-)

https://ramdajs.com/

retrocryptid4• 3 years ago

The first thing I did when I encountered js in the 90s was write map and reduce. After leaving Mozilla I published a npm package with my hacks. I was very happy to find someone else had done a much better job at it and was actually maintaining it.

also, have you seen lazyjs? Also very cool.

scns• 3 years ago

Slides:

https://fsharpforfunandprofit.com/pipeline/

Code:

https://github.com/swlaschin/pipeline_oriented_programming_t...

akkad33• 3 years ago

This seems to be a rare update to the site which hasn't seen new content in more than 3 years?

bob10292• 3 years ago

Expressions (LINQ) are the most powerful feature available in C#. There isn't much you can't do with them if you get a bit clever. I can't imagine how many total hours of development time these abstractions have saved. I also find it relatively easy to explain these bits of code to non-developers. Performance is a common argument for not utilizing code like this, but modern .NET does all kinds of insane runtime optimization techniques to make sure things go fast (if they need to!).

We just recently started looking at using EFCore (instead of raw SQL via Dapper) and it is dawning on me that it was a strategic mistake to not further utilize the leverage that expressions can bring to the party.

louthy3• 3 years ago

If you want to see this taken to the max, then my library [1] has many monadic types (which is what LINQ is for) implemented. Including an implementation of Haskell Pipes [2], with some examples [3]

[1] https://github.com/louthy/language-ext

[2] https://louthy.github.io/language-ext/LanguageExt.Core/Effec...

[3] https://github.com/louthy/language-ext/blob/main/Samples/Eff...

lacrimacida• 3 years ago

For some reason your github link repeatedly crashes for me. Not sure whether it’s because im trying to view it from my iphone..

ako• 3 years ago

Seems like you could do the same in Sql with CTEs or views, with the benefit of the query optimizer finding the fastest way to execute your pipeline on a large dataset.

electroly• 3 years ago

modern .NET does all kinds of insane runtime optimization techniques to make sure things go fast

Not as much of this applies to LINQ as we'd all wish. If you've got LINQ in a performance hotspot, the first thing to do is remove the LINQ. I love LINQ but plainly written loops are faster every time, and removing LINQ from profiled hotspots has always been a successful optimization for me in practice. Avoid it in performance hotspots. Once you've removed the LINQ, eliminate any inner-loop heap allocations and you're probably done optimizing. LINQ and heap allocation are the two major killers of performance in typical C# code.

AndrewDucker2• 3 years ago

This is also how PowerShell works. It's incredibly effective for generating/fetching data and processing it.

6c696e75782• 3 years ago

Unless things have changed the pipelines in powershell are collectors, unlike fifos in unix. For small tasks it's fine, but when shifting large blocks of data it can be noticeable slower.

AndrewDucker2• 3 years ago

The pipelines pass one item at a time unless you deliberately make them into collectors.

louthy3• 3 years ago

Not sure why Scott Wlaschin needs to keep renaming existing concepts ('Monads' to 'Railway Oriented Programming' and 'Function Composition' to 'Pipeline Oriented Programming'), but I really don't think it helps shared understanding.

It feels, at worst, like the height of arrogance to assign new terms to existing concepts, as though you've just invented the concept. And, at best, a form of anti-intellectualism - to try and make functional concepts accessible to ex-OO devs. But it feels to me like all it does is muddy the waters.

Not everything needs to be oriented programming.

jmaker• 3 years ago

It might be purely marketing-oriented, or perhaps it’s just sales-funnel-oriented, who knows. And if anything unexpected pops up, look no further, the author can immediately come up with a new name for a decades old solution. Same happened to the “hexagonal,” “ports-and-adapters” and so on, which I noted mostly in the .NET community btw, correct me if that's wrong. Like you know every new consulting agency coming up with some new names for old stuff to stand out perhaps?

eterps• 3 years ago

For what it's worth, it helped me a great deal in understanding.

AnimalMuppet2• 3 years ago

Yeah. If the new term conveys the actual meaning better than the old term, then why not invent a new term when dealing with the new people? We didn't always fully understand what X was when we coined X as the term for it; sometimes better names are possible. Why not start to use them?

Or, probably more often, a new term for the same old thing is not better, serves no useful purpose, and just confuses.

retrocryptid4• 3 years ago

People keep rediscovering and rensnming lisp features. I do think he's talking more about composition as a design concept, which is good for people who have only heard shout OO their entire career.

But yeah, it was weird hearing about this "new" idea I've been using for 30 years.

I worry that lazy evaluation will wind up as "filter based programming" next year.

But if it puts it in a form python or c# programmers can readily consume, it's probably okay.

Also nice to see someone doing something with f#.

andsoitis• 3 years ago

Spatial Computing is another such ploy to differentiate at the surface level.

While I tend to agree with your critique I also appreciate that a different lens can be more accessible or less held back by historical baggage.

whartung• 3 years ago

A perfect example of this was back when I had my breakthrough with Scheme and Lisp.

I always struggled with fundamental terminology used in this family of languages, particularly lambda. Lambda as a term does carry historical baggage. The term wasn’t picked out of thin air. And casually looking at Lisp and Scheme code, if you don’t know that baggage, lambda is (at least to me) an opaque term.

But I luckily stumbled upon a book “Simply Scheme”. And the first thing they did was redefine a bunch of terms. No more CAR, CDR, CADR. Instead you got things like FIRST, REST. SECOND. Importantly, they redefined LAMBDA to FUNCTION.

Well, Shazam. Ding, ding, ding. This was an “aha” moment for me. Boy did a lot of stuff suddenly fell into place.

So, simplifying the vocabulary can be very helpful for introducing gateway concepts.

mdarens• 3 years ago

He's not insisting that people already familiar with functional programming concepts not use the terminology they have historically used, he's using concrete metaphors as a teaching strategy.

karmakaze• 3 years ago

Monads/Railway seems one step more than Pipeline in concept. The latter is merely connecting outputs to inputs (though I didn't get to the end of the video). Monads/Railway also handle other aspects such as errors (or side effects etc).

[It is very 'branding' and a disservice to not to mention the popular terms.]

I do see them all as the style I think and write in.

ioyourtoe• 3 years ago

It feels, at worst, like the height of arrogance to assign new terms to existing concepts, as though you've just invented the concept.

Hilariously I’ve done exactly that for the exact opposite reason.

I have in the past “come up” with a programming paradigm that perfectly satisfied a need and when I went to tune it up for general use I realized it was similar to a concept I was unfamiliar with, or struggled to grasp previously or previously hated working with in the manner it was taught or interfaced.

~”This looks like a monad, maybe I’ll call this a monad, but I’m unsure what precisely is the mathematical proof supporting, or what the limits and boundaries of the concept of a monad are so to avoid someone mocking me saying ‘this is the worst monad implementation, it’s hardly even a monad, it’s like a low effort subset of what a monad is.’ I’ll just name it something else.”

simplify• 3 years ago

at best, a form of anti-intellectualism

This is an uncharitable viewpoint. Bringing FP concepts to more devs is a good thing. I don't see how teaching better ways is any form of anti-intellectualism.

louthy3• 3 years ago

Because (if the assertion is true) he eschews the terminology used by the FP community for his own wording in an attempt to pander to people who aren't (apparently) willing or able to learn the existing terminology. In my FP library [1] (targeting C# programmers) I regularly get people asking "does the library support Railway Oriented Programming?", which they (if they'd have been taught the terminology used by the FP community and not misled by Scott Wlaschin) wouldn't ask after 10 seconds of looking at the readme to see the word 'monad' all over it.

It's treating OO programmers, who are curious about FP, as though they can't handle anything unless it's got "oriented programming" after it. In my mind that's a form of anti-intellectualism.

I really don't buy that it's needed at all. And I feel I'm in a pretty good place to state that, language-ext has ~6K stars on GitHub and the package has been downloaded nearly 13 million times [2]. It shows that devs that use an OO-first language can and are able to learn FP terminology. I never pandered and I get many OO devs coming to the library. Yeah, sometimes they need some guidance, but I've never needed to invented new terminology.

OO engineers are just as smart as FP engineers, they've just worked in different paradigms and need to learn the lingo when switching - the sooner they do it the better able they are to communicate with others in the FP community.

[1] https://github.com/louthy/language-ext

[2] https://www.nuget.org/packages/LanguageExt.Core

Jtsummers• 3 years ago

https://en.wikipedia.org/wiki/Pipeline_(software)

It's not a renaming. Pipelines have been a thing in software for ages. Correlating differing (surface) systems to each other is very useful when it turns out that they're expressing the same "thing" but in different manners.

ReleaseCandidat• 3 years ago

Not sure why Scott Wlaschin needs to keep renaming existing concepts

That's because most programmers ignore you as soon as they hear the m-word (or Haskell), as all that useless in "the real world"(TM).

corethree• 3 years ago

This what I hate most about the field. Everyone trying to look smart by using big words for stupidly obvious concepts. OO is especially guilty for this with all the design patterns.

Every so often on hacker news you get some article like this that tries to introduce a "new concept" that they didn't know has already existed for the longest time or you get some other person writing a blog post about some programming analogy how programming is like "theory building". I just roll my eyes. I really wonder why you only see programmers who like to think of themselves as "highly intelligent" and calling what they're doing "theory building" and why civil engineers or electrical engineers don't supply this endless vault of "analogy blog posts" for HN readers to self pleasure themselves to.

jbverschoor• 3 years ago

Strange that Elixir pipes are not mentioned at all..

fredrikholm2• 3 years ago

Why? The talk is about F#.

retrocryptid4• 3 years ago

There's a community that has experience doing analysis, design and coding with functional composition as an abnstraction. It's useful to reference that community so the viewer can get more information and more examples on applying it.

Also, sooner or later people will notice difficulties in communicating exceptional events up and down the chain and they would be greatly served by how Erlang/Elixir handles errors and process trees. Even if you want to stay in c# or f#, there's still wisdom to be gained by looking at how things work in functional languages and applying what you can in non functional languages.

bjourne2• 3 years ago

Pipelines are effective for expressing functional concepts but are not suitable for general programming. E.g writing merge sort or a chaining hash table as a pipeline won't work out well. The problem is state. There are dataflow programming languages like Lustre and Esterel that tried to incorporate state as back arrows in their pipelines. It didn't work out so well.

anonzzzies3• 3 years ago

But most things most people do during the day is not general programming. it would benefit from having to use principles like this instead of using 'general programming' to do them in a way that's harder to understand, harder to debug etc.

AnimalMuppet2• 3 years ago

But most things most people do during the day is not general programming.

Well, in one sense, nobody ever writes a general program - they only write specific ones.

But if your claim is "most things most people do during the day could be pipelined", I'd like to see your supporting evidence, because I'm not at all sure that claim is true.

For example, as far as I understand, GUIs are very hard to pipeline. How much of the programming done in the average day is on GUIs?

bjourne2• 3 years ago

I don't believe in creating methodologies for blub programmers which they nevertheless have to abandon to do "real" programming. 90% of the problems comes from 10% of the code so that's what you got to address. I.e. Give me a blabla-oriented programming paradigm that makes writing efficient merge sorts easier and I'm all ears, one that can't even realistically do it just makes me yawn.

mrkeen2• 3 years ago

"The problem is state" is precisely why I do my general programming with functions.

marginalia_nu• 3 years ago

Isn't the call stack also a form of state?

trashtensor• 3 years ago

There are some things it's useful for and some things it isn't. Go figure.

AndyKluger• 3 years ago

If anyone's interested to see the slide examples but in Factor, I wrote them out:

https://programming.dev/comment/6718327

Here's a bit:

  : add1 ( x -- x' ) 1 + ;
  : square ( x -- x' ) dup * ;
  : double ( x -- x' ) 2 * ;

  : demo1 ( -- n )
    5
    add1
    square
    double
  ;

Pipeline-Oriented Programming [video]

Related Stories