Flow-Based Programming

https://jpaulm.github.io/fbp/index.html

by hypomnemata • 6 years ago

217 103 6 years ago

jpaulm.github.io

Similar concept to what I do when programming in F#/OCaml by default. You create your data types and then they "flow" through functions. Each function is `input -> output` but bigger workflow are also input -> output.

adamnemecek3• 6 years ago

The main difference is that with data flow, the flows happen conceptually at the same time.

hpoe• 6 years ago

As I was reading this I was enthusiastically agreeing with the idea, but something felt super familiar about it. Then I realized this is basically the same concept as Unix pipes.

I have my little individual programs, grep, awk, sed, jq, etc, and then I can endlessly mix and match those different components to do what I want.

The limitation that I have seen with Unix pipes isn't often with the ability to process or manage the data it is that it only works if all the data is setup the proper way. As I have been in the industry longer it seems to me that most code that gets written isn't about actual computing but just centered around importing and transforming data that is expressed in different ways.

Is there something that makes reading in disparate data records easier so I can focus on the computing part of computer program and less time on parsing data?

adamnemecek3• 6 years ago

Unix pipes make it hard to have a large graph. Like most pipes take one thing and produce one thing. You don’t really have complex data flow graphs.

taeric• 6 years ago

Though, this really kind of fits with the metaphor. Pipes typically just send liquid.

You could have a conduit connector to join a bunch of wires/signals, I suppose. But, realistically, that is a mess always.

lispweeniejr• 5 years ago

you aren't wrong, but i wanted to point out that through `tee` and `cat` you can split or merge pipelines, and with FIFOs you can awkwardly create graph cycles

phreeza• 6 years ago

Apache Beam seems to be an implementation of this idea. It works well when the things you have to do matches the logic, but it gets tricky if you need to do stuff iteratively or recursively.

samuell11• 6 years ago

Beam has some overlap, but in my understanding it has a rather involved syntax for defining the data flows, quite far from the simple list of connections between in and out ports that you see in FBP systems following J P Morrisons principles more closely.

Apache NiFi comes quite a lot closer, with the main difference that they only have a single in-port, instead of separate named ones. Also it seems to be among the more heavy and complex implementations (for good and bad).

jpaulmorrison2• 6 years ago

"They" meaning "FBP-like" or "FBP-inspired systems", to use Joe Witt's terminology... They are usually control-flow oriented, and to my mind do not yield the productivity and maintainability advantages of the "classical" FBP paradigm shift. I recently asked Node-RED's Nick O'Leary why Node-RED only allows one input port, and the answer was something like "because we have never run into the requirement"... a) this is not something that can easily be added after the fact, without totally rethinking the product architecture, and b) trying to build complex systems without that feature would be, to me, like trying to hang wallpaper with only one hand! One litmus test I use is that of concatenating data streams using the two different paradigms - I tried to describe this in an article a year or so ago, recently updated: https://jpaulm.github.io/fbp/concat.html Cheers!

bergie2• 6 years ago

Nice to see this here, as it has been the inspiration for quite a few years of my work: https://noflojs.org/

leetrout• 6 years ago

NoFlo really is impressive. Great work.

bmitc3• 6 years ago

Do you have a shining example of someone using NoFlo? It doesn't have to be big, but one that you think really exemplifies what it excels at.

(By the way, I'm a big fan of visual programming, so I'm just curious.)

bergie2• 6 years ago

There has been quite a lot of IoT automation / rule engine stuff done in NoFlo in the last couple of companies I’ve worked at, but obviously I can’t really go into specifics there.

Then there was this startup way back when: https://www.cloudamqp.com/blog/2017-05-31-work-queues-in-rab... https://librearts.org/2014/10/artificial-intelligence-design...

On the front-end side, Flowhub uses NoFlo for state management, a bit similarly to how you’d use Redux: https://github.com/noflo/noflo-ui

amelius• 6 years ago

Flow-based programming is cool, until the flow starts altering the flow.

cjohnson318• 6 years ago

I think at that point it's a complex dynamic system.

samuell11• 6 years ago

The core of the FBP principles are the holy grail of true componentized function architecture.

Instead of losing yourself in ever more complex syntax convolutions as has happened in a lot of functional programming, you make the components (even long running stateful ones) self contained with ports for data input and output as the sole means of communicating with them, over buffered channels to allow asynchronous computation, and most importantly keep the network definition separate.

Just this idea in itself is just brilliant. Hats off to Mr Morrison for that!

It allows to decouple complex software into reusable components, without clever FP syntax.

(Though, FP is perfect for implementing the components themselves. It just doesn't really scale all too well for whole program architecture, in my experience).

One point to note: The visual component of many fbp systems is completely optional, as is the idea of using novel DSLs. You can as well define your networks and components in pure code. See GoFlow https://github.com/trustmaster/goflow) and my own little experiment FlowBase (https://flowbase.org) for examples of that, in Go.

I successfully built a rather complex little app to convert from Semantic web RDF format to (semantic) mediawiki XML dump format, in two weeks straight, of linear development time: for each component (of ca 7), implement, test, go to the next component (See: https://github.com/rdfio/rdf2smw)

The same implementation in procedural PHP took months, and still doesn't have all bugs and strange behaviours filed out.

the_duke• 6 years ago

you make the components self contained with ports for data input and output as the sole means of communicating with them, over buffered channels to allow asynchronous computation

This concept sounds exactly like actor systems like Erlang/OTP and Akka, only with a different set of terminology.

The submitted site and your comment don't mention those anywhere.

Are there appreciable differences between actor systems and FBP?

samuell11• 6 years ago

Actor systems differ in their means of communication. While actor systems use the actors as the main units that you connect, with fire and forget messaging to each actor's inbox, FBP proposes communication between named ports over channels with bounded buffers, very much like the CSP implementation in Go (just that you have long running components as in actor systems - something that can easily be implemented in Go with structs with go channels in fields for the ports, as shown in the FlowBase library I mentioned in another comment).

The FBP model provides implicit backpressure, because of the bounded buffers on the channels. The actor model on the other hand is more loosely coupled and flexible.

This actually means FBP systems are slightly more optimal for efficient within-one-node, in-memory parallellisation, whereas actor systems shine more for distributed systems.

They shine on different scales, in other words.

I wrote a post many years ago on this, that seems to have aligned with the experience of a number of people, based on the reactions in the comments: https://rillabs.com/posts/flowbased-vs-erlang-message-passin...

megameter2• 6 years ago

FBP is most analogous to the unit record machines of yore:

https://en.m.wikipedia.org/wiki/Unit_record_equipment

What you are programming is the graph of how these machines connect and flow data(keypunch cards) through each other, while the machines themselves and their function are abstracted out. Data does not stay "at rest", it's presumed to reach a terminating point where it exits the graph. A buildup of unprocessed data in a machine's inbox results in an overflow.

It's a very useful model for making a debuggable asynchronous system since it imposes static constraints everywhere that you can map to your real hardware resources, versus the emphasis on dynamism seen in the actor model(actors hold private data, modify their state, create new actors - all explicitly disallowed in FBP).

dustingetz5• 6 years ago

FP decouples the AST (symbolic functions with input ports and an output port) from the evaluation context, which might be async, or sync, backpressured, stateful, exceptional, incremental/reactive ... mix-n-match whatever behaviors you want, all for the same abstract AST

samuell11• 6 years ago

I actually didn't want to make this too much of an FP-bashing comment, as I find great use of many FP concepts, daily.

I have yet to see an FP concept for composition though, that is as simple and generic in its implementation, as the FBP principles.

I have sometimes thought FBP networks provide roughly the same function (pun not intended) as a monad, although I never seem to fully grasp what a monad is, so I can't tell for sure :o)

dmbarbour2• 6 years ago

I think most FP languages don't decouple AST from evaluation context. E.g. I cannot perform abstract interpretation on a function's AST, nor rewrite a function's AST for incremental computation, nor perform explicit partial evaluation during function composition by composing the ASTs. I only can access opaque `Input -> Output`.

Also, FP algorithms developed for one evaluation context cannot easily be ported to another. They are implicitly entangled with assumptions. For example, I cannot evaluate a lazy algorithm in an eager evaluation context without paying a huge price. Adding back-pressure for list processing where a program was not designed for it would easily result in deadlock. Adding exceptions without adding unwind (or bracket, try/finally, etc.) to the existing program expressions will easily result in buggy code.

I think your assertion might be true for some specific integrations of FP nodes into FBP. Is this what you mean?

codetrotter• 6 years ago

Your flowbase.org domain redirects to a different domain. I thought that perhaps you had a typo in the URL or forgot to renew your registration of the domain, but upon a closer look I think it’s probably just a misconfiguration in your webserver config causing it to redirect to another one of your projects.

My guess is that you forgot to put the definition of the domain sans www in your webserver config.

If my guess is correct then this link should work:

https://www.flowbase.org

Edit: doesn’t work either. Perhaps you don’t have a TLS cert for that domain? In which case maybe this link will work instead:

http://www.flowbase.org

Edit 2: Without https it works. And like the person responding to this comment said it redirects to the GH repo.

wizzwizz4• 6 years ago

The link's trying to point you here: https://github.com/flowbase/flowbase

samuell11• 6 years ago

Ouch, sorry, and thanks! Wrote on mobile on the go and didn't manage to double check the link.

FigmentEngine• 6 years ago

Sounds like XProc 3.0 https://xproc.org/

megameter2• 6 years ago

Having studied and implemented FBP systems in the past, one major takeaway I've gleaned is that most automation problems start off as a linear sequence of processes, and so the branching-graph of FBP looks unwieldy and superfluous. But this is deceptive; you probably don't want to have a huge number of branches in the design, but you will want them as an optimization step or a way to combine inputs.

So it's useful to design with FBP in mind but with a linear interface as the entry point.

Another aspect of this is that FBP graphs are static but you may have a need to reconfigure them frequently; that is, you may want to have a graph compilation step drawn from a source language, rather than manually wiring it up.

A way of making that graph compilation more than a syntax is to include a formal constraint solver: Excel, for example, flows the data after determining a solution for how cells relate to each other. The power of the spreadsheet paradigm really lies in these combinations of concepts.

Lastly, there isn't really magic in the algorithmic/implementation aspects of FBP. It grew out of 1960's mainframe types of problems, and so it can be implemented in a low level way with static pools of memory and pieces of assembly code. But it remains conceptually just as relevant to today's massive distributed systems.

goliatone• 6 years ago

The flowbase.org link is redirecting me to a polish website, not sure what’s going on there

samuell11• 6 years ago

Sorry about this, I was typing in a hurry. The links should redirect to https://github.com/flowbase/flowbase but seems to be misconfigured for https. Looking into it.

vitalhead• 6 years ago

Visual aspect of FBP is also brilliant and helps to navigate/comprehend inherently parallel execution flow. The same complexity network expressed in linear text would require more effort to grasp. And it allows you to leverage your brain's visual capacity which is quite powerful - humans are visual creatures (https://www.seyens.com/humans-are-visual-creatures). For example in Excel the update dependency network is completely hidden. The maintenance of complex Excel models is really hard.

For same reasons we start looking at FBP and functional reactive programing to simplify the design and maintenance of complex interactive UIs. End up implementing Kelp (https://kelp.app) with visual FBP editor and reactive framework (https://kefirjs.github.io/kefir/).

akavel• 6 years ago

Mandatory fanboi mention: https://luna-lang.org/ - now Enso: https://medium.com/@enso_org/enso-dev-blog-18th-december-202...

samuell11• 6 years ago

I'm super intrigued by luna/enso. Only that every time I've tried it, the editor has had serious stability problems. I wonder if a more traditional tooling and editor support wouldn't be more successful for wide adoption?

pantsforbirds2• 6 years ago

Luna is one of the coolest projects I've seen. I've really enjoyed watching the progress.

Curious to see if it (or concepts from it) gets picked up for programming education one day.

fahrrad34• 6 years ago

Do I understand it correct when I think that clojure’s core.async is an example implementation?

dustingetz5• 6 years ago

not quite, but this seems close: https://github.com/leonoel/missionary

adamkl• 6 years ago

Rich Hickey gave a good talk on core.async and it does seem pretty similar in concept:

https://youtu.be/drmNlZVkUeE

teknopurge• 6 years ago

https://nodered.org/ - great project for all sorts of needs.

jcims• 6 years ago

I recently used nodered for some process automation and it was absolutely fantastic for quickly prototyping, dashboarding, doing sensor integration, etc. Highly recommended!!!

jgraettinger1• 6 years ago

We're building a tool, Estuary Flow, which seeks to be an end-to-end realization of practical, configuration driven, and scale-out flow-based programming -- with an important twist.

The central concept is a "collection", an append-only set of schematized documents, which can be captured and materialized into other systems (e.x. pub/sub, S3 buckets, etc). "Derivations" are collections defined in terms of source collections, and stateful transformations/joins/aggregations which are applied to them.

A key twist is that collections are simultaneously a batch dataset (backed by cloud-storage) and also a real-time stream. They unify the current dichotomy of "historical" vs "streaming" data into a single addressed concept. Declare a new derivation, and it automatically back-fills over history right from S3, then seamlessly transitions to live data.

If this sounds interesting, check out our docs [0]. We're early, but love feedback!

[0] https://estuary.readthedocs.io/en/latest/README.html

ledauphin2• 6 years ago

I don't mean to derail the conversation, but this really does remind me of the game Factorio, though sort of in reverse.

In Factorio, you build a larger and larger factory out of pre-established functional components (assemblers, labs, chemical plants, etc) that take in a limited set of inputs and produce (usually) a single output. Your challenge is not to define the functional core processes, but instead to wire together those functional components by connecting their inputs and outputs in ever-more-automated fashion, starting by hand, then using simple belts (pipes) that eventually allow arbitrary load-balancing via "splitters", and eventually through to trains (the forking and load balancing happening via backpressure in the train system) and robots (where everything is managed essentially as a single state database of requests, and backpressure is provided by output limitations, usually per functional component).

Naively, I think that someday a decent chunk of programming might actually look like this, and parts even be represented visually (though in my opinion likely still defined formally as text). Only I think programmers will continue to write the functional components themselves, unlike in Factorio. They'll just live on different levels of the "codebase", and the "pipes" level will likely be a lot more abstracted than it is in Factorio.

As a software developer, I find this paradigm to map very well to serverless architectures, because you generally want to think a level higher than the per-machine basis. It does require a willingness to forgo handy and well-established tools like the filesystem and Unix pipes in favor of higher level abstractions around transfer and storage of data.

freeqaz2• 6 years ago

You have, from first principles, reconstructed a huge portion of my thought process for building https://refinery.io

Factorio and Minecraft automation mods are a big inspiration! Check out InfiniFactory too :)

Bridging existing applications to the Serverless paradigm is far from simple. That's one of the biggest struggles I've experienced trying to build a Flow-based software platform.

Learn more every day though. Thank you for the interesting comment!

mikewarot2• 6 years ago

I took a look at your site (refinery.io), and the "watch demo" is really a "read introduction"... I was expecting to sit back and let you show me 10-15 minutes of video that sells the idea.

It looks very professionally done, but reading screens isn't as easy as it used to be for me, a video is better.

Good luck!

dgb23• 6 years ago

Another video game that works similarly is Oxygen not Included.

One of the most compelling arguments for a data-flow/flow-based programming is the mental model and the visualization aspect of it. This opens up opportunities for monitoring, visual, data-driven programming and it is white board friendly.

In Elements of Clojure[0], the author discusses the concept of "principled components and adaptive systems". And a flow based design reminds me of exactly that. The semantics of composition and communication are well-defined and universal, but internally the components can (should) be specific and concrete.

Similar can be said about Small Talk as well. A primary aspect of its design was the mental model, understanding and learning. The core idea was that learners (especially children) understand things in terms of their operational semantics.

I don't mean to derail the conversation, but this really does remind me of the game Factorio, though sort of in reverse.

So no, I don't think this is a derailment, but likely one of the most important aspects of paradigms like this.

[0]https://elementsofclojure.com/

ledauphin2• 6 years ago

thank you for making this connection - I had recently been reading Elements of Clojure and noticed the concept of principled components, and I had been slowly working my way towards this realization/analogy myself.

analog31• 6 years ago

It might not tick all of the boxes for being a complete software development tool, but Excel strikes me as being a dataflow programming model. I wonder if it's a reason why it's easy for laypeople to learn.

Geminidog• 6 years ago

This type of programming is actually a subset of functional programming called point free programming. It's equivalent to programming using only combinators as the fundamental unit.

https://en.wikipedia.org/wiki/Tacit_programming

All programs are actually pipelines of data flowing from a low entropy state to high entropy state with IO and state as the endpoints of the pipes-. Using the point free style or flow based programming makes this entropy and the pipelined nature of all programs more explicit.

Using OOP or regular programming the pipeline nature of data flowing to and from state and IO becomes less evident and more convoluted.

mikewarot2• 6 years ago

Side effects are forbidden by structure, flows could be monitored in a GUI/Debugger, and as a result components can be tested as a unit, instead of a whole system. I love it!

It is easier to design digital circuits when you have a whole catalog of 7400 and 4000 series gates, than it is using individual transistors. It is easier to wire a house when you're not making wires and switches with a hammer and a forge.

I welcome this new higher level of abstraction, and am willing to pay the cost in terms of CPU and Memory to get there, just as I'm willing to waste transistors or copper wire to have something done and working.

dang• 6 years ago

If curious see also

from 2015, on the same article: https://news.ycombinator.com/item?id=8867584

2019 (a bit) https://news.ycombinator.com/item?id=20215592

2019 (another bit) https://news.ycombinator.com/item?id=19203642

2019 https://news.ycombinator.com/item?id=18859019

2015 (not that good) https://news.ycombinator.com/item?id=10755250

2015 (a bit better) https://news.ycombinator.com/item?id=9718868

2015 https://news.ycombinator.com/item?id=8992281

2013 https://news.ycombinator.com/item?id=6264657

2012 https://news.ycombinator.com/item?id=4239274

homieg33• 6 years ago

Very useful list. Thanks for providing.

pantsforbirds2• 6 years ago

You can write pretty interesting data pipelines using akka streams with a very similar idea to FBP. It's probably not technically FBP, but the whole reactive stream implementation is similar in thought but allows for things like disparate data source speeds without blowing out part of the flow.

hexo• 6 years ago

This reminds me of reactive programming, especially functional reactive programming.

u678u• 6 years ago

Its great but surely its more like OO programming perhaps without inheritance.

galaxyLogic7• 6 years ago

In Dataflow -programming (or something like that) how do you program a decision that depends on the result of some component?

It would seem like I need to suspend my computation then "ask" the result from some component, get the result back, and then alter my computation based on that result.

Pure flow-forward would not seem to support this easily. Or can it? Or does it come down to that we always will need BOTH sync- and async- functions? (unless of course we limit the problem domain)

erichocean2• 6 years ago

Spreadsheets are a variant of Dataflow programming, so however they do it.

galaxyLogic7• 6 years ago

Right but spreadsheets are hardly used for "general purpose programming". They are good for the tasks they are used for but "systems" are not implemented with spreadsheets.

dustingetz5• 6 years ago

I think you are talking about Self-Adjusting Computation, so the answer is yes dataflow can do that https://blog.janestreet.com/breaking-down-frp/

DonaldFisk7• 6 years ago

Yes, it can.

The simplest way is with when/unless vertices which take a boolean input and some other inputs, and which either output the other inputs if the boolean input is true/false, or output nothing.

galaxyLogic7• 6 years ago

But can the boolean input come from some other component to which I send data?

jancsika3• 6 years ago

This gets unwieldy quickly. Consider the ajax pattern:

  b = a
    .filter1()
    .filter2()
    .filter3()
    .filter4();
  if (b.foo()) {
      return a.filter5();
  } else {
      return a.filter6();
  }

For visual programming, AFAICT you have to feed `a` downstream twice: once to filter5 and once filter6 in preparation for the conditional branch. That requires one of the following:

* intersecting lines, which are way more visually complicated to read than the code above (and can quickly create visual ambiguities which as a class do not exist in a text-based language)

* one line which loops under the bottom of one of the downstream objects, which is visually distracting (especially if you have more than two branches)

edit: clarification

bmitc3• 6 years ago

For dataflow programming, you introduce structures. For example, in LabVIEW, one has many types of structures such as for loops, while loops, case structures (for making decisions), event structures (for listening to and responding to user events), type specialization structures (for reacting, at development/compile time, to types in a dynamic way as a type of development/compile time polymorphism), conditional compile structures (for conditionally compiling different parts of code), and disable structures (for disabling certain parts of code).

Here's a simple case structure for a decision based upon a boolean value: https://imgur.com/a/YyzXVhU

This is really no different than the type of dataflow you have in other functional languages like F# or Racket. Values flow into special syntax forms such as if or match that allow branching.

voldacar• 6 years ago

This is a good article, in the history or multithreading sections it would have been nice to mention the various old hardware implementations of this model, such as the transputer or connection machine

smartmic• 6 years ago

A nice and powerful C++ flow-based programming framework is DSPatch [1]. It is definitely worth to mention here, check out also the examples in the audio domain.

[1] http://flowbasedprogramming.com/docs/html/index.html

blackrock• 6 years ago

Fascinating. This is how I build my programs. I never knew it had a particular name for the style. I just assumed it was functional-style programming.

This allows me to build ever greater programs, that does very complicated things, and has thousands of lines of code and logic, but everything gets boiled down to small bite-sized pieces.

It’s a more data oriented approach. And it seems to follow an assembly line model.

I liken it to building code pyramids. Where I keep stacking and chaining one function to another, to build even bigger pyramids.

At the end, all the code is heavily unit tested, and I have a high degree of trust in the fidelity of the codebase.

0x445442• 6 years ago

Yeah, when Java 8 introduced the Function interface I started to program this way as well. I get push back some times from the those not initiated to the functional style because I have a ton of little function that do one thing. Unfortunately in Java this means a bunch of class files.

The other thing I noticed when I started using this style is that I was able to ditch unit testing mocking frameworks. This made me realize, when the Function interface was introduce it was a backdoor way to get folks back to coding to an interface, which we should have been doing all along.

jes5199• 6 years ago

neat to see this here! I've recently become convinced Flow Based Programming is the ideal that other programming paradigms are reaching for. But in practice, there's a lot that's not obvious to me how it would actually work-- so it's great to have a bunch of new reading suggestions!

golemotron• 6 years ago

Thank you, Doug McIlroy.

tyingq• 6 years ago

The analogy to the soft drink bottling seems...off. I would imagine there's a contract of sorts between stations on rate of output/input, spacing/placement of bottles, etc, between stations that makes it not very asynchronous.

cs404• 6 years ago

What is the difference from flow and event driven programming

Ericson2314• 6 years ago

This is wildly under developed, mathematically, but eventually with enough FP and category theory and things, we will get back there.

The problem is everyone wants to write little nodes and "just" wire them up. But all the real complexity is not in the nodes, but the nature of the wires and their composition.

We'll get there though, stay tuned.

ngneer• 6 years ago

Sounds like TPL Dataflow

pgt• 6 years ago

Honourable mention for Petri Nets: https://en.wikipedia.org/wiki/Petri_net

mujina93• 6 years ago

Have a look at the Oz PL and its canonical textbook: https://en.m.wikipedia.org/wiki/Oz_(programming_language)

wyqydsyq• 6 years ago

This paradigm reminds me a lot of my favourite JS/TS framework at the moment: Cycle.JS https://cycle.js.org/

jpaulmorrison2• 6 years ago

General comment about the Wikipedia article - https://en.wikipedia.org/wiki/Flow-based_programming : given all the interesting discussion on the 'Net, including this thread, it might be time to update the Wikipedia article.

The most substantive addition in recent years is Akaigoro's comment about Actors, added Jan. 2020 (@guitarvydas, care to jump in?!), preceeded, I think, by Joe Witt's reference to NiFi in mid-2018. This general lack of activity I feel might lead readers to assume that FBP is an outdated concept, whereas this thread proves it's definitely alive and kicking! I, personally, am not allowed to update the article, due to the WP ban on self-promotion, so I would like to encourage people to add topics, controversies, anything, to the WP article... Freshen it up a bit, as it were! Thanks, and stay safe, everyone!

Flow-Based Programming

Related Stories