Practical Railway-Oriented Pipelines in Ruby

https://ismaelcelis.com/posts/practical-railway-oriented-pipelines-in-ruby/

by ismaelct • 2 years ago

67 66 2 years ago

Draiken3• 2 years ago

If you prefer inheritance over composition

That's not what this boils down to.

These are procedures. OOP is generally about objects sending messages to each other. So this solution is all about executing procedural calls, which has nothing to do with inheritance/composition.

You can very easily have OO code with composition without having to rely on everything having one single interface "call/run/execute".

ismaelct22• 2 years ago

Perhaps I phrased this badly. I don't think the entire article boils down to inheritance vs composition. But in discussing these patterns elsewhere, some of the pushback has been that many Ruby devs prefer to decompose problems via sub-classing instead of composition of command objects, so I tried to cater to that objection with that line.

Pipeline steps are command objects (a pretty standard OO pattern), so they have a single entry point / public method. But they can still fully leverage any other OO pattern in their implementation. The more complex ones I use may instantiate other objects, pass messages between them, etc.

But the single-method #call API is what makes composition easy. See Rack, or any number of middleware-style designs, for other common uses of this in Ruby.

Draiken3• 2 years ago

Yeah, my main gripe is with the phrasing. It gives the impression that if you like composition over inheritance (which people mostly take as true without even thinking about it), you should use this.

I feel like that's the kind of over-simplification that then makes people pick a style like this without fully considering the implications.

Every single place that I've seen use these "command pattern" systems devolved into a complete utter mess of procedures. People forget the basics of OO and write everything into "steps" because that's now the hammer and everything is a nail.

If they stuck to the outer layers as the orchestrator for everything, that'd be great. But that never happens in my experience.

corytheboyd7• 2 years ago

I’m glad people find value in patterns like these, but I’m so sick of them. Plain Ruby code is so much easier to debug, which is all that matters in the long run. The senior engineer who introduced the thing like this inevitably leaves and nobody cares to learn some bespoke abstraction enough to keep using it.

I’m sure this is great and solves real problems for OP and friends. I just ask that you think twice about it before forcing it on coworkers. Please, this poor soul can’t take it anymore.

ismaelct22• 2 years ago

Thanks for your feedback. Out of curiosity, when you say "plain Ruby code" what do you mean, exactly? Presumably you're still making use of _some_ patterns that you think are Ok.

graypegg2• 2 years ago

If I had to guess, “plain Ruby” is ruby that fails with a sensible stack trace without internal things showing up. That’s always been my trouble with complex abstractions, it can make it very difficult to use with a debugger especially, which is how I always work with Ruby. New flow control models like this are the main culprits to weird debugger issues.

I haven’t gotten a chance to mess around with this specifically, how much of its internals are mixed into execution?

codesnik4• 2 years ago

The senior engineer who introduced the thing like this ...

I actually consider the urge to introduce overengineered patterns like this a sure sign of a "mid-level developer". Senior knows better.

ismaelct22• 2 years ago

I would have thought that "senior" means assessing things in context instead of falling back to truisms. "over engineering" only makes sense relative to a concrete problem you want to solve.

corytheboyd7• 2 years ago

I’d say mid-levels will follow along, but newer seniors will confidently throw ideas out for all to observe how gloriously smart and perfect they are.

ativzzz• 2 years ago

Senior knows better.

Sometimes, the cost of keeping really talented senior engineers who have solved a very difficult part of your tech stack is to give them the freedom to over engineer solutions that are probably a bit overcomplicated (maybe like OP). The alternative is they get bored and leave and take a wealth of technical knowledge with them

jtms2• 2 years ago

I'd like to think this is kind of just the cost of discovering that 1 pattern out of many thousands that is actually useful. I am going to guess the Gang of Four didn't just sit down and bang out the entire catalog of design patterns in one extremely productive programming session. They likely battle tested hundreds of different patterns, most of them being thrown out as not useful or a bad abstraction before arriving at the set they published in the book.

Draiken3• 2 years ago

I wrote a whole blog post on this because of the amount of harm caused by these types of systems. Interactors, interactions, pipelines or whatever other name, can very easily ruin a codebase in my experience. Already happened multiple times in my career.

Developers don't like to think about abstractions and naming is a known hard problem. People pick this up and now everything is easy. Every name becomes: DoThis, DoThat, DoThisOtherThing and only answers to `call`. No need to think!

Now you have deeply nested procedures that oversimplify their interfaces to match the library and suddenly you no longer have meaningful error types, you have strings. You no longer have abstractions, you only have procedural code with some objects in the middle. The list goes on.

Like most things in development, used in the right places and with careful consideration, this can be very valuable. But this is a very insidious change that turns your code into a functional style with several drawbacks that are rarely considered.

ismaelct22• 2 years ago

I agree with the drawbacks listed here. I would add that "careful consideration" of any pattern we use is the job description.

I would further add that we should extend that thoughtfulness to the opinionated frameworks many of us rely on. They usually come with hundreds of complicated patterns baked in, and we "oversimplify our interfaces" to match the framework as a matter of course. All I'm saying is that committing to any mental model provided by a design pattern or framework has similar drawbacks. In this case, the mental model is "a big operation is a list of smaller operations in a chain". Use with caution.

No need to think!

Not sure I agree with fully. If anything, I've found that I need to think harder about what constitutes a step in a workflow, what are the names of each stage, what concept they encapsulate. I can't just chuck everything into a god object or a deeply nested hierarchy tree somewhere.

But again I agree. Using this pattern I've definitely over-complicated or gone down the wrong path at times. I would say though that I found it easier to roll back and change direction when compared deeply nested object graphs, for example.

lamontcg5• 2 years ago

Developers don't like to think about abstractions and naming is a known hard problem. People pick this up and now everything is easy. Every name becomes: DoThis, DoThat, DoThisOtherThing and only answers to `call`. No need to think!

This 'framework' doesn't require that though:

    def double_number(r) = r.continue(r.value * 2)
    def add_one(r) = r.continue(r.value + 1)
    def square_number(r) = r.continue(r.value ** 2)

    pipeline = Pipeline.new do |pl|
      pl.step method(:double_number)
      pl.step method(:add_one)
      pl.step method(:square_number)
    end

Just methods, with no need to implement call() because that is what a method/proc/lambda in ruby implements. In other languages like C# you could have an API that can take a Func<Result, Result> so that you can just pass delgates/lambdas/methods to it. And adding a bit more DSL could make the construction of the pipeline even less verbose.

You can use this without going full-Java and creating a dozen (or a hundred) 5-line classes in different files.

allknowingfrog3• 2 years ago

Have you ever noticed that everyone who drives faster than you is a maniac, and everyone who drives slower than you is a moron? I think code abstraction fits a similar pattern.

corytheboyd7• 2 years ago

Sortaaaa, it’s definitely harder to read existing code than it is to write new code, which is what you’re getting at. To bend your analogy a little further, we can all identify the person on the road who was clearly wrong, putting us all in danger. It’s the difference between someone slowing down traffic by camping in the fast late and someone else barreling down a residential street 50 mph over the limit. The former is annoying, the latter struck and killed someone.

Not that code is this serious at all. Idk.

ismaelct22• 2 years ago

I'm so going to steal this

whstl4• 2 years ago

I wonder if I'm witnessing the birth of yet another Hacker News thought terminating cliche.

There is a place and time for everything. You can drive as fast as you can in a Formula 1 race, but when you're around pedestrians you gotta slow down.

Similarly, complex abstractions can often be fast to write but difficult to maintain. Similar to a lack of them.

But sure, it all has a time and place. Except the attitude of dismissing or never discussing this topic. That's the only thing that is "wrong" here.

matheusmoreira• 2 years ago

If they drive faster than me, then godspeed. Let them safely pass me and then swiftly disappear into the horizon, hopefully not before I manage to learn a couple badass driver tricks from them.

If they drive slower then yeah they're morons and I need to pass them as soon as humanly possible.

In both cases they become non-factors and I can drive in peace. The fast disappear into the horizon and the slow get left behind. This plays out until I am alone in the road and no longer need to consider the actions of other humans behind the wheel. That is when peace is achieved. Sadly it does not last long.

jtms2• 2 years ago

I might be a bit biased (I wrote and maintain a similar gem), but I think this is a great pattern that solves a real problem that is encountered and does so in a very Ruby kinda way. I particularly like the mental model of stringing together multiple service objects into a "pipeline" and the semantics chosen for the API. Kudos to OP for putting this out there!

ismaelct22• 2 years ago

Thanks. Where can I find your library? I'd love to compare notes!

nathanappere• 2 years ago

I use a functional version of this that can chain any "callables" + support extensions on the calling interface https://docs.rubykit.org/kit-organizer/edge/Kit.Organizer.Se...

yakshaving_jgt3• 2 years ago

I'm still bitter about having to work on a team with some Clojurists who thought Haskell and monads were bad and stupid, but then forced ROP on the rest of us in a Clojure project.

ismaelct22• 2 years ago

Odd to see ROP and monads as an either/or problem (see what I did there?). Most ROP implementations I've seen rely on the result monad.

yakshaving_jgt3• 2 years ago

Exactly my point.

tome• 2 years ago

see what I did there?

Maybe

throwaway918274• 2 years ago

We use the `interactor` gem at $DAY_JOB and it does 90% of the stuff described here, and the last 10% came naturally and we just kinda engineered it ourselves intuitively.

definitely works better than "fat models" that do everything and ruby on rails callback hell

mitchellh• 2 years ago

This pattern is effectively how Vagrant (for anyone who remembers that) always worked, also in Ruby! I even gave a talk on it at MountainWest RubyConf back somewhere around 2013, although I compared it moreso to a "middleware" pattern. Even the API/DSL is almost identical.

The middleware pattern had a lot of the same concepts present in this post: we called context "state" and you could use special exceptions to halt or pause a middleware chain in the middle.

This was a really great way for over a decade (to this day!) to represent a long-running series of steps that individually may branch, fail, accumulate values, etc. I don't recall the exact count, but an old `vagrant up` used to execute something like 40 "actions" (as we called each step).

I'm not trying to disregard this blog post in any way, I'm only pointing out this pattern is indeed very useful and has been proven in production environments for a very long time!

ismaelct22• 2 years ago

Thanks Mitchell (big fan, btw!).

Indeed!https://github.com/hashicorp/vagrant/blob/main/lib/vagrant/a...

Yes, it's not a new pattern by any means, and there's many ways to "halt" the pipeline as you say. For example ActiveRecord stops callback chains if any callback throws ":halt".

Other examples are Redis.rb's pipelining API https://github.com/redis/redis-rb?tab=readme-ov-file#pipelin...

Or more generally any builder-style pattern that composes a set of operations for later execution, including again ActiveRecord's query chaining.

In my article I tried to show a specific implementation using the Railway pattern (where the result must only respond to "#continue?() => Boolean")

lamontcg5• 2 years ago

the code in the blog post has an error that you fixed in the gist:

   def step(callable = nil, &block)

you're missing the nil default in the blog post so it ArgumentError's when passed only a block

ismaelct22• 2 years ago

Well spotted. Fixed. Thank you!

Fire-Dragon-DoL2• 2 years ago

I wrote something like this in the past, it doesn't make sense in ruby.

The closest ruby way is:

    result = DoStuff
    return something if result.nil? # error, early exit

    result = OtherStuff(result)
    return something if result.nil? # error, early exit

No additional complexity. Otherwise usually it ends up just rewriting "programming".

With pattern matching there are also some alternatives now.

ismaelct22• 2 years ago

What do you think of Ruby's built-in function composition? https://ruby-doc.org/3.2.2/Proc.html#method-i-3E-3E

Fire-Dragon-DoL2• 2 years ago

Interesting, never used it. Usually I have methods but not proc, by the time I have to write `.method(something)` a bunch of times I'd it doesn't make much sense anymore.

I love curried functions by default, but if it's not the default, it never works out even in functional languages like Elixir. You need to have every developer (and new hired developer) pretty much on board with that and exclude every library that doesn't do this, or wrap it (huge burden).

Given those, I'd rather stick to what I wrote. Not my favorite, I loved my brief experience with Elm

Practical Railway-Oriented Pipelines in Ruby

Related Stories