Rama on Clojure's terms, and the magic of continuation-passing style

https://blog.redplanetlabs.com/2024/10/10/rama-on-clojures-terms-and-the-magic-of-continuation-passing-style/

by nathanmarz • 2 years ago

106 73 2 years ago

blog.redplanetlabs.com

fire_lake• 2 years ago

What is the best introductory post to Rama right now?

I would like to skip the marketing and understand how it compares to a wiring together Kafka, Spark, MySQL, etc.

bbor2• 2 years ago

AFAICT it's not actually published yet? It's a bit of a confusing situation because you can "download Rama" but it's not "full", whatever that means. See https://redplanetlabs.com/learn-rama

E: this download has "the full Rama API for use in simulated clusters within a single process", but also "Rama is currently available in a private beta." That's a highly unusual way to release what appears to be a Java library at the end of the day, but hopefully that's because it's unusually awesome! Looking forward to actual info some time in the future. I wonder if the "private beta" costs money...

nathanmarz13• 2 years ago

What we've released publicly on our public Maven repository is a different build of Rama which can only be used for testing/experimentation in a single process. So it can't be used to run full clusters. It's API-equivalent to the full Rama build.

This will change when we move out of private beta, when Rama will be free to use for production for small-scale applications.

nathanmarz13• 2 years ago

The examples in this repository are representative of what it's like to use Rama for different kinds of use cases: https://github.com/redplanetlabs/rama-demo-gallery

beders• 2 years ago

A Rama operation does not return a value to its caller. It emits values to its continuation. This is a critical distinction, as part of what makes Rama operations more general than functions is how they can emit multiple times, not emit at all, or emit asynchronously.

thom3• 2 years ago

Which is obviously very similar to how transducers already work in Clojure, but they still lack some of concurrency options of reducers. Getting all this on a smart, distributed runtime seems very promising.

mschaef4• 2 years ago

Getting all this on a smart, distributed runtime seems very promising.

Hopefully it is.

This CPS article is the first of the Rama blog posts where it seemed like there might be something there. The earlier posts - "I built Twitter scale Twitter in 10Kloc" - were never really all that convincing. The thing they claimed to have built was too ambitious a claim.

dig1• 2 years ago

There is a library called tesser [1] (by Jepsen/Riemann author) that behaves like parallel transducers with more Clojure "native" syntax. With transducers, you have to use "comp," and with tesser, you use "->>" as you'd use with lazy functions.

Sadly, tesser is not advertised as it should; I find it much more flexible than transducers. E.g. you could parallelize tesser code over Spark/Hadoop cluster.

[1] https://github.com/aphyr/tesser

moomin3• 2 years ago

I feel like CPS is one of those tar pits smart developers fall into. It’s just a fundamentally unfriendly API, like mutexes. We saw this with node as well: eventually the language designers just sighed and added promises.

You’re better off with an asynchronous result stream, which is equivalent in power but much easier to reason about. C#’s got IAsyncEnumerable, I know that Rust is working on designing something similar. Even then, it can be hard to analyse the behaviour of multiple levels of asynchronous streams and passing pieces of information from the top level to the bottom level like a tag is a pain in the neck.

greener_grass• 2 years ago

CPS might be the underlying structure, but that doesn't mean that CPS is the interface.

oersted• 2 years ago

I agree. I'm sure that CPS has much more robust theoretical roots and that it's more general and powerful, but in practice it doesn't often look much different from classic callback-hell.

Generally, I prefer the coroutine/generator style, it is more explicit and straightforward syntax-wise. More importantly, it decouples operation execution from chaining. A function that emits multiple values in sync/async shouldn't be responsible for running the next function in the pipeline directly. It's better when the user of the interface has direct control over what function is run over which values and when, particularly for parallelizing pipelines.

I do understand that Rama builds such a syntax on top of CPS, and a compiler that implements generators has a similar execution model (perhaps an explicit state-machine rather than leveraging the function stack to do the same thing implicitly).

pyrale3• 2 years ago

That's because CPS is the callback hell.

Promises are a mechanism that was devised to separate the composition mechanism and the function itself, much like shell pipes exist to separate the control flow from the called function.

In this article, they implement a pipe-like mechanism, that avoids having to do "traditional" CPS. That is why they say the continuation is implicit. That being said, that mechanism goes further than that, and looks very much like Haskell's do-notation which enables programmers to use functional languages in an imperative style without knowing too much of the underlying implementation.

gpderetta2• 2 years ago

As far as I know, CPS is supposed to be an intermediate form for compilers and a theoretical framework to understand control flow and concurrency constructs. The second part is I think a very good reason why most programmers should learn about CPS; but I don't think CPS was ever meant to be something you were supposed to routinely program into, in the same way you don't routinely program in assembler.

mschaef4• 2 years ago

I feel like CPS is one of those tar pits smart developers fall into. ... eventually the language designers just sighed and added promises.

Bear with me, but raising kids taught me a lot about this kind of things.

Even at two or three years old, I could say things to my children that relied on them understanding sequence, selection, and iteration - the fundamentals of imperative programming. This early understanding of these basic concepts why you can teach simple imperative programming to children in grade school.

This puts the more advanced techniques (CPS, FP, etc.) at a disadvantage. For a programmer graduating college and entering the workforce, they've had life time of understanding and working with sequencing, etc. and comparatively very little exposure to the more advanced techniques.

This is not to say it's not possible to learn and become skillful with these techniques, just that it's later in life, slower to arrive, and for many, mastery doesn't get there at all.

pyrale3• 2 years ago

I feel like these explanations based on cognitive development always end up with unprovable assertions which inevitably support their author's views. The same exist about natural language, and they're always (unconvincingly) used to rationalize why language A is better than language B.

In my experience, when you ask people to tell you what "basic" operations they do for e.g. multi-digit number additions or multiplications, you get many different answers, and it is not obvious that one is better than another. I don't see why it would be different for languages, and any attempt to prove something would have a high bar to pass.

moomin3• 2 years ago

I take your point about mastery. Especially FP, where it's very clear that mastery of it is extremely powerful. On the other hand, there are some like our regular synchronization primitives where not even mastery will save you. Even experienced developers will make mistakes and find them harder to deal with than other higher-level abstractions. Where CPS fits on this curve, I don't know. I feel pretty confident about where FP and Mutexes sit. But I have yet to see something where I feel I'd rather use CPS than an async stream result.

packetlost• 2 years ago

CPS is one of the most obnoxious ways to write code.

Unless you're Gleam, in which case it feels natural and looks pleasant.

andrewflnr• 2 years ago

Yeah, CPS is best used as an implementation technique, which is how it's used here. I think they even use it to build a stream-like API for their "operations".

neonsunset2• 2 years ago

I found myself liking the F# way of consuming/producing IAsyncEnumerable's with taskseq. It's very terse and looks nice: https://github.com/fsprojects/FSharp.Control.TaskSeq?tab=rea...

moomin3• 2 years ago

Apparently Microsoft hold a patent on “yield!”, which makes it all the more frustrating that they haven’t included it in C#.

cryptonector8• 2 years ago

IMO TFA makes much of CPS because that's how they chose to implement generators, but the main thing about the language is that it has generators.

Now having generators is nothing new, but I don't want to take too much away from TFA, as there are some interesting things there. I'll limit myself to pointing out that the Icon programming language had generators and pervasive backtracking using CPS in the Icon-to-C compiler, and that other languages with generators and pervasive backtracking have been implemented with CPS as well as with bytecode VMs that don't have any explicit (internally) continuations. Examples include Prolog, Icon, and jq, to name just three, and now of course, Rama.

cryptonector8• 2 years ago

I've actually used hand-coded CPS in an evented program for C10K. Hand-coding CPS is a real pain, but CPS is usually used as an implementation detail, and as such it's not an API. In some cases you can get at an implicit continuation to then use it explicitly (call/cc comes to mind), and then it's an API, sure, but typically one does not have to use it.

thunkingdeep• 2 years ago

I’ve always thought that CPS is a good barometer for finding out whether a developer is talented or whether they THINK they’re talented enough to design and/or implement these kinds of compiler components. This kind of thing but definitely CPS in particular is so much trickier to nail down than it initially seems if you’ve written a compiler before. Up there in difficulty with automatic parallelization and loop transforming. I tried to write a very small POC lisp once with an idea to have all vectors of known sizes get map’d in parallel and I never could nail it down.

Kudos to all involved. Clojure is such a mind bending tool. God only knows what it takes these people to maintain the guts of it all.

kamma4434• 2 years ago

I don’t want to be a party pooper, but from my very cursory look at the page, I don’t think this will go much farther than a very small community. I feel like you’re adding a lot of complexity compared to normal backhand/frontend/websetvices Scenario that everybody already understand.

diggan• 2 years ago

With that mindset, should we just stop trying to improve anything regarding backend/frontend/webservices since "everybody already understand it"?

jgalt212• 2 years ago

You're arguing change (a broadly defined term) is not necessarily bad, but the OP is arguing adding complexity (a type of change) is bad.

kimi3• 2 years ago

I second the OP - I'm not sure where the big prize is. I have a feeling that whomever wrote the article thinks there is a 10x (or 100x) improvement to be made, but I was not able to see it.

I find the syntax very clunky, and I have been programming professional Clojure for at least 10 years. It reminds me of clojure.async - wonderful idea, but if you use the wrong sigil at the wrong place, you are dead in the water. Been there, done that - thanks but no thanks.

OTOH I know who Nathan is, so I'm sure there is a gem hidden somewhere. But the article did not convince me that I should go the Rama way for my next webapp. I doubt the average JS programmer will be convinced. Maybe someone else will find the gem, polish it, and everybody will be using a derivative in 5 years.

nathanmarz13• 2 years ago

Actually, we've eliminated a massive amount of the complexity of backend development. This is most pronounced at large scale, but it's true at small scale as well.

Our Twitter-scale Mastodon example is literally 100x less code than Twitter wrote to build the equivalent (just the consumer product), and it's 40% less code than the official Mastodon implementation (which isn't scalable). We're seeing similar code reduction from private beta users who have rewritten their applications on top of Rama.

Line of code is a flawed metric of course, but when it's reducing by such large amounts that says something. Being able to use the optimal data model for every one of your use cases, use your domain data directly, express fault-tolerant distributed computation with ease, and not have to engineer custom deployment routines has a massive effect on reducing complexity and code.

Here's a post I wrote expanding on the fundamental complexities we eliminate from databases: https://blog.redplanetlabs.com/2024/01/09/everything-wrong-w...

btown• 2 years ago

The original post makes so much more sense in this context! One of the "holy grails" in my mind is making CQRS and dataflow programming as easy to learn and maintain as existing imperative programming languages - and easy to weave into real-time UX.

There are so many backend endpoints in the wild that do a bunch of things in a loop, many of which will require I/O or calls to slow external endpoints, transform the results with arbitrary code, and need to return the result to the original requestor. How do you do that in a minimal number of readable lines? Right now, the easiest answer is to give up on trying to do this in dataflow, define a function in an imperative programming language, maybe have it do some things locally in parallel with green threads (Node.js does this inherently, and Python+gevent makes this quite fluent as well), and by the end of that function you have the context of the original request as well as the results of your queries.

But there's a duality between "request my feed" and "materialize/cache the most complex/common feeds" that's not taken into account here. The fact that the request was made is a thing that should kick off a set of updates to views, not necessarily on the same machine, that can then be re-correlated with the request. And to do that, you need a way of declaring a pipeline and tracking context through that pipeline.

https://materialize.com is a really interesting approach here, letting you describe all of this in SQL as a pipeline of materialized views that update in real time, and compiling that into dataflow. But most programmers don't naturally describe this kind of business logic in SQL.

Rama's CPS assignment syntax is really cool in this context. I do wish we could go beyond "this unlocks an entire paradigm to people who know Clojure" towards "this unlocks an entire paradigm to people who only know Javascript/Python" - but it's a massive step in the right direction!

waffletower• 2 years ago

Could I politely suggest more Clojure-like naming? `deframaop -> defop`. You can always `(require 'com.rpl.rama :as rama)` and invoke with `(rama/defop ...)` for the desired level of clarity and improved readability.

nathanmarz13• 2 years ago

I named it like this so there would be consistency between deframaop and deframafn. Shortening deframafn like you suggest would be "deffn" or "deffunction", which would be very confusing. And I'd rather have deframaop + deframafn than defop + deframafn.

knubie• 2 years ago

Why not rama/defop and rama/defn?

arunix• 2 years ago

Why is it called Rama? (there may be an FAQ, but I couldn't find it)

robertlagrant• 2 years ago

This article is written for people who know Clojure, or at least the examples are. It might be nice to see the examples written in a non-LISP as well.

E.g.

  (?<-
    (ops/explode [1 2 3 4] :> *v)
      (println "Val:" *v))

Is (I believe) the equivalent of Python's

  for element in [1, 2, 3, 4]:
    print(element)

nathanmarz13• 2 years ago

That Python code has the same effect, but the equivalent Python in CPS would be:

  def explode(args, cont):
    for e in args:
       cont(e)
  
  (explode [1, 2, 3, 4], print)

gpderetta2• 2 years ago

That's not in CPS form as cont returns into explode. Even if we treat the for loop in explode as an implementation detail, cont should take the next continuation as a parameter.

cryptonector8• 2 years ago

The whole thing about distinguishing functions that yield once from all other functions is pretty clever. Reminds me of how in Icon a `return` is a C-like return that unwinds the stack while `suspend` is more like calling a continuation with the yielded value. In the old Icon-to-C compiler that's exactly how `return` and `suspend` worked. But Icon does not analyze function ("procedure") bodies to see if they only return, so the `return`-unwinds optimization is a run-time one. Whereas Rama does a compile-time optimization, which makes a lot of sense.

cryptonector8• 2 years ago

Just hours (a day?) earlier there was this very interesting -and IMO relevant here- HN post:

  Can logic programming be liberated
  from predicates and backtracking?
  [pdf] (uni-kiel.de)
  
  https://news.ycombinator.com/item?id=41816545

That deals with backtracking, which is often implemented with continuations, as in TFA.

Rama on Clojure's terms, and the magic of continuation-passing style

Related Stories