Communicating Sequential Processes

http://swannodette.github.io/2013/07/12/communicating-sequential-processes/

by swannodette • 13 years ago

268 75 13 years ago

swannodette.github.io

saljam• 13 years ago

Brilliant! I had always hoped CoffeeScript or somehow a JS library would add blocking channels like this. Writing JavaScript is a pain after being spoilt by Go.

I'm quite curious about what the compiled code looks like. (On a mobile now, I'll have a look when I can.)

jcromartie• 13 years ago

Compiling ClojureScript to JavaScript doesn't yield anything too surprising, and that's great.

But core.async does some amazing stuff to actually analyze the code inside a "go" block and recompile it as a state machine so that a single thread can be "parked" and resumed. This allows core.async to work without multiple threads. It does this all at macro expansion time. This is the sort of thing that really demonstrates the power of Lisp. There's no way to do this without modifying the compiler in a language without a macro system.

So the output of a core.async program is going to be startlingly different than a normal ClojureScript program, because of the way it takes your "go" blocks apart and rewrites them when the macro is expanded.

Here's an example.

https://gist.github.com/jcromartie/5991947

matthavener• 13 years ago

go blocks are transformed into little state machines. Details here http://tomasp.net/blog/async-compilation-internals.aspx via https://twitter.com/timbaldridge/status/354613181886369793

amasad• 13 years ago

It seems like the compiled js code from the page's examples is quite large: csp.js minified ~83kb

I wonder if most of that is the runtime and libraries or the resulting JavaScript code is just that big.

swannodette15• 13 years ago

There is no runtime and note that's 22k gzipped. This is far smaller than jQuery + Underscore.js + persistent data structure library + stream implementation + async control flow library gzipped which is what you're getting. jQuery alone is 33k gzipped.

VMG3• 13 years ago

Very cool - I didn't realize that core.async has made it into ClojureScript. Seems more like a viable language for web programming every day.

Sukotto• 13 years ago

In your opinion, what is missing that it needs to become viable?

VMG3• 13 years ago

The last time I tried it I didn't find a broadly accepted server-side framework that automatically compiles ClojureScript assets, but I haven't looked in a while.

realrocker3• 13 years ago

Why do you call alts! input choices as non deterministic ?

sgrove• 13 years ago

You won't know which channel is going to give you a value ahead of time, it depends on runtime conditions.

But really, you don't care if the user pushes the keyboard or moves the mouse first, you just want to handle each event as it happens in essentially a sequential manner.

realrocker3• 13 years ago

If I am forcing it to be sequential, aren't the choices now determinitic?

brandonbloom3• 13 years ago

Any time you're dealing with multiple control paths (regardless of if they are real threads, logical threads, callbacks, asynchronous messaging, etc), small changes in timing or synchronization will introduce non-determinism. For example, you don't know how quickly a remote service is going to respond.

If you had some code that communicated with channels and used only a single real thread without using any non-deterministic operations, then you'd actually have deterministic choice. So if you write a unit test with some mock services, you would always read from a particular service because the execution is truly deterministic. That might also mean that if two services respond at similar times (even non-deterministically) and you prefer one over another in at a choice, you could wind up unfairly ignoring one of the channels.

To combat this issue, Golang & core.async introduce randomness when multiplexing. If two channels are both ready, one is selected at random.

brucehauman2• 13 years ago

Great post!

And here's a todo's example that I wrote https://github.com/bhauman/async-explore/blob/master/src/tod...

mseepgood4• 13 years ago

"languages like CoffeeScript, Dart, and TypeScript. These languages fail to address the single largest source of incidental complexity for any sizeable client side application - concurrency."

Dart fails to address concurrency? I don't think so: http://www.dartlang.org/docs/dart-up-and-running/contents/ch...

swannodette15• 13 years ago

I'm not sure you actually read the post, isolates do not address the same issues at all - you can't share data between isolates and you can't put isolates onto isolates limiting coordination possibilities - all of these are possible with channels.

mseepgood4• 13 years ago

You can at least send a SendPort over a SendPort:

"Isolates send messages using SendPorts, and receive them using ReceivePorts. The content of a message can be any of the following:

* (...)

* An instance of SendPort"

You can prefer one or the other model, but saying that Dart fails to address concurrency is just not true.

brandonbloom3• 13 years ago

I didn't realize Dart supports such an actor-like primitive. Neat.

That said, I agree with Rich's analysis in the "Wither actors?" section of the core.async announcement:

"They still couple the producer with the consumer. Yes, one can emulate or implement certain kinds of queues with actors (and, notably, people often do), but since any actor mechanism already incorporates a queue, it seems evident that queues are more primitive."

See: http://clojure.com/blog/2013/06/28/clojure-core-async-channe...

It should be noted that actors better fit in a distributed system, which is a step above and beyond concurrent systems.

mraleph4• 13 years ago

dart:async also provides the Stream abstraction that can be compared to Clojure(Script)'s core.async channels.

Libraries like dart:html and dart:io express their asynchronous APIs in terms provided by dart:async so the model is unified across the stack (e.g. HTML events are delivered through streams out of the box).

Here is the first example and third examples from David's post rewritten to Dart

https://gist.github.com/mraleph/5991990

you can see that it is not that much different from ClojureScript versions.

swannodette15• 13 years ago

Thanks for the comparison, the first example is just for fun - I'd be more far curious to see the final Rob Pike example.

alco4• 13 years ago

This looks nice. Less readable than Go (is there GoScript yet?), but still fairly straightforward.

One remark I'd like to make is that I don't see this as having a big advantage over FRP.

The examples with mouse coordinates look like an imperative approach to FRP. With the latter, one wouldn't have to explicitly set up a channel and read from it -- that is an implementation detail in FRP.

To emphasize the contrast, I'd call this channel-based approach an Imperative Communicating Programming (ICP). Your code does not react to changes like it does with FRP. Instead, you as a programmer ask for changes, very often in an infinite loop, and then transform those changes into a side-effect in the browser.

On one hand, channels + goroutines provide a single abstraction with which it is kind of straightforward to implement various concurrently executing threads of code. On the other hand, those infinite loops and their proliferation in the channel-based code will definitely cause some people to cringe and go with FRP instead.

I don't think FRP is as applicable as channels to as many use cases. Channels are more low level and therefore more potent. But for what FRP does, it provides a better (read "more concise", or "declarative") abstraction for it.

swannodette15• 13 years ago

Note that in some ways core.async is a bit more flexible than Go - for example you can dynamically select over a changing list of channels. I've found this to be really handy especially when you're receiving channels on channels.

you can do everything in an FRP style if you like with core.async, it's flexible enough for that. But I think for less toy examples this is undesirable as you allude.

mseepgood4• 13 years ago

Note that in some ways core.async is a bit more flexible than Go - for example you can dynamically select over a changing list of channels

http://golang.org/pkg/reflect/#Select

jzelinskie• 13 years ago

Go has a trick where you can set a channel to nil in order to get its functionality ignored in your select. If you gave the select all the possible channels and turned them on and off rather than adding/removing channels you can obtain similar functionality.

alco4• 13 years ago

One thing that is not covered in the post is how should we clean up those goroutines that are blocked on channels. I'm not familiar with internal workings of core.async, so I'm looking at it from Go's point of view.

Imagine that in the search example, inside the `google` function, we want to read from 100 channels. And each call to `fastest` is supplied with 100 arguments. If on average we get 100 results back in the allotted time, 900 goroutines will be left in a blocked state waiting for someone to read from their respective channel.

I'm guessing those goroutines are cheap and are implemented as some sort of state machine that maps to callbacks. But all those blocked goroutines should still incur runtime cost. Will the GC kick in collect those?

If this issue is really there (i.e. if I'm not mistaken), imagine what will become of your one-page-web-app after you click on that Search button a dozen of times.

swannodette15• 13 years ago

go blocks are cheap to construct and yes they will be GCed. You can trivially debounce the click channel.

alco4• 13 years ago

Thanks for the info! I'm willing to get my hands dirty and see how it's all implemented there.

Just for the record, in Go, if you leave a goroutine blocked on a channel, the channel and the goroutine will remain on the heap unless you read everything from the channel (normal way) or close it (and get a runtime panic).

bascule• 13 years ago

If you're interested in this sort of thing, also take a look at MenTaLguY's WebActors:

https://github.com/mental/webactors

swannodette15• 13 years ago

Yeah I've seen this before though it doesn't really allow you to share arbitrary data as it relies on Web Workers (if you share data with a worker it will disappear from the sending process). I'd also be concerned about the performance of the implementation, core.async adopts C#'s approach which is pretty light weight.

airlocksoftware2• 13 years ago

Cool stuff! A couple of days ago I listened to Rich Hickey talk more about core.async on the most recent ThinkRelevance podcast:

http://thinkrelevance.com/blog/2013/07/10/rich-hickey-and-co...

I've written a few client side projects in ClojureScript, and I haven't been totally happy with the way I end up doing event handling. I'm excited to give this a try.

By the way, I love that this is the sort of thing you can do with just a _library_ in Clojure.

dominotw• 13 years ago

Brilliant!!. Can someone point me to an open source project which uses core.async so I can see it in action.

airlocksoftware2• 13 years ago

It's pretty brand new, so I don't think there are any "real" projects using it yet. But you can check out this example repo from David Nolen (the blog posts author) for some more examples.

https://github.com/swannodette/async-tests

juliangamble• 13 years ago

This is an example of Peter Norvig's spellchecker written with core.async by Stuart Sierra (@stuartsierra): https://gist.github.com/stuartsierra/5954022

This is an example of Rock Paper Scissors by Alex Miller (@puredanger): http://tech.puredanger.com/2013/07/10/rps-core-async/

vdm• 13 years ago

Microsoft Rx -style operations https://groups.google.com/forum/#!topic/clojure/L4nEVho555k

ganarajpr• 13 years ago

Can some of the clojure and clojurescript gurus take some time to write beginner friendly, hand holding tutorials? I mean, I can see the power of these technologies but the very thought of climbing a learning curve AS WELL AS a getting started curve ( that too without any external help! ) is the single most important thing that turns people off.. especially me. My general complaint is with regard to how many tutorials and even books are out there on these topics! Countable, with the fingers in one hand perhaps!

calibraxis• 13 years ago

Discoverability is a problem, but the Clojurewerkz people are top-notch (http://clojurewerkz.org/). These are helpful:

* Intro: http://clojure-doc.org/

* Libs: http://www.clojure-toolbox.com/

* Examples: http://clojuredocs.org/quickref/Clojure%20Core

* Pedestal: https://github.com/pedestal/app-tutorial

Also a few books at different levels:

* http://www.clojurebook.com/

* https://leanpub.com/fp-oo

* (2nd ed early access available) http://joyofclojure.com/

* http://pragprog.com/book/shcloj2/programming-clojure

* http://www.manning.com/rathore2/

* http://www.packtpub.com/clojure-data-analysis-cookbook/book

gtani• 13 years ago

Clojurescript book from oreilly, http://shop.oreilly.com/product/0636920025139.do

(On the whole, a really excellent set of books for the language)

alexott• 13 years ago

I would also mention following tutorial on CLJS: https://github.com/magomimmo/modern-cljs

graue• 13 years ago

If you're referring to core.async specifically, keep in mind it's very new. Its readme[1] goes as far as to say, "This library is work-in-progress, experimental, broken etc. It should only be used by people working on it, ATM." With more maturity will surely come more examples and tutorials.

But if you mean Clojure(Script) in general, I kind of agree. There are some beginner friendly tutorials; lately Andrew Brehaut's intro to the Clojure web stack[2] has been smooth going for me, despite being a bit dated. But you have to look around. In particular, I've found the small-scale mechanics of the language easy to pick up via Clojure Koans[3], 4clojure[4], and ClojureDocs[5]. What's missing the most IMHO is approachable introductions to the big-picture concepts. Especially the less generically lispy, more Clojure-specific parts, concerned with concurrency and state. (I hear the book Clojure Programming by Emerick et al. is good for that... have yet to read.)

I kind of understand Clojure not being easy or quick to learn, because powerful tools take time to master. But even with that said, it could be better. When I get confident with Clojure I hope to be part of a new wave of Clojurists who write really beginner-friendly tutorials :) I think it won't be the "gurus" who write them, but regular joes/janes like you and me for whom the initial difficulty of this stuff is still a fresh memory.

[1] https://github.com/clojure/core.async

[2] http://brehaut.net/blog/2011/ring_introduction

[3] https://github.com/functional-koans/clojure-koans

[4] http://4clojure.com/

[5] http://clojuredocs.org/

zhemao2• 13 years ago

I looked at Clojure at one point and fell in love with the lispy but not too parenthetical syntax. I left it behind after a while because I couldn't deal with the awful exception tracebacks which were supremely unhelpful when debugging. Most of the more arcane messages had to do with fundamental syntax errors, which was probably not as big a deal for an experienced programmer. However, it made it very difficult for a bigger to debug when you couldn't so much as see the line number on which you made the mistake.

They may have fixed that recently, and this CSP implementation is really cool, so I may take a look back at the language.

edoloughlin• 13 years ago

I agree that the exception traces are pretty awful. Having used Clojure for a couple of years now, all I can say is that you do get used to them. Despite the complexity of the libraries and the 'foreign-ness' of the concepts coming from an imperative background, Clojure is fundamentally a simple language/environment.

I've developed a sense for the types of mistakes that I usually make and I tend to scan the stacktrace for my own namespace, ignoring the clojure.* lines[1]. Once I see what line caused the problem, if it's not a typo then it just takes less than a minute to reason about what's going on and find a solution. The REPL helps a lot, because you can make changes on the spot or try out a modified code fragment.

As some others have said here: yes, it is difficult to make the adjustment from an imperative way of thinking but it's really, really worth it once you get there.

[1] You can probably even de-emphasise clojure.lang stacktrace lines in Eclipse using the Grep Console plugin but I'm not sure about this off the top of my head, because the Eclipse console might override stderr output.

rdtsc• 13 years ago

Maybe it is just me, but I prefer logically to think of actors (or processes) as concurrency primitives not channels, kind of like what Erlang has. For example in go the primary question is "what channel to I write or read from?", in Erlang it is "what is the pid of the process, so I can send it a message?".

aufreak32• 13 years ago

Channels are a neat generalization of what Erlang has. Btw the first example used in the post isn't all that "fall out of your chair". Here is the same implemented in pure JS using my toy IO.js library - https://gist.github.com/srikumarks/5992336

swannodette15• 13 years ago

People keep focusing on the first example. If you're going to compare, I think it's much more illuminating to see versions of the final example.

arnarbi• 13 years ago

I like to view pids as a special case of a channel. A pid is just the one-and-only channel of a process, and a process cannot create more. If I'm not mistaken, Erlang advocates creating processes for very specific tasks, so most of the time one channel is enough. But if you ever find yourself writing processes that check the "type" of an incoming message and dispatch that to different handlers, then consider if it could be easier if each message type came in on a different channel.

The security implications of channels are neat too, you can treat them as capabilities to specific parts of a process' API.

alco4• 13 years ago

Channels by themselves are not a concurrency primitive. If you try to write and read from a channel in the same context, you'll deadlock. Goroutines (or "go blocks") provide the concurrent semantics, channels are a means of communication.

In Go you don't actually ask "what channel do I read from?". You are handed a channel (as a function argument or as a value from another channel) which you can then use to communicate. From the encapsulation point of view, pids and channels are very similar -- the code using the channel/pid doesn't have to know anything about the other end.

But the semantics of channels and message passing are very different: always synchronous vs asynchronous by default.

novaleaf2• 13 years ago

the article is a bit of a cheap shot against Typescript. They have had async in the roadmap the last 6 months at least (and the public project is only about 10 months old).

IIRC, async is the next big feature on the roadmap, and going to be in before v1.0 (v0.9, released 2 months ago, added generics)

given the pace of development, typescript v1 should be done in 6 months or less.

chongli2• 13 years ago

I don't see how that makes it a cheap shot. Everyone is drawing from the same pool of ideas that were discovered decades ago. Could you be more specific?

novaleaf2• 13 years ago

typescript is very popular for how young it is. so given that the thing is still a very young, alpha product, saying that they are neglecting feature X is a bit too much. better to just not even mention it.

mrcactu5• 13 years ago

Where does the blog style come from ?

brucehauman2• 13 years ago

I'm pretty sure it's standard jekyll http://jekyllrb.com/

vanderZwan2• 13 years ago

Does anyone have any idea why the mouse-over examples are a more CPU intensive task for Firefox? On my laptop, moving the mouse a lot results in 15% CPU usage, and if I jitter it fast enough it maxes out at 22%. That won't lock up the browser if I'm not doing anything else but still feels like it will drain the laptop battery notably faster with intensive use, if you catch my drift.

With Chrome I can't get it above 4%. IE 10 is somewhere in the middle with 10% (and also has a bug when resizing the window, giving Y values like 2236.65990234375, with the fraction part being constant).

EDIT: Both Opera 12 and Opera 15: 6 to 8%

Also, the second/third examples seem a bit laggy with all browsers: if I shake the mouse quickly the numbers keep updating for a while after I stop. I guess that this is because instead of showing the most recent value it shows all the location values, but the screen updates are slower than the mouse updates?

Either way, when the mouse stops moving while the numbers still update, Firefox only takes 6% of the CPU. Not sure if that clarifies anything, but it might help identify the cause.

swannodette15• 13 years ago

I don't think you can come to any conclusions from these numbers. I see the exact same CPU utilization across browsers if I replicate the styles and write this JavaScript instead:

   var loc = document.getElementById("location");

   window.addEventListener("mousemove", function(e) {
     loc.innerHTML = e.pageX + ", " + e.pageY;
   });

Which is saying something, the ClojureScript is doing a lot more work and it ain't slow ;)

I suspect all the cost is from repainting large areas at the speed of mouse movement. But the mouse examples are for illustration - in production code you'll probably want to throttle (which would avoid piling up the repaints you observed on slower machines) as well as address the IE 10 edge case.

vanderZwan2• 13 years ago

Well, my question was why one browser was slower than the other - I didn't expect ClojureScript to be the cause of the problem. I was just curious about what might cause the difference.

I suspect all the cost is from repainting large areas at the speed of mouse movement.

I'm not really a web dev, but isn't this a reflow, not a repaint? Could that be the reason for the difference in performance too? What happens if you use canvas instead of changing a div?

BTW, I forgot to mention this in the previous comment, but: this is pretty cool! I echo the "being spoilt by Go" sentiments of others, so I'm happy to see this library.

MaysonL• 13 years ago

Obligatory Hoare link: http://usingcsp.com/cspbook.pdf

Communicating Sequential Processes

Related Stories