Let’s Talk Concurrency: Panel with Sir Tony Hoare, Joe Armstrong, Carl Hewitt

https://www.erlang-solutions.com/blog/let-s-talkconcurrency-panel-discussion-with-sir-tony-hoare-joe-armstrong-and-carl-hewitt.html

by mpweiher • 7 years ago

250 113 7 years ago

erlang-solutions.com

lelf2• 7 years ago

Videos https://www.youtube.com/user/ErlangSolutions/videos

paulhallett• 7 years ago

Do they all talk at the same time?

throwaway8879• 7 years ago

Concurrency is not parallelism, friend.

cwingrav• 7 years ago

This guy. Fun at parties. ;)

A good point but most would think of the two the same. I know I had a hard time getting some ideas across on some research I was doing until I separated these two concepts out. Then people started to understand.

glockenspielen2• 7 years ago

"concurrency includes parallelism as a special case." -Carl Hewitt http://lambda-the-ultimate.org/node/5231?from=200&comments_p...

RobertoG• 7 years ago

No, they optimize the talk filling the others speakers pauses.

avip• 7 years ago

Organizers gave each a single fork for dinner.

AzzieElbab• 7 years ago

Is this happening over a dinner?

inopinatus• 7 years ago

They try, but then each stops, waits a random interval, then speaks again.

eddd• 7 years ago

I used to drink a lot of cool-aid from erlang. It is true, that they have got concurrency model right. They embraced actor model, messages between processies are copied over for a good reason, etc. However, I don't think it is enough to call it a day. Comparing to modern languages, erlang lacks a lot. Elixir is trying to fill those gaps, but how many layers of abstraction can you add to an ecosystem before it is unusable? With go and rust available, erlang/elixir looks like a very good tool but for a very limited pool of use cases - routing/filtering/messaging.

keymone• 7 years ago

erlang/elixir looks like a very good tool but for a very limited pool of use cases - routing/filtering/messaging.

wasn't that the original goal for erlang?

b3orn3• 7 years ago

Comparing to modern languages, erlang lacks a lot.

Could you elaborate on that? What exactly does erlang as a language (tooling/libraries/frameworks aside) lack in your opinion? What do those modern language have that erlang doesn't?

lelf2• 7 years ago

Comparing to modern languages, erlang lacks a lot.

Like?

With go and rust available,

Yeah, tell me when those “modern languages” will be distributed, stop using stop-world GC, and contain at least 1% of introspection Erlang has.

christophilus• 7 years ago

Rust, at least doesn't use GC. But yeah. Introspection and distribution is something BEAM does super well. I don't know of a better platform for that.

jerf• 7 years ago

(Actually a meta-reply to several comments.) Erlang got a lot right a long time before anyone else, but when that happens, it is a strong temptation to assume that everything about that early success is fundamental to the solution and anyone lacking it can't possibly succeed. But you don't really have the evidence for that, because for a long time, you had only one data point, and you can't draw lines (let alone n-dimensional hyperplanes) through one point meaningfully.

The evidence says that modern people mostly don't care about distribution, don't care about stop-the-world GC in a very large percent of the cases (especially as stop-the-world time is getting shorter and shorter), and don't need Erlang-style introspection very often. I know how useful the latter in particular is, because I also used Erlang for a long time and I used it a lot. But again, what happens, especially when you''ve got the only solution, is that the one solution selected early gets worked into the system very deeply and looks very fundamental, but that doesn't mean that it's the only viable way to do it. So when running an Erlang system, I needed a lot of introspection to keep it running. But when I run non-Erlang systems, I do not simply collapse into a heap and wail that I don't have introspection and watch helplessly as the system collapses around me... I solve the problem in other ways. Entire communities of people are also solving the problem, and sharing their solutions, and refining them as they go as well, just as Erlang did with their solutions.

The Erlang community has basically been complaining for the whole nearly 10 years of Go's 1.0+ existence that it doesn't have every single Erlang feature, but it was never about having every single feature that Erlang has. Erlang is a local optimum, and while I think it's a very interesting and educational one (and I mean that, quite sincerely and with great respect; anyone designing a new language today ought to at least look at Erlang and anyone designing a language with a concurrency story ought to study it for sure), I'm not even remotely convinced it's a global optimum. To get to any other optimum does mean that you'll have to take some steps back down the hill, but if all you look at are the steps down the Erlang slope but not the steps up some other slope, you won't get the whole story.

(I would call out the type system in particular as something deeply sub-optimal. I understand where it came from. I understand why someone in the 1986, with 33 fewer collective years of experience than we have now, would say that the easiest way to have total isolation is to use total immutability, and the simplest way to ensure types can't go out of sync is to not have types. But it crunked up the language to have immutability within a process (that is, "A = 1, A = 2" could have totally worked like any other language without breaking Erlang itself; have separate operators for what = does today and an operator for "unconditional match&bind" and everything works fine), when all it needed was to ensure that messages can't contain references/pointers but can only copy concrete values. And it doesn't solve the problem of preventing things from going out of sync to simply not have types, because you still have types in all of the bad ways (if two nodes have two different implementations of {dict, ...}, you still lose in all the same ways), you just don't get types in any of the good ways. It was a superb, excellent stab at the problem in 1986, and again, I mean that with great respect for the achievement. But in the cold harsh world of engineering reality, it is one of the huge openings for later languages to exploit, and they are. There are others, too; this is in my opinion one of the biggest, but not the only.)

fwip• 7 years ago

Rust doesn't use GC unless you explicitly use a garbage collector library.

Go's stop-the-world pauses are usually sub-millisecond for heaps measured in the tens or hundreds of gigabytes.

Jtsummers• 7 years ago

go and rust are for a different situation. While they both have good concurrency capabilities (from using go, from reading about rust), neither has OTP. Erlang isn't just a language, it's basically a distributed operating system (nodes on the same or different computers). It's been a while since I've used go, and I haven't used rust for anything beyond small toy stuffs to get a feel for it, but do either have, built-in, the distribution capability of erlang?

_asummers2• 7 years ago

I fundamentally disagree with your premise, despite seeing how you came to that conclusion. Elixir/Erlang are particularly optimized for the operations you're speaking about, but Elixir is very Lisp-ey under the hood. Macros are a game changer. Combine that with a strong standard library, many of which is delegated down to the Erlang calls anyway, and the the pure developer joy that comes from coding in Elixir in both the small and the large, increased debuggability from having highly readable, functional code.

But the real power comes from the BEAM. Turns out modern servers map very strongly to phone switches of the past, and the distributed system primitives given by the BEAM keep on ticking, 30 years later. Modeling a web server as a single process per request, the supervision model, and the power of preemptive scheduling is something I don't see in other languages, at least as explicitly. Preemptive scheduling is really a wonderful thing, and I don't think Go or Rust provide this. Please correct me if I'm wrong. This is to say nothing of the observability, hot code reloads, or any of the more fundamental parts of the BEAM that you wind up needing in practice.

I'll be frank, I think Go is an unnecessarily verbose language. I don't like reading it, and any time I've had to write it, I have not enjoyed it. I find Go's concurrency model worse than Erlang's despite being similar at first glance. GenServers are a much better abstraction to me than GoRoutines and friends. If it weren't from Rob Pike and the marketing of Google, I don't think it would be nearly as popular as it is. The type system from Rust is great, and the borrow checker is a fantastic addition to type systems especially in that class of language, I have no use for Rust in my daily life. It is on my short list of languages to become more familiar with, though.

eternalban6• 7 years ago

If it weren't from Rob Pike and the marketing of Google, I don't think it would be nearly as popular as it is.

This is only true in part. The fact is that Go is a supremely accessible language (warts and all) that helps you get things done.

cdoxsey• 7 years ago

Go is not fully preemptive, but in practice it usually is (it preempts at function calls): https://github.com/golang/go/issues/10958.

Thaxll4• 7 years ago

The thing is the BEAM model doesn't have a bright future because it can be replaced by Kubernetes and is language neutral, almost all the feature BEAM provides are better done in k8s ( HA, deployment ect ... ). As for hot code reload, I've never seen why you would need that since you can use blue / green or canary / rolling deployment, the only reason I see is to keep some state in your app, which I think is a terrible idea.

Two others things:

- deploying Erlang / Elixir app is difficult ( even with distillery... )

- Erlang is slow, much much slower than Go

nostrademons2• 7 years ago

"Modeling a web server as a single process per request, the supervision model, and the power of preemptive scheduling is something I don't see in other languages, at least as explicitly."

That's how most production websites of the past 20 years have been built, but these services are pushed up to the OS level rather than the language level. Apache, PHP, CGI, and everything built on that ecosystem used a process-per-request model. The OS provided preemptive scheduling. If you were doing anything in production you'd use a tool like supervisord or monit to automatically monitor the health & liveness of your server process and restart it if it crashes. The OS process model restricts most crashes to just the one request, anyway.

There was a time in the early-mid 2000s when this model gave way to event-driven (epoll, libevent, etc.) servers and more complicated threading models like SEDA, but the need for much of that disappeared with NPTL and the O(1) scheduler for Linux, though process-creation overhead still discourages some people from using this model. Many Java servers are quite happy using a thread-per-request or thread-pool model with no shared state between threads, though, which is semantically identical but with better efficiency and weaker security/reliability guarantees.

Now, there continues to be a big debate over whether the OS or the programming language is the proper place for concurrency & isolation. That's not going to be resolved anytime soon, and I've flipped back and forth on it a few times. The OS can generate better security & robustness guarantees because you know that different processes do not share memory; the language can often be more efficient because it operates at a finer granularity than the page and has more knowledge about the invariants in the program itself. One of the interesting things about BEAM (and to a lesser extent, the JVM) is that it duplicates a lot of services that are traditionally provided by the OS or independent programs running within the OS. In some ways this is a good thing (batteries included!), but in other ways it can be frustratingly limited.

steveklabnik• 7 years ago

Rust offers no specific scheduling; only type system affordances for describing important properties around parallelism and concurrency. The standard library gives an API for the OS’s threads, and soon, an API for defining cooperative tasks. Being a low-level language, you can implement any sort of primitives you want. There’s an actor library built on top of said tasks, for example.

zzzcpan5• 7 years ago

I don't see a point here. Elixir doesn't actually fix all that much. And other modern languages still haven't managed to get concurrency model right. So the answer is we need new languages, not Go and Rust. With high performance actor model runtimes, AOT compilers and while we are at it addressing modern security problems of speculative CPUs, 3rd party packages, etc.

pjmlp8• 7 years ago

While I do like Rust, many of its improvements are already present in Ada, and Rust still lacks something like SPARK.

Naturally since Ada is tainted by its history, other approaches more interesting to the common crowd are needed.

SZJX3• 7 years ago

I wonder what you see that Elixir doesn't fix. Syntax-wise, Elixir definitely makes writing the code a much much more pleasant and enjoyable experience. And I would say the package ecosystem has been great and really promising so far. I guess the biggest complaint against BEAM would still be its performance and lack of easy AOT compilation, as you wrote there. However, the dynamic nature of Erlang/Elixir makes the development process much easier and more rapid in many cases. Also, compared with most of the dynamic languages that I've used so far, it's amazingly easy to maintain and bug-free. Of course you can always pine for the one perfect language, but PL design is always a process of making tradeoffs, and IMO Erlang/Elixir already satisfies the vast majority of use cases while being extremely enjoyable to work with. There are also some attempts to develop statically typed languages targeting the BEAM though the message passing between actors makes it not that easy.

gurraman2• 7 years ago

Many of the systems I build do require routing/filtering/messaging and I have yet to find a more pleasant environment to work in than Erlang. I can agree that the ecosystem is a bit lacking if you want to build a quick web application, but out of curiosity, what are you referring to when you say that Erlang lacks a lot compared to modern languages?

mercer• 7 years ago

How do you feel about Phoenix (and, by extension, Elixir) as something that might scratch the 'quick web application' itch?

As a web developer, I've been quite happy learning Elixir/Phoenix in the past year, and learning Erlang has been on my list, so I'm very interested in hearing from people with more Erlang experience when it comes to 'web stuff'.

rlander• 7 years ago

It is like complaining that Prolog got logic programming right but that comparing to “modern languages” (whatever your own definition of that is: Rust? JS? Haskell? Swift?) it lacks a lot. Prolog, as Erlang, was never designed to be a general-purpose language.

pjmlp8• 7 years ago

Actually from my dabblings with Erlang I never felt the need to deal with Elixir, then again I was a big Prolog fan during university days (yes I do know that the resemblance is only superficial).

notduncansmith• 7 years ago

Not to be contrary but I’d call it more than superficial, the inventors of Erlang were using Prolog before they created Erlang. Also, IIRC the first version of what would become Erlang was actually a Prolog variant.

yellowapple• 7 years ago

The problem is that Erlang and its BEAM-based descendants still seem to be the only languages that actually do get concurrency completely right. Lots of languages have some form of an actor model, sure (whether as libraries or - like Pony - baked into the language), but all of them seem to rely either on OS threads per process (which are obscenely heavy) or green threads / coroutines (which lack preemption) (if you happen to know of any other languages/runtimes that offer lightweight preemptive concurrent actors/processes, let me know).

Until that happens, BEAM is unfortunately a hard dependency on getting the concurrency model fully "right".

rdtsc4• 7 years ago

Comparing to modern languages, erlang lacks a lot.

The "a lot" part is emphasized. It has to be something massive? Classes? Objects?

With the same token one can say most other languages lack a lot as well. That "lot" would be fault tolerance. Notice how most operating systems today use isolated processes instead of sharing memory like Windows 3 or DOS did. There is a reason for that. When the word processing application crashed, it would take down the calculator and media player with it. So modern operating systems have isolated concurrency units. And so do languages built on BEAM VM.

And of course you could still spawn OS processes and run a language that uses shared memory between its concurrency units (threads, goroutines, callback chains). But you can't spawn too many. Or even worse, everything has to run in a container, so now you have a container framework as another layer. And the question then becomes "how many layers of abstraction can you add to an ecosystem before it is unusable?" ;-)

Elixir is trying to fill those gaps, but how many layers of abstraction

Elixir is not built on top of Erlang the language. It compiles to the same BEAM VM bytecode. But the intermediary representation layers between the running bytecode and the language syntax was already there. They didn't add another layer on "top" so to speak.

They embraced actor model,

Don't think so. The creators at the time had no idea about the actor model. They embraced fault tolerance, concurrency and low response time most of all.

0815test2• 7 years ago

That "lot" would be fault tolerance. Notice how most operating systems today use isolated processes instead of sharing memory like Windows 3 or DOS did.

Rust has full memory safety for concurrent code (if you restrict yourself to the Safe Rust subset), unlike other commonly-used languages such as C/C++, Go, JVM/CLR-hosted languages etc. This provides a far more general model of concurrency than something like the Actor model; it can also express other patterns very naturally, such as immutability (like Haskell), channels/CSP (like Go), plain old synchronized access etc. Of course Actors can be expressed too, as can isolated processes ala Erlang (that's what exclusive mutability is for!) but you're never restricted by the model.

jeremyjh• 7 years ago

Elixir is not built on top of Erlang the language. It compiles to the same BEAM VM bytecode.

Actually it is, at a semantic level. Elixir source is reduced to Erlang's abstract syntax tree format, which is represented in Erlang terms. The same Erlang compiler is used to generate BEAM code for both languages. This isn't just a detail - the ramifications of using Erlang semantics permeates the language. But not to the language's detriment in any way.

carlehewitt9• 7 years ago

The Actor Model is a formalization of what needs to be done for IoT and many-core computers. The ideas were circulating widely before work began on Erlang even if the engineers did not read the literature.

gambler2• 7 years ago

how many layers of abstraction can you add to an ecosystem before it is unusable?

Isn't this even more true when speaking about serverless and all the surrounding technologies?

liveoneggs• 7 years ago

ecosystem and various beam details aside erlang has tons of great stuff but it is often difficult to use, which limits it hugely and makes it feel "old" etc.

Simplifying, modernizing, and exposing all of the tracing and debugging and releasing and other goodies inside of BEAM would be huge.

carlehewitt9• 7 years ago

Erlang lacks holes in the region of mutual exclusion of an Actor making it very difficult to things like a readers/writer scheduler for a database.

See the following:

http://web.stanford.edu/class/ee380/Abstracts/190123.html

glockenspielen2• 7 years ago

Mutual exclusion: https://en.m.wikipedia.org/wiki/Mutual_exclusion

Are you saying that the Erlang VM enforces mutual exclusion within its Actors, and that mutual exclusion should not be enforced within Actors?

qaq2• 7 years ago

Pony looks like an interesting option but the ecosystem is tiny at the moment

mruts3• 7 years ago

The Actor model sounds really great, and while I haven't used Erlang in a serious context, I have used Scala Akka Actors extensively and I find them very difficult to reason about. Actors might be great if you have a huge system over the network that you need to manage, but I don't find them that great for single computer programs.

For example, I tried to implement a Akka bittorrent client and found it very difficult and confusing. You end up with a big graph of actors in which most of the messages sent are related to keeping track of state and actors.

Moreover, actors are untyped so it's not obvious what messages they accept of send. Without a lot of documentation and discipline, the code becomes extremely unreadable very very quickly. Also bugs are very hard to find and very easy to create.

Maybe it's less elegant, but for a lot of problems, just using select or epoll is a lot easier.

thedoops• 7 years ago

Scala and Akka are... less simple. It's also not like the actor model takes over the whole paradigm. Actors are largely a runtime concern. You're still using modules and functions. The JVM is also less suited to implementing an Actor model. A lot of the BEAM's power comes from the preemptive scheduler.

haspok2• 7 years ago

That (the preemptive scheduler) and OTP/gen_server, which covers about 90% of the cases Erlang should be used for anyway. These are the foundations of Erlang-land, however, sadly neither of those have equivalents in Akka or on the JVM.

haspok2• 7 years ago

"You end up with a big graph of actors" - you are almost certainly doing something wrong, the actor graph shouldn't need to be "big", and the supervisor hierarchy almost never deeper than 2-3 levels. A typical mistake when doing actor-based system design is to think that actors are replacements for modules or classes (or taken to the extreme: function calls), which is not the case.

chongli• 7 years ago

According to Carl Hewitt, the actor model is based on physics (especially quantum physics). Doesn't this then imply that the natural scope of such a system entails huge numbers of actors, interacting in a probabilistic way?

So in order to understand our systems with huge graphs of actors, we should attempt to develop something akin to the heat equation for actors. Trying to understand the behaviour of every individual actor in the context of the whole system is just too much.

carlehewitt9• 7 years ago

Massive Inconsistency Robust Ontologies will have trillions of Actors on a many-core computer.

Please see the YouTube video here:

  http://web.stanford.edu/class/ee380/Abstracts/190123.html

mruts3• 7 years ago

I will admit that I do not have much experience creating actor systems. But the actor model in inherently complex. To be fair, that's because it's trying to solve a complex problem. But unless you actually need thousands of concurrent processes, often you can do without it.

dkarl• 7 years ago

It's helpful to keep in mind some parallels between actor systems and inheritance hierarchies.

- The tendency is to start extremely elaborate and become much more minimalistic with experience.

- The educational materials do too good a job at explaining advanced techniques without warning about how important it is to keep things simple in practice.

- A complex actor system or inheritance hierarchy should both reflect AND help tame complexity in the problem you're solving. If it doesn't meet both of these criteria, it should be simplified.

- There is no semantic reason that any relationship in a problem domain needs to be modeled by inheritance or actors despite being painful to program that way. The technique should only be applied where it helps.

I think (not from personal experience, but from what I read) that the big exception to the above is when you're using actors to optimize latency or throughput on a big multi-core machine. Then you will find yourself taking on complexity for the sake of maximum performance. I think educational materials often present techniques from these difficult, carefully-built systems as if they're helpful for writing simple, small services. This is also a parallel to inheritance, if you take a step back: high-performance data processing systems are the canonical problem for this era, just like complex native GUI programming was the canonical problem in the heyday of big inheritance hierarchies. Lessons from those domains can be misleading when applied to simpler problems.

bjoli• 7 years ago

I have a long time been a evangelising concurrent ml. It generalises the actor model, and for me it has always hit the sweet spot between "do everything yourself" and "one inflexible way or the highway".

zzzcpan5• 7 years ago

IIRC, it tries to "generalize" CSP, or rather just make it usable, because it isn't in its default form at all. CSP is not as general as actor model. But you can't generalize actor model into a more general concurrency model, unbounded nondeterminism is as far as we can go.

carlehewitt9• 7 years ago

Actually, the Actor Model generalizes Concurrent ML because message and types are Actors.

For example, if anAccount:Account then the following

     anAccount.deposit[$5]

is defined as follows:

     Account.send[anAccount. deposit[$5]

_ZeD_• 7 years ago

I played with erlang at uni, and now I'm working with scala, and I find the latter very messier than the former

lliamander2• 7 years ago

I worked on a hierarchical, distributed control system written in Erlang, and I found the actor model well suited to that problem domain.

We had to control various devices based on sensor data, but we also needed to make sure that the workload was distributed across the devices in an efficient way. To implement that, we had one actor per device that was solely responsible for adjusting the device output based on sensor data. Then we had another actor that was responsible for looking at the behavior of a cluster of devices and distributing the load between them. This "load distribution" actor wouldn't control the devices directly, but instead would send adjustments to the actors directly controlling the device.

I find it very natural to anthropomorphize actors. That is, if you were to think about how you would organize a group of people to do the work of the system (in a very Taylorist sort of way) the roles of those people map neatly onto the actors one would create.

One common pattern is the manager/supervisor/worker pattern[0]. I see this as mapping fairly neatly onto the project manager/people manager/employee structure that you see in many organizations.

[0] https://zxq9.com/archives/1311

euroclydon• 7 years ago

I think what most people want to know is whether they could re-write a multi-threaded program that is currently using synchronization primitives (locks, semaphores, mutexes, etc) with actors.

Like, show me the producer-consumer pattern using actors.

carlehewitt9• 7 years ago

In a wide-ranging discussion, there were some fundamental disagreements among the panelists as follows:

I disagreed with Tony Hoare about using synchronous communication as the primitive because it is too slow for both IoT and many-core chips. Instead, the primitive for communication should be asynchronous sending and receiving, from which more complex protocols can be constructed.

Also, I disagreed with Tony about sequential actions (using ";") as being foundational. Instead, concurrent actions are foundational for digital systems as follows:

    * Receipt of a communication activated sending other communications
    * An Actor received one communication before it received another communication

Consequently, a computation is a partial order of causality. Tony and I did agree that tooling is needed for navigating the partial order. We just disagreed about whether sequential actions (using ";") are foundational.

Furthermore, class hierarchies are not a suitable foundation for Scalable Intelligent Systems. Interfaces instead of subclassing should be used for IoT communication. Also, entities and descriptions in large ontologies do not fit in an object class hierarchy, e.g., Java and C++. Subclassing is not secure because it allows a subclass to impersonate a superclass.

I disagreed with Joe Armstrong about requiring use of external mailboxes because they are inefficient in both space and time. Instead of requiring an external mailbox for each Actor, buffering/reordering/scheduling should be performed inside an Actor as required.

Of course, Tony and Joe made other great points with which we agree entirely. See the following for more information:

  http://web.stanford.edu/class/ee380/Abstracts/190123.html

acje• 7 years ago

One of the things I can't see the actor model explain is how to scale an actor vertically, which obviously has it's purpose. Amdahl's law and all that.

However these days we see people build actor frameworks using CSP (golang) and "restricted shared memory" in Rust http://actix.rs

Isn't it ironic? Yet shows that the actor model can be inclusive of many "competing" paradigms, and gain from them, perhaps even need them to scale vertically?

carlehewitt9• 7 years ago

This is a very good question.

Citadels are larger scale Actors which can be incorporated into other Citadels, where a Citadel is an Actor for a systems of Actors (perhaps including IoT).

See the following:

   http://web.stanford.edu/class/ee380/Abstracts/190123.html

carlehewitt9• 7 years ago

Amdahl's law imposes no performance limitation on a system which does not have a sequential part. Having a sequential part is a bad idea because it is a single point of failure.

Let’s Talk Concurrency: Panel with Sir Tony Hoare, Joe Armstrong, Carl Hewitt

Related Stories