Elixir multiple processes: Basics

http://whatdidilearn.info/2017/12/17/elixir-multiple-processes-basics.html

by ck3g • 9 years ago

131 27 9 years ago

I'm in the process of learning Elixir and the observation I'm about to make may be completely off-base. With that said...

Seems like the section on "When to use processes" is selling processes short a bit. Certain kinds of state management would seem to call for processes to either manage or even hold state... but processes (as I understand them) are also key in designing for reliability. So I would think I may well want to organize my code relative to processes as I'm also figuring out the supervision trees and various failure scenarios. And yes, concurrency issues as well. If I'm wrong on this, I'd be happy to be set straight.

Anyway, yes, the section I speak of does get to some of the other parts of what I mention, but the emphasis on state management seems to distort and unbalance the view of what you might want as a process.

[edited a touch for clarity]

macintux3• 9 years ago

You are correct.

That's something I've been a bit worried about with people coming to Elixir thinking it's basically a faster/robust Ruby: are they absorbing the full Erlang Weltanschauung?

udfalkso• 9 years ago

(I had to look it up)

Welt·an·schau·ung ˈveltˌänˌSHouəNG

a particular philosophy or view of life; the worldview of an individual or group.

Personally I came for the faster/robust Ruby first and then over time discovered (and am still discovering) the other powerful pieces. I think that's ok.

debacle• 9 years ago

Was Elixir designed with Ruby in mind?

ck3g2• 9 years ago

I think you are right. Good point.

I am also in the process of learning, so your reply is a valuable for me.

Thanks

But does reliability in Elixir/Erlang come only from processes or is it result of the "teamwork" of several features of the platform itself?

dozzie• 9 years ago

You won't have Erlang-style reliability (predictable recovery after localized errors) without links, monitors, and supervisors, which themselves are built with links and monitors.

fasteo• 9 years ago

Elixir newbie here.

My understanding if that Agent[1] was designed specifically to keep state. But then again, an Agent is a GenServer is a process, so I guess the post just served as a introduction to the concept of a Elixir process.

[1] https://hexdocs.pm/elixir/Agent.html

ck3g2• 9 years ago

Yes. That is correct. I am going to cover Agents, Tasks and OTP in the following articles. I thought the article would be too big if I cover all three topics at once.

redshirt• 9 years ago

Very cool. I'm adding this to my list of easy concurrency tools. The main thing I like is the statelessness. That's where most people screw up parallel programs.

Seems like there are many languages/libraries trying to make concurrency easier to implement in practice. Most notably for C++ (my fav since I have to use it for most of my work projects): Intel TBB (definitely the go to for most things), RaftLib (saw at C++Now last year) is probably the easiest to understand (same theme as this post, super easy concurrency for c++). Even Java seems to make concurrency rather easy with it's thread pools and relatively strait-forward synchronized sections.

MaxBarraclough• 9 years ago

I recommend taking a look at .Net/C#'s concurrency machinery. It makes it terribly easy to use its thread-pool, and its async/await stuff is very cool. Think TBB tasks, but with very nice sugar.

My understanding is that Java's current concurrency offerings are much like C#'s, but... worse.

lostcolony2• 9 years ago

Okay, taking a step back - this isn't just another concurrency tool. This is a language built for concurrency.

Taking another step back - this isn't just a language built for concurrency. This is a language built for -resiliency-.

Taking another step back - this isn't just a language built for resiliency, this is a language built atop a VM built for resiliency -over 30 years ago- (the Erlang VM, aka, the BEAM, which Elixir runs on).

Okay. Why does this matter? Well, the thing is, resiliency encompasses concurrency and distribution, both. It prioritizes error minimization and, more importantly, the ability to recover from errors. This isn't just a try/catch; this is a "something you completely failed to even expect caused things to fail in a way you can't even imagine, and the system still handled it".

It achieves that via immutability, concurrency, and distribution. Ensure your data is immutable, so that state has to be very explicit (it's not stateless...a process has state. But it's very explicit state; as a developer you can't help but handle it and be very aware of it). Ensure bad states are dropped, and the system can recover the execution unit from a good state. If it can't, allow a user defined subsystem to fail, as the intricacies between the entire subsystem are implicitly stateful, and restart the whole thing from a known good state. If even that fails, keep climbing the supervisor tree, restarting larger and larger subsystems, until you restart the entire -application-, assuming that the intricacies across subsystems have gotten into a bad state, and again, restart from a good state.

These principles have been around a long time, but Erlang is one of the first languages to put them into practice, again, over 30 years ago. There's been a lot of time since to see they actually work, and to further refine them. The difficulty with concurrency is not actually being concurrent (per your post, there are a LOT of ways to implement concurrency); the difficulty is doing it in a way that it behaves how you want it, even in the face of user's doing things you don't anticipate, external resources doing things you don't anticipate, your own code doing things you don't anticipate, etc. The design decisions that went into Erlang focused on minimizing errors...in so doing, it provides a way that most kinds of errors are handled transparently (from logic bugs to actual machine failure), while making it much harder to do things that it can't recover from (memory leaks are comparatively difficult to cause, as are deadlocks, for instance).

To give you an idea, the first commercial product built with Erlang boasted (famously) 9 9s of uptime. Meaning something on the order of ~30ms of downtime a year. That includes planned downtime, visible errors, etc.

I've seen Erlang systems in production...even with a rather critical bug in one, the system just -worked- for -years-, before someone noted an oddity in the logs, dug into it, and went "Oh my God" over how severe the issue was. But, again, Erlang's supervisor process just restarted it, and it was never noticed.

I helped write CNN's current video ingest system in Erlang. It's been working without issue for years, despite no maintenance or attention (to where even most of the developers have left, but it still just...works). Even much less complex Ruby, Java, and Javascript systems are plagued with constant bugs. I would not say the devs on this project were just that much better (though the process was a little different, with little product owner involvement), but that the language we picked was so much better geared toward fault tolerance.

nickjj4• 9 years ago

Maybe I'm thinking about this incorrectly but when it comes to web application development and concurrency, the things I would typically want to run in a separate process are very important tasks.

For example, let's say you're sending emails out.

In Rails, Flask, etc. you would typically offload this to a background worker like Sidekiq or Celery. These are dedicated libraries that run in their own process and handle job processing.

Both tools allow you to see if that job was successful or failed and deals with retries. They also use Redis as a back-end so your state persists outside of the application code.

If you just willynilly spawn a process and start sending stuff through it, how do you ensure everything works as expected and what happens to all of your state if the Erlang VM goes down?

I love the idea of doing this, but in real world practice, it sounds like you would still need the Elixir equiv. of Sidekiq / Celery to handle ensuring these spawned tasks are trackable, right?

narrowtux2• 9 years ago

Of course you could make such a system in Elixir, and I'd always recommend doing things within the erlang VM (beam) instead of figuring out how to deploy another service (like install sidekiq with all its dependencies).

However, processes in Erlang/Elixir are used ubiquitously. Most of the time, they run forever like some sort of service and are restarted when they crash. Sometimes, you won't care about the result.

That's the reason why features such as tracking success/failure in a persistent data store is not a core feature. You could easily add those features to your processes though, the necessary API is already there.

The alternative is to use message queues that are durable. Only ack the message when you successfully handled it. If your application crashes before the task was completed, the task will be redelivered the next time the application starts.

nickjj4• 9 years ago

I definitely wouldn't want to be responsible for writing this code myself. Something like Sidekiq has been worked on for 5+ years by hundreds of people.

Does anything exist in the Elixir world that's as battle hardened and has comparable features to Sidekiq / Celery?

sbuttgereit4• 9 years ago

I don't think so. First, just to make sure there aren't any points of confusion... Erlang/Elixir processes are not OS processes (https://elixir-lang.github.io/getting-started/processes.html). As such, we really can't speak of processes in Erlang/Elixir and other kinds of systems on any sort of equal footing. In Erlang, processes are much lighter weight than OS processes. And you do expect them to go wrong... so Erlang has the idea of "Supervision Tree" where there are Supervisor processes whose job it is to monitor other processes and to manage their failure when it occurs. There can be multiple Supervisors which influence one another, or not, as you design them too (thus the "tree" bit). Naturally you plan for these sorts of failures in designing what is/is not a process, what dependencies there may be, what any Supervisor watches and how the Supervisors relate to one another.

Erlang/Elixir also seems to have one of the strongest availability and concurrency stories out there on its own. Part of this comes from the aforementioned built in assumptions around isolating and managing failed processes, but the other is in terms of the relative ease of starting processes as needed, and accessing the processes of other Erlang VMs (same server or not). It's clear you have to architect and build things correctly to take full advantage of these capabilities, but I think you can get to the place you describe without necessarily taking on a bundle of different applications for different purposes.

For example, for many of the use cases I think of Redis, Erlang offers a number of different ways to deal with such task out of the box: https://blog.codeship.com/elixir-ets-vs-redis/ Again, you have to think about how you architect things, but you can use the same toolbox for much of the work.

I'm still new to Elixir/Erlang, but I've been studying capabilities and architecture for several months now. I think there are sufficient differences in Erlang's approach as compared to other more common stacks that it really does benefit one to come at it fresh and not make very many comparative assumptions. In my own learning I've found that to just get the basic feel for what this Erlang/Elixir animal really is, that the Elixir side is more approachable. The documentation tends to be a little more direct and tends to be more beginner friendly. After I got the very basics down, I was able to look at the Erlang side and understand really what they were trying to say.

sbuttgereit4• 9 years ago

FYI... I think that https://news.ycombinator.com/item?id=15952005 give a more complete and rounded answer in many ways, definitely worth a look.

auxbuss• 9 years ago

Saša Jurić's Solid Ground talk is a lovely demonstration of Elixir's processes: https://www.youtube.com/watch?v=pO4_Wlq8JeI

(Saša is also the author of Elixir in Action; a highly recommend read once you have your head around Elixir's fundamentals.)

Elixir multiple processes: Basics

Related Stories