Erlang Scheduler Details and Why They Matter

https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-scheduler-details.html

by hamidreza-s • 10 years ago

188 30 10 years ago

hamidreza-s.github.io

Lots of Erlang articles making it to the front page... I like.

wheaties• 10 years ago

Is there a resurgence of interest in Erlang because of Elixir? Or has the Actor model really taken a hold of developers because of it's direct and easy to understand model of concurrency?

lmm• 10 years ago

I think there's a growing appreciation for functional style more generally (e.g. pattern matching) and Erlang's just caught up in it. Compare OCaml.

hamidreza-s3• 10 years ago

There are lots of demands for making things concurrent, fault-tolerant and distributed nowadays, not only in big companies but also among startups. Erlang already has those features as well as standard tools (OTP) which lets everyone to create something simply.

However among all the languages which run on top of Erlang virtual machine (BEAM) like LFE, Elixir, Efene, Luerl, Erlog, and such the Elixir has an active and bigger community, more interesting frameworks and wider adoption. So it could be claimed that it helped to introduce Erlang values to a wider range of developers.

Everhusk• 10 years ago

A lot of the rails community is migrating to Phoenix so I would imagine it has something to do with it.

davidw2• 10 years ago

The short version:

* 'event driven' things like Node.js got popular because they use fewer resources to serve the same amount of data (generally). But for a lot of people, JS is not really their idea of a good time. Erlang (and Go) fill this niche pretty well.

* Elixir finally made Erlang more palatable to more people. Jose has done a superb job with it, because it's not just a nicer syntax, there is a bunch of nice stuff he's built into it. It doesn't hurt that as a former(?) member of the Rails core team, he has a deep understanding of web programming and its requirements.

* It is solid, solid tech. There's lots of new development happening on top of, say, Elixir, like Phoenix, but the underlying system is pretty hardened.

bryanjos• 10 years ago

The Erlang scheduler is very interesting. I attempted to reimplement it in JavaScript with some success. It works like the first version of the scheduler but I could see web workers being used to make it more like the current version. https://github.com/bryanjos/processes

kitd• 10 years ago

    in case of high level platforms, languages or libraries     
    it can be claimed that Erlang virtual machine is almost 
    unique because JVM threads depend on operating system 
    schedulers, CAF which is a C++ actor library uses 
    cooperative scheduling, Golang is not fully preemptive 
    and it also applies to Python’s Twisted, Ruby’s Event 
    Machine and Nodejs.

Most languages/runtimes that are taking concurrency seriously do so using libraries or frameworks (eg Akka on the JVM) that look a lot like Erlang's under the hood. Erlang is not quite as unique as it was, but obviously baking it into the language is v useful.

phamilton2• 10 years ago

Akka is not preemptive. That's the problem with the library solutions, it's still a squarish peg in a round hole.

strmpnk3• 10 years ago

As far as I understand, you can make an actor reentrant which frees the thread. It's definitely a leaky concern compared to Erlang's process implementation.

felixgallo2• 10 years ago

to be fair, the original article is fundamentally incorrect and Erlang is not truly preemptive either. It uses a reduction-counting based cooperative multitasking system which yields at function calls, but a badly programmed NIF can still ruin your day if you don't put it on a dirty scheduler.

bad_login• 10 years ago

FYI racket and gambit VM use preemtive green thread, but they have nothing to do with SMP.

bnchrch• 10 years ago

I am enjoying the steam that both Erlang and Elixir are gaining here on Hacker News. They're both interesting, well designed languages built atop a rock solid VM with some great abstractions for concurrency.

I'm hoping that this trend continues because out of all the noise of new languages and frameworks I feel like Erlang/Elixir are very deserving of becoming the new way forward for developing anything that runs on a network.

dfischer• 10 years ago

Absolutely this. I've enjoyed JavaScript the past few years. (Dare I admit that). And the only light I see is something like Elixir that has a fundamentally superior foundation for scale and concurrency. I'm intrigued and am hoping elixir / erlang / Phoenix keep growing.

markatkinson2• 10 years ago

I stumbled onto an Elixir article a few months back (on HN) and was enchanted by it, ever since then I have been banging away in my free time at Elixir books and tutorials, finished intro to Elixir last night and half way through Programming Elixir.

It is a super exciting language to learn and those moments where the ball finally drops on concepts like pattern matching are glorious, it has reignited that magical feeling of being some sort of tech wizard that I first got when I started programming as a child!

Anyway, enough of the gospel of Elixir, I too look forward to seeing how it/Erlang develop and feel like it is already getting some noteworthy traction.

P.S. I also scan HN daily for anything Erlang/Elixir, and frequency of posts seem to be increasing.

cvik• 10 years ago

To my knowledge, Erlang uses a cooperative scheduler. The process itself counts downs its reductions and yields when it is time. The programmer can call erlang:yield() to do this before time if he wants. Calling something like receive will also yield. I guess it comes down to definitions, but since Erlang is said to have only soft real-time (as opposed to hard real-time) properties, this makes more sense. The schedulers doesn't give any guarantees as to when a process gets to run and the scheduler never really preempts a process. Again, to my knowledge.

Great article though!

hyperpape• 10 years ago

I was also confused by this phrasing. The only thing I could think of was that from the programmer's perspective, it's not cooperative (you don't have to call yield), but that's also true of threads.

I see people down thread repeating this claim. Can anyone explain how the scheduler is preemptive?

sbalea• 10 years ago

You are absolutely correct, the schedulers never preempt a process in the classical sense, but from an Erlang code perspective, the system behaves as if it would be preemptive. The schedulers are actually running a byte code interpreter and the interpreter will switch processes once a certain number of reductions is reached (or the code yields). This breaks down when running native code in the form of NIFs - it is entirely possible to block a scheduler indefinitely while running native code.

sasa5552• 10 years ago

I like to think of it as preemptive because (putting NIFs aside) it's not possible for a single process to completely block the entire scheduler, even if it ends up in an infinite CPU bound loop.

So in other words, a task (i.e. process) doesn't have to cooperate with others. It can do whatever it wants, be it I/O or CPU bound, and doesn't have to yield explicitly. Due to aggressive "preemption" it won't block other pending tasks significantly.

In contrast, if this is not guaranteed (which is not in most other Erlang "alternatives" I've looked at), then a task might accidentally paralyse large portion of the system.

hamidreza-s3• 10 years ago

That's a good pint! I also can say that it is a matter of perspective. From an Erlang developer point of view, the processes are executing in a preemptive manner without telling them when to yield. The important factor is the reduction limit of the actors as well as being reentrant. It guarantees that each actor will sure yield even if it is still running and have something to do, and will be selected whenever scheduler select them again for execution.

BuckRogers• 10 years ago

I've been into Erlang for about 5 years now and once I found it, I never looked back. They really nailed it and many competitive technologies like Go don't have a preemptive scheduler yet. I wish Elixir had a Pythonic syntax, but I'm all for anything BEAM gaining steam and have enjoyed using Elixir.

I personally feel that if someone made a Pythonic version of Elixir, say "Hydra", it would take over the world. If anyone starts such a project, look me up!

eggy• 10 years ago

I second that. Elixir is great, but I was not a Ruby programmer, and prefer other syntax. I've tried LFE, and I love Lisp, but it has not gained traction. Perhaps I'll look into Luerl. Both LFE and Luerl were created by Robert Virding one of the original workers on Erlang with Joe Armstrong. I just might go with straight Erlang. I don't think the syntax is that bad.

Erlang Scheduler Details and Why They Matter

Related Stories