The Erlang Challenge

http://yarivsblog.com/articles/2008/02/08/the-erlang-challenge/

by tb • 18 years ago

21 22 18 years ago

I'm guessing that very few people here on news.yc are qualified to compose the Cobol Challenge: the problem for which Cobol is uniquely qualified to provide the elegant solution.

Of course, the budget and the legacy code will have to be a very explicit part of that problem.

Meanwhile, this Erlang Challenge only tends to emphasize my impression -- which is just a stereotype, really -- that Erlang is certainly not the hundred-year language: it's a special-purpose language that is brilliant at a narrow class of problem. I'm convinced that, if I have some process that I want to spread across half a million processor cores, I should ask my company's Erlang guru to work on it (because, come on, if you can afford half a million processor cores you can afford at least one Erlang guru). What I'm not yet convinced of is that a world with N programmers who work in general-purpose languages needs more than (0.005)N Erlang gurus.

The challenge of the Erlang advocate is not to convince me, over and over, that Erlang wins its class; the challenge is to convince me that Erlang's class of problem is so important to my life that I should study Erlang rather than vascular surgery, or television repair, or other obscure technical skills that I don't know that much about.

lg4• 18 years ago

Well the PS3 has 9 cores, or something like that. Don't you think that says something about the computers we'll all be using, and by extension, the parallel-processing capabilities we'll have to take advantage of in the future? Personally, I'd rather not learn Erlang either, but I hope that this sort of easy parallel processing becomes one of those popular features that every language will implement sooner or later. Hm, I wonder if it should be part of Arc's core...

zach• 18 years ago

The "eventually everything will be massively parallel" argument is not very convincing. The Be guys made this argument 15 years ago and we're only now getting to where they were back then (2/4 processor cores). Consumers aren't that excited by more cores, and developers still don't know what to do with them. So there is massive investment in advancing the status quo.

Sony thought they could impose the PS3's multiprocessing regime on game developers through market force. But developers found the programming overhead unmanageable, which led to a less-than-compelling lineup, which failed to interest consumers, which means that now developers just work on the Xbox 360 version and port to PS3.

I was offered a job porting an existing game engine to the PS3 under the gun (the game had to ship in six months) and I would not touch it with a ten-foot pole.

iamwil2• 18 years ago

The follow-up question then would be how many existing problems can be converted to the type that Erlang can solve well? And how many problems previously impractical are now practical to solve? So if it ends up to be a bigger class than you thought, you may well be limiting the number problems that you can solve easily and practically out of the ones that will be important in the future.

I think it's important because it allows a program to easier to scale out rather than scale up. We'd want speedups in this way because CPUs are becoming multi-cores, and to take advantage of them one will have to write some type of parallel program, since it's been difficult to fully automate parallelizing serial programs.

In addition, parallelized systems (not just erlang) can be more fault tolerant and can fail more gracefully (or hobble along, if you'd like to call it that). Sensor networks are one example. Instead of a single radar to detect the environment, you many sensors out there, and they network themselves and report what they see/hear. If some of them gets destroyed that's ok, because the system's still functioning with less sensors.

So what concrete examples of problems fall into this class Erlang is good at? The obvious ones are the embarrassingly parallelized algorithms, like genetic algorithms, neural networks, 3d rendering, and if I'm not mistaken, FFTs. Indexing web pages is another. That said, there are problems that are inherently serial. It takes 9 months to begat a baby, no matter how many pregnant women you have.

Ruby and Haxe language writers are both implementing the actor model like Erlang, if that's any indication of how important they think it is. While I don't think Erlang will be the 100 year language, the ideas by which it's a poster child will reverberate in the descendant languages for a long time to come.

randomhack• 18 years ago

well I have never understood the "use erlang because CPUs are going multicore" argument. See if I write a program in C (which for some mysterious reason doesnt use threads) and run it on a single core and it runs in 1 sec. If an equivalent erlang program runs in 10 seconds on 1 core, then I will need at least 10 cores before it beats C. That is assuming 100% of my application/algorithm is parallel.

While erlang is a nice language, saying that it runs faster on multicores is just a lame excuse.

yariv5• 18 years ago

Just because Erlang is very good at solving a certain class of problems, doesn't mean it's inferior at solving general computing problems.

The Erlang challenge highlights what Erlang is good at, but it's not meant to convince you that Erlang is good at everything. I've never claimed that this challenge proves you should use Erlang to solve any problem that comes your way.

lg4• 18 years ago

You said it. I checked out the Erlang tutorial just now, and apparently you can't use user-defined functions in the test-expression of a conditional, to prevent people from putting side-effecty functions in there? Now maybe there's some good reason for that, that has to do with the mysteries of handling concurrency or something, but it gives me the prima-facie feeling of java for the fp crowd.

mechanical_fish2• 18 years ago

Well, okay, it sounds like I have therefore interpreted the Erlang Challenge correctly: it highlights what Erlang is good at. And that's fine. There's nothing wrong with ass-kicking special-purpose programming languages, and I look forward to the day when I will actually spend ten minutes learning Erlang and be able to start discussing it instead of just marveling at it from afar.

The thing is, the Arc Challenges (I presume there will be others...) are explicitly intended to convince us that Arc is an awesome language for solving general computing problems. (Of course, they're also tools for designing Arc: If Arc proves unconvincingly awesome at any point, the plan is to twiddle its design knobs until the awesomeness returns. We will see how it goes.)

When you adopted the Arc Challenge's title and style, it left me wondering if you were aiming at the same goal. If you were, I wanted to invite you to shoot again, because you kinda missed. However, it seems that "convincing me that Erlang is good at everything" was never your goal at all, so I'm sorry to have interpreted it that way. (My interpretation may have been unduly influenced by the original title of the submission...)

yariv5• 18 years ago

I must say that I am taken aback by the title you used to link to my article. I did not mean it as a snide response to the Arc challenge. I thought the Arc challenge was pretty cool, and I replied to it with an Erlang implementation -- you can see it here http://arclanguage.org/item?id=722. If it come across as snide, then I didn't communicate it as I had intended. The reason I said it's in the "spirit" of the Arc challenge is that the Arc challenge highlights what Arc is good at, and the Erlang challenge highlights what Erlang is good at.

neilk2• 18 years ago

Interesting. I thought you were making a sly joke too.

I guess this just proves people read their biases into whatever they encounter. Like many, I thought the Arc challenge amounted to a test of how much a language was like Arc (guess what, Arc wins). So I assumed your post had to be satire.

And -- maybe the requirement of spawning a million threads doesn't seem like a cruel joke to you, but it is to anyone who has to work in a language other than Erlang.

yariv5• 18 years ago

Apparently, I should have been more sensitive to the way people would interpret this challenge and picked my words differently.

A few other languages, including Scala and Stackless Python, have lightweight concurrency capabilities. AFAIK those languages do allow you to spawn millions of processes (or actors), but they don't handle garbage collection, scheduling, IO, distributed programming and native code interfacing as well as Erlang does.

pg• 18 years ago

I lopped off the extra editorial remarks on the title. It didn't seem particularly snide to me.

yariv5• 18 years ago

Thanks :)

The Erlang Challenge

Related Stories