Erlang at Basho, Five Years Later

http://basho.com/erlang-at-basho-five-years-later/

by bsg75 • 13 years ago

188 47 13 years ago

bsg75• 13 years ago

"While it’s theoretically a nice bonus for someone to bring knowledge of all the tools we use, we’ve hired a significant number of engineers that had no prior Erlang experience and they’ve worked out well."

Nice to hear a company understand that talent and abiliaty does not equate to n-years using a specific toolset ('not “expert hammer wielder”').

cinbun8• 13 years ago

I like that they focus on the experience related to DBs and distributed systems instead of hiring engineers that can program in X. You can pick up a language relatively faster since you'll need to build the other skill slowly with real world experience.

xaritas• 13 years ago

I found Erlang to be much easier to digest the second time through. The first time I looked at Erlang, I knew of its benefits, but didn't have much experience with functional programming. The unusual syntax and unfamiliar paradigm led me to put it aside.

Later, after I learned me some foldr and flatMaps (mostly thanks to Martin Odersky's Scala/FP course on Coursera), I revisited the language when I saw an article here about an Erlang based CMS (http://zotonic.com/). It seemed pretty cool, I host some websites for local businesses, so I set up it to evaluate it and realized that I now had no trouble at all with Erlang and the syntax made more sense. Searching around for some other web stuff I found Chicago Boss (http://chicagoboss.org/), a Rails-inspired web framework with ridiculously easy Comet/WebSocket/"real time" support.

Now I'm really excited about it and I can't wait to do some cool things with it.

aeden3• 13 years ago

My experience was quite similar, although it took me three tries to get it. I think it was http://learnyousomeerlang.com that finally helped me understand enough to really start to enjoy Erlang.

Shish2k• 13 years ago

I've also found similar - I wonder if there's something about functional programming that means the language syntax /has/ to be obtuse...

(Similar with Haskell too -- learning that at university, I loved the concepts, and I'm sure it's technically great, but the APIs seem almost deliberately horrible; one-character variable names, array append and prepend being called "cdr" and "car" or something like that, stuff like that all throughout the standard library and tutorials. I don't care that there's a historic reason for that based in the PDP's Lisp-to-assembly compiler, if the function's job is to append it should be called "append()" -_-;; </rant>)

mosburger• 13 years ago

This is sort of similar to pg's "Python Paradox" essay. http://www.paulgraham.com/pypar.html

It's a slightly different conclusion (use best tool for the job, earn respect of developers vs. use esoteric tool for the job, attract developers eager to learn something off-the beaten-path), but with similar ends. Using an offbeat language can have its benefits.

roncohen• 13 years ago

I do believe that Erlang is the right tool for the job when you're building a distributed database, especially for the distribution part. For a company like Basho it's understandably a good investment to take expert distributed systems/database engineers and let them take their time learning Erlang.

If, however, you, like most of us, are not building a distributed database, think carefully before you choose Erlang, especially if you're in a startup.

Most startups don't have a distributed database as their core technology, and for many startups, it's important that developers can push production code from day one.

derefr2• 13 years ago

I think the important question for HN, here, is: what "app ideas" map, in the end, to "having a distributed database as core technology"?

If you're building the next Facebook, is that, fundamentally, a distributed database?

If you're building an MMO, is that, fundamentally, a distributed database?

If you're building a payments platform, is that, fundamentally, a distributed database?

And so forth.

derefr2• 13 years ago

Jeez, and here I was hoping someone would have an answer ready to hand. Instead, I get 13 upvotes (at time of this writing) from people who are presumably wondering the same thing. I guess this might be something worth doing a bit of research and writing an article about. :)

oneweekwonder• 13 years ago

What distributed application can not be "philosofed" into essentially being a distributed database, and I feel like I'm just repeating your question.

With that in mind, I believe it boils down to where is the logic located, with the data, or not. Because a lot of "databases" are just used as a datastore.

So let me try to create some scenarios:

1. To push out a product(Startup/Prototype/POC): Develop in the technology you know, and let the stake holders know you are acquiring Technical Debt. After startup/prototype product is a success. Explain to stake holders why you need to move to a better technology stack that supports distributed systems "better".

2. If you have a design or product, develop directly in a distributed technology stack.

It boils down to use the correct tool for the correct job, but you will mostly only know if you used the correct tool later in the development of the product.

davidw• 13 years ago

It's a pity they didn't talk about Erlang's strengths and weaknesses in a bit more detail.

I like Erlang a lot, and pushed to use it in a project I'm working on - it is the right tool for the job in this case.

But programming languages aren't like hammers or saws. You can do a ton of things with any one of them. This particular project probably could have been made to work one way or the other with Java, Javascript, Tcl, Ruby, Python, Perl, PHP, Haskell or any number of other things. And this is, in some ways, a problem for Erlang. It's really, really good at a few things, ok at a number of things, but feels a bit spotty for other things where other languages have something in place. At least that is my impression.

eredmond• 13 years ago

I think the biggest pluses for Erlang as a language are that fault-tolerance (Erlang "let it crash" + OTP supervision trees) and distribution of tasks (process interact via message passing, be it across procs or nodes) are built-in language features. They aren't libraries or afterthoughts. This subtle shift manifests itself in all Erlang code. There's rarely a fear that some esoteric library you want to use has a memory leak because it's keeping state in some static hashtable, or that semaphores are used incorrectly causing a race condition.

JPKab2• 13 years ago

Quick question: I don't know much about Erlang compared to other functional languages, but how is Erlang superior to another language which claims to have some of these traits, such as Go?

plainOldText• 13 years ago

"It's a pity they didn't talk about Erlang's strengths and weaknesses in a bit more detail."

I feel the exact same way. Erlang is a very peculiar language; quite different than, pretty much everything there is out there. More details about why Erlang is/is not a good choice for a specific job are always very welcomed.

aeden3• 13 years ago

I'll be presenting on using Erlang for an authoritative DNS server at StrangeLoop this year (https://thestrangeloop.com/sessions/erlang-for-authoritative...). If you can make it hopefully you'll get some good tidbits on why I selected Erlang and what kinds of benefits and drawbacks became evident as the system developed. If not then perhaps the sessions will be recorded (I believe they are supposed to be, but I'm not 100%).

granitepail3• 13 years ago

I saw the article as more of a reaction to all of the technical (and often times inane) blog posts about why company x uses technology y.

The main points I took away from the article were:

- They did their homework, and - They knew that, even though it is not the most widely known technology stack, they were seeking the kind of engineer who would not only be able to pick it up, but understand their rational for choosing it

And it seems as though they've been successful! As engineers, most of us are interested in the many interesting technologies available to us, so much so that, when examining a company like Basho, many of us ask "Why Erlang?" before we ask "What are you making and why?"

Without putting words in their mouth, it is my impression that Basho sought to create an incredibly powerful and easily scalable database. They chose Erlang because it was, in their opinion, the tool for the job. As I see it, 10gen got the ball rolling towards approachable scaling/development and now Basho and the team behind RethinkDB are trying to improve by creating systems free of MongoDB's many limitations.

leif• 13 years ago

What do you see as MongoDB's limitations, and are any/most of them solved by (the product I work on,) TokuMX[1]?

[1] http://www.tokutek.com/products/tokumx-for-mongodb/

coffeemug• 13 years ago

I'm the founder @ Rethink. Just to clarify, we don't see approachability and scalability as a dichotomy, we think one could do both really well. There are of course some tradeoffs involved (which is why the [awesome] team @ Basho chose a dynamo-style system and we chose a master-slave type system), but in the grand scheme of things, high scale and approachability are definitely possible and we (and possibly others) are going to make it happen.

outworlder• 13 years ago

The only thing I didn't like about Erlang is its syntax - but I am sure I could adapt. I just don't like too much syntax in general, one reason why I like Scheme.

Now, Erlang's runtime seems fantastic.

I wonder if they (Basho) investigated Elixir recently and their opinion of it.

seancribbs• 13 years ago

Elixir has some good things about it (hygienic macros, protocols, and so on) but the syntax is not enough by itself to make us switch. "Ugh, Erlang syntax sucks" is a refrain heard around the Internet, but it has not hindered us from getting good engineers who are happy to develop with it. I daresay that the syntax is the least of your problems when learning Erlang; the message-passing concurrency and fault-tolerance models, and functional style are more difficult to understand. Those problems don't go away when you use Elixir instead.

If you want something Scheme-like on the Erlang runtime, there's LFE (Lisp-Flavored Erlang): https://github.com/rvirding/lfe

tiglionabbit• 13 years ago

Neat! Does it come in Haskell flavor too?

rdtsc2• 13 years ago

I daresay that the syntax is the least of your problems when learning Erlang; the message-passing concurrency and fault-tolerance models, and functional style are more difficult to understand.

That's my feel as well. I hear tirades about commas and periods. What are they going to do when they hit a net-split.

Bashing Erlang and stopping at syntax is like bashing a new battle tank because it doesn't have a leather interior. Yeah it is nice if it had leather seats and a mini fridge but if that is really the main criteria used in picking it, one has to wonder...

waffle_ss• 13 years ago

If you like Lisp and Erlang's runtime, then check out Joxa: http://joxa.org/

peteretep• 13 years ago

Do other people have much experience cross-training people? I am struggling to find decent Senior Perl developers at the moment in London. I am open to the idea of hiring a Senior X Dev and cross-training them ... I'm just not sure how well it'd work out.

JanezStupar• 13 years ago

I would rather have a wise and experienced developer X than a developer with Y experience in Z.

As long as you are hiring people with skills in solving similar problems to the ones you are having or are anticipating on having.

Edit: However that may not work for you. I think that a senior Perl developer would be more than competent in any language you point towards. While the opposite may hardly hold true. The presumption is that one believes that a senior Perl developer really exists.

hernan604• 13 years ago

There are some really good perl coders that are not working with perl because its hard to get a perl job. This situation happens in brasil. Consider telecomute =)

seldo• 13 years ago

I think the most important point here is that you should hire engineers from your domain, but not necessarily your exact technology stack -- in their case it's simple because almost no engineers know Erlang well, so they have to hire "database people" and convert them, but I think it's more generally applicable.

Personally, I would much prefer to hire a "web person" regardless of the stack they're used to than, say, a "Java person" who has never written a website. But even more specifically, I would look to find somebody whose interests are in my domain, as they will always have better ideas and motivation than a strictly mercenary engineer.

smegel• 13 years ago

Not to mention the fact that most experts will spend their time on concept on design rather than code cutting. I think Rob Pike said it well when he said he didn't event have commit privileges at Google while working on Go because he still coded in K&R C.

fusiongyro• 13 years ago

The large database company example is especially telling. The company I work for chose Java mainly for the benefit of a large talent pool, but we have great difficulty hiring people. I think it's because we're in the middle of nowhere and don't pay Silicon Valley wages. I think if we had chosen a more "offbeat" language (to borrow mosburger's adjective) we would have had a novelty factor that would increase interest in the job. After all, we're doing interesting stuff, but our technology decisions make us seem very dull.

Edit: Please note I don't have the authority to make a technology change and see what the effect would be on our hiring, despite the good ideas of those replying to me.

geoka9• 13 years ago

You could start migrating to Scala, which is simple with a Java-based stack. Scala is sufficiently hip at the moment to attract programming enthusiasts.

mixedbit2• 13 years ago

Similarly Clojure is JVM based and it draws interest of many Java gurus, because they do not need to learn the platform from scratch (the Java library is available).

rch• 13 years ago

This is a phenomena worth understanding, if you don't already:

"I had an entertaining and ironic conversation about this recently with a manager at a large database company. He explained to me that we had clearly made the wrong choice, and that we should have chosen Java (like his team) in order to expand the recruiting pool. Then, without breaking a stride, he asked if I could send any candidates his way, to fill his gaps in finding talented people."

josh2600• 13 years ago

We use Erlang at 2600hz and I feel like this is an Echo Chamber of the kinds of conversations we have around the table. Erlang is a beast, but if you're doing hard work it's more about the idea than the tools.

Lots of wisdom on this blog post. The truth is that if you're passionate about doing something hard, and communicate that passion, people are drawn to it.

benmmurphy• 13 years ago

it would be interesting if they covered some of the technical problems they've had with using erlang in a database. writing a db in a VM language like erlang or java you inevitably end up fighting with the VM for some workloads. i noticed that recently one of their engineers tried to submit a patch to erlang to force it to not put schedulers to sleep when there is work to be done (http://erlang.org/pipermail/erlang-patches/2013-June/004109....). it had an interesting flag name '+zdnfgtse' which looks like z-do-not-f-go-to-sleep-ever :)

Erlang at Basho, Five Years Later

Related Stories