Big changes to Erlang

http://joearms.github.io/2014/02/01/big-changes-to-erlang.html

by waffle_ss • 12 years ago

325 133 12 years ago

joearms.github.io

sdegutis• 12 years ago

It seems odd that we haven't standardized on a name for maps/tables/dictionaries/associative-arrays/hashes.

Well, hashes in particular implies a very specific kind of map/table (one that uses hashing). I think the most common non-implementation-dependent names are maps, dict(ionarie)s, tables, and associative arrays.

logicallee2• 12 years ago

the reason it's in OP's list is that that's just what Perl calls associative arrays. (http://en.wikibooks.org/wiki/Perl_Programming/Hash_Variables)

Hence it's literally the exact same thing as a dictionary, but is named a hash in Perl.

kevinmchugh• 12 years ago

And in JS, it's an object.

Figuring out how to store arbitrary key-value pairs should be one of the first steps in learning a new language, given that every language has a different name for it.

Associative Array is probably my least favorite name for it, since it implies arrays aren't associative. It was a clojure book that first made me realize an array is a map that only allows ints for keys.

Skinney• 12 years ago

Actually an object in JS stores string keys only, if you try to insert something else it's auto-converted to strings. Array is an object wich stores integer keys. You don't really have a datastructure that stores arbitrary keys in JS (when not including third party libraries). This will be changed with Ecmascript Harmony though (dedicated Map type)

kilon• 12 years ago

Its also an object in Python , Pharo and most likely Ruby too.

banachtarski3• 12 years ago

Well, there is the distinction between a sparse array and a contiguous one, both of which are technically associative. I associate the former with the term "associative array" though.

jaredsohn• 12 years ago

an array is a map that only allows ints for keys

Actually, it only allows nonnegative ints for keys.

Also, it has the added undesirable property of requiring memory proportional to the largest key value, rather than number of keys.

colanderman13• 12 years ago

Well, most of those really conflate two things: static heterogeneous "maps", and dynamic homogeneous maps. The former being a structure whose keys are known at compile time (static) and whose values may be of different types (heterogeneous); the latter being a structure whose keys are not known until run time (dynamic) and whose values must all be of the same type (homogeneous).

Most languages conflate these, because they kind of look the same, but they are entirely different use cases, with different use patterns, permit different optimizations, and their generalization (a dynamic heterogenous structure) is pretty useless.

The relational model [1], which has been around since 1969, does distinguish these. The static heterogeneous structure is called a "tuple", and the dynamic homogeneous structure is called a "relation".

If only language designers would stop conflating the implementation of these structures with how they are used, we could drop the cringeworthy names "associative array" and "hash", stop calling tuples "maps" (which is a specific kind of relation), and stop calling relations "tables" (which are two-dimensional arrays, which are another kind of relation).

Erlang at least as of R17 has mostly kept this distinction (Erlang maps mostly support the static heterogeneous use case). However the majority of users are keen to jump on the "maps" bandwagon as a way to get syntax support for dynamic homogeneous use (which makes utterly no sense to me, as most of the syntax is for pattern matching and construction, which makes little sense in the dynamic case!). I am glad R17 does not include many of the proposals for such syntax that I've seen on the mailing list; and instead sticks closely to the original static heterogeneous design proposed by Richard O'Keefe.

[1] http://en.wikipedia.org/wiki/Relational_model

Morgawr2• 12 years ago

I agree with what you said but I don't see how a heterogeneous dynamic structure is "pretty useless" though

jeffdavis3• 12 years ago

I don't follow how a relation is a "dynamic homogenous map". Can you elaborate?

brickcap• 12 years ago

Maps look really interesting. They can also be saved in mnesia right?

jlouis2• 12 years ago

Yes, and passed via distribution

urbit• 12 years ago

"This trick is well known to old-style functional programmers, they waffle on about Y combinators and eat this stuff for breakfast, but it’s the kind of stuff that gives functional programming a bad name. Try explaining this to first year students who had a heavy night out at the pub the evening before."

Priceless (and completely true). I really think this attitude is why Erlang is such a success.

thirsteh2• 12 years ago

I think there's definitely a rule against dismissing Y combinators on news.ycombinator.com.

derefr2• 12 years ago

They're important in CS theory, sure, but you sure don't want to encounter one in raw form lurking in a regular codebase. (Just like you wouldn't want to encounter monadic IO in raw >>= form.)

tel4• 12 years ago

Meh, the Y combinator is a funny hack. A lot of people waffle on about it because they're blown away by how crazy untyped lambda calculus is. There are all kinds of ways to introduce recursion into a language, though. It's simply interesting that you might get it on accident in a language will as little restriction as UTLC.

colanderman13• 12 years ago

Oh don't worry, you can make a Y combinator in a typed lambda calculus too, it's just a little more involved: http://en.wikipedia.org/wiki/Fixed-point_combinator#Type_for...

dschiptsov• 12 years ago

There are two side notes. 1. Should we really be constrained by frequent pub visitors when teaching CS? Adapting to stupidity (OK, mediocrity) caused Java and switch from Scheme to Python for intro courses at MIT (with great help of mr. Guttag, I suppose). 2. Should we ever hire a person who couldn't get what Y Combinator is or not being able to write one in Scheme?

Update: Python is heavily used in Biology and related research (so they said) and transition at MIT seemed like reasonable, but I still believe that Scheme shall be the language for serious introductory courses, like classic Berkeley CS61a was. Now PLT/Racket folks are doing great job of introducing proper concepts and developing "good habits" to students.

portmanteaufu2• 12 years ago

Whoa. This makes the language feel much more approachable. I love that they've provided both an "upsert" and a semantically distinct "update" for their maps. Working without upsert is painful, but the problem of accidentally inserting when you meant to update is an irritating pitfall. They've really picked a good middle way here.

Does anyone know how often Learn You Some Erlang is updated? I'd definitely take another pass at the language with these features in place. Kudos, team!

sandal• 12 years ago

Not a direct answer to your question but Joe's "Programming Erlang" book has been fully updated, and includes a chapter on the new map feature.

jasonlotito• 12 years ago

As someone who owns both of his PragProg books, the second edition is also a big improvement over the first. If you ever tried learning Erlang, and didn't stick with it, but still want to learn it, I highly recommend the second edition.

mononcqc4• 12 years ago

Author here. I haven't yet updated it, but plan to eventually update the website with these features. The thing is a lot of it won't need to change majorly.

Maps should be a replacement of data structures like dicts and gb_trees, but I personally do not see them as a replacement of records within a module, where I feel their restrictiveness is welcome, for two main reasons:

1. especially to crash early in live code upgrades, despite, I'm sure, a lot of people disagreeing with me.

2. The module isolation inherent to records makes people think at a protocol level and with their API much, much better than the common pattern of saying "screw it", sharing the state around, and breaking abstraction all over. I like how it constrains the programmer to think of what should be passed around in messages, and that maps may remove that "think hard" part of the problem.

Maps should be especially nice and enable more complex dictionary manipulations, nested key/val mapping, and so on, and in terseness of operations. More elegantly, they could be a decent fix to 'use ETS to optimize K/V operations', although they won't benefit from the same parallel access.

I plan to explain this and possibly revisit some code snippets from the book in an add-on chapter, and also show what I wouldn't change.

Regarding Funs, I probably will just add a section to the anonymous function part, and see if I ever used recursive anoynmous functions before. It's likely that I avoided them on purpose in the book and as such, won't need to add too much there.

Let me know if that sounds good to you.

sitkack• 12 years ago

I definitely agree on records forming the basis for a protocol. Maps do not serve the same purpose, I could see maps _used_ in a protocol but not the basis for it.

Long live records!

It would be nice if records became a concrete thing in Erlang.

colanderman13• 12 years ago

Maps should be a replacement of data structures like dicts and gb_trees

Why? Why do you need to pattern-match on a structure whose keys you don't know a priori? Why should the syntax prefer one map-like data structure over another?

Don't get me wrong, I love that maps are in the language now – as a replacement for records, where syntax and pattern matching make sense and are used all the time, and the data structure issue is much less sensitive. Please don't hamper their performance by forcing them to support the use case for which dicts and gb_trees are intended!

portmanteaufu2• 12 years ago

Interesting! I'm surprised to hear that maps aren't viewed as a replacement for records as Joe Armstrong (probably jokingly , I now realize) declared. Your explanation makes complete sense. Thanks for sharing your perspective, and thanks for maintaining LYSE!

NDizzle2• 12 years ago

This is great. I can attack the few chapters I skipped over in Joe's most recent book[1] that required R17.

[1] http://pragprog.com/book/jaerlang2/programming-erlang

fidotron• 12 years ago

I have to say, this is one of my favourite programming books, even though I've never been paid to write any Erlang at all. Just the insights into how to go about building reliable systems are inspiring and profoundly affected my work in other languages.

That and it's written in a brilliantly approachable style.

NDizzle2• 12 years ago

Yes, it's really good. Don't worry. I can't write any erlang either. I'm still in the try-hard phase.

Shish2k• 12 years ago

Just the insights into how to go about building reliable systems are inspiring

I found that too; I wonder if the programming community would be better off if more people were pushed towards Erlang? Similar to how every programmer should know C, so that when they write python they have some idea what's happening under the hood, it would be good for every programmer to know Erlang/OTP so they learn how large-scale reliable systems should work?

Buttons840• 12 years ago

Thank you for mentioning this book. I intend to learn Erlang after I finish studying Haskell, and I'll probably use this book because of your recommendation.

dipthegeezer• 12 years ago

Brilliant Erlang gets even better.

talklittle• 12 years ago

EEP-43 [0] standards draft, with an extensive overview of new Maps features, plenty of examples, and discussion on a few different syntax proposals they had.

(Is there any newer official documentation for maps? I couldn't find anything.)

[0] http://www.erlang.org/eeps/eep-0043.html

gordonguthrie3• 12 years ago

I am in the process of writing an OTP-ish dialect of Erlang that compiles to Javascript and has its own runtime environment (http://luvv.ie/mission.html) so maps is a more work for me, so I am a bit yay!/sigh!...

The thing that I am less clear about with maps is their relationship with Mnesia tables going forward. Mnesia tables take their schema from records. I suspect we will go down the route of having records that contain 'privileged' fields (effectively ones you can have indexes on) and field that contain maps which allow to 'extend' the schema without performing difficult state transforms.

It will certainly help when you need to add a field to a record and thread that change through a running system without stopping it.

rdtsc9• 12 years ago

Hey, luvvie looks awesome. Thanks for all your great work. I am following it. Sorry don't have much insight as to what happens with Mnesia. Just wanted to thank you for your work.

gordonguthrie3• 12 years ago

Cool, just slogging away, its is still early doors...

kibwen• 12 years ago

As someone who's completely foreign to Erlang, can anyone explain the rationale behind replacing records with maps? At a first glance it seems as though records are nominal types and maps are structural types, and if true that would suggest to me that these are complementary features rather than supplementary ones.

masklinn5• 12 years ago

As someone who's completely foreign to Erlang, can anyone explain the rationale behind replacing records with maps?

Simplicity and flexibility. Erlang records have to be declared and they're nothing more than syntactic sugar for tuples.

At a first glance it seems as though records are nominal types

They're not, not in the way you'd usually expect anyway. They're little more than macros for accessing tuple fields by name:

    -module(test).

    -record(foo, {bar, baz}).

    main(_) ->
        R = #foo{bar=1, baz=2},
        io:format("~w~n", [R]).

will print:

    {foo,1,2}

the record information does not exist at runtime. In fact, you can just create the corresponding tuple and tell Erlang to interpret it as a record:

    S = {foo, 2, 3},
    T = S#foo{bar=3},
    io:format("~w ~w~n", [S, T]).

will generate no warning from either erlang itself or dialyzer, and will print

    {foo,2,3} {foo,3,3}

And although dialyzer (erlang's "type checker") can use records as type specs, it will be updated to handle map specs[0], allowing for the exact same static expressivity since you can define a map with explicitly specified keys e.g.

    -type foo() :: #{ 'status' => 'update' | 'keep', 'c' => integer() }

[0] http://www.erlang.org/eeps/eep-0043.html section "Dialyzer and Type specification", sadly not directly linkable.

lucaspiller• 12 years ago

Having the record not exist at runtime can lead to issues if you serialise you records using term_to_binary. If the format of the record changes, old records can no longer be accessed.

    1> rd(person, {name, email}).
    person
    2> P = #person{name = 'John', email = 'john@example.com'}.
    #person{name = 'John',email = 'john@example.com'}
    3> P#person.email.
    'john@example.com'
    4> rd(person, {name, email, age}).
    person
    5> P#person.email.
    ** exception error: {badrecord,person}

lkrubner• 12 years ago

I love this:

"Key := Val Updates an existing Key The key must be present in the old map. If Key is not in the old map Erlang will complain loudly. Using two operations instead of one has several advantages: A spelling mistake cannot accidentally introduce a new key"

Lord knows how many times I accidentally created a new key with a spelling typo, and then later when I wanted to retrieve the data, I was like "What the hell is wrong? I know I put data in that key, so why is nothing coming back? I guess I have discovered a rare bug in the compiler that no one has ever discovered before."

bjourne2• 12 years ago

To counteract such bugs I usually try to use as short names as possible. I mean, there is no point in writing productionDatabase when prodDb works just as well. It has several advantages but one of them is that the latter is much less likely to be misspelled.

laureny2• 12 years ago

I'd say the latter is much more likely to be misspelled since you abbreviated it arbitrarily. It's easy to write it prodDB or proddb, or was is productionDb?

Using actual English names actually minimizes the risks of mistakes.

And of course, using a statically typed language makes all these concerns moot since the compiler won't let you make these mistakes in the first place.

draegtun2• 12 years ago

I prefer to counteract these potential bugs by locking the keys of a hash.

For eg. in Perl:

  use Hash::Util 'lock_keys';

  my %hash = (key1 => 'foo', key2 => 'bar', key3 => 'baz');  
  lock_keys %hash;

Now any attempts to lookup or amend a key that doesn't exist produces a runtime error...

  $hash{key1} = "FOO";   # updates fine
  $hash{key4} = "XXX";   # throws error

rdtsc9• 12 years ago

I also like how such rather large changes are being introduced in a language that has been in production for many decades. That is pretty remarkable.

davidw3• 12 years ago

I think Erlang is sort of unique in this way. Many languages go through a lot of changes when they are not all that popular, and then blow up in popularity and finally have a lot of users who are wary of Big Changes. Ruby, for instance, was around for years and years before it started to get popular because of Rails. Python, too, was a product of the early 90ies, but as late as some Linux conference in... '99 or 2000 or thereabouts, I recall sitting at a BoF table with Guido and just a few other people, and the language not being that big a deal.

Erlang is still not really all that popular, but has always had a lot of Important Code in production thanks to Ericsson, which has made it more difficult to change even though there are some things that even the creators acknowledge as not particularly elegant.

masklinn5• 12 years ago

Rather large change? As far as I understand it's just adding a new built-in collection and literal form for it (oh, and the pattern-matching machinery for it obviously).

Python added a set builtin in 2.4 and a literal syntax in 2.7, Objective-C added NSNumber, NSArray and NSDictionary literals only last year.

Uchikoma• 12 years ago

I don't want to sound pessimistic, but adding a map structure to an language sounds rather small to me.

cdelsolar• 12 years ago

many = 2 ?

judk2• 12 years ago

That syntax for that purpose was totally stolen from Go. Nice to see the cross pollination going in both directions.

BarkMore• 12 years ago

Go's := (short variable declaration) is closer to Erlang's => (introduce a new key) then it is to Erlang's := (update an existing key).

ams6110• 12 years ago

But now the culprit typo is typing -> when you meant :=

Maybe less likely, but if you've been writing a lot of code using -> and then in one case need := I can see it happening quite easily.

Tuna-Fish• 12 years ago

The system would complain loudly in an error message with a line number.

The problem with accidental key typos is that at their worst they are completely silent, and if the value you intend to replace was another valid value, finding the bug might not be easy.

varg• 12 years ago

Great! Maps looks really nice :) Will mnesia be extended to us maps as an alternative to records?

Cushman• 12 years ago

These are great changes, and a reminder that I must find a reason to write more Erlang.

For the record, one of these is another thing JavaScript got right, sort of, with named function expressions:

  setTimeout(function innerName() {
    if (shouldRecurse())
      innerName();
  });
  typeof innerName; // undefined

I don't think I've ever seen anyone actually use this in production code, of course. And you're free to have your own issues with function declarations and named function expressions having the exact same syntax, which causes a big problem in IE<=8 that is probably responsible for this feature not being in common use (and omitted from CoffeeScript, alas).

masklinn5• 12 years ago

You can also do that in Clojure, since you'd usually self-recurse with (recur) and a function will create a recursion point.

    (set-timeout (fn []
      (if (should-recurse)
        (recur)))

Morgawr2• 12 years ago

You can also name fns in clojure

(map (fn whatever [a] (println a)) [1 2 3])

ps: this code is stupid but I'm too lazy to write useful clojure from my phone, you get the gist

draegtun2• 12 years ago

And you can also do that in Perl:

  set_timeout sub {
      __SUB__->() if should_recurse();
  };

NB. __SUB__ was added at Perl 5.16 and so requires either:

  use 5.016;  # or newer

  use feature 'current_sub';

colanderman13• 12 years ago

There's been lots of chatter about maps on the mailing list; but the named inline functions thing is totally out of the blue to me, and totally welcome. Erlang does such a great job of fixing everything that OCaml almost but didn't quite get right.

nox_6• 12 years ago

Implementor here. Richard O'Keefe deserves all the credit as he had the idea and suggested the syntax (EEP 37). I just noticed that it was easy to implement it through let rec expressions in Core Erlang. No VM changes were required.

siraaj• 12 years ago

Could you elaborate on what you feel OCaml didn't quite get right WRT recursive definitions?

Groxx• 12 years ago

If we now build a large list of objects, all with the same set of keys, then they can share a single key descriptor. This means that asymptotically maps will be as efficient in terms of storage as records.

Credit for this idea comes from Richard O'Keefe who pointed this out years ago. I don’t think any other programming language does this, but I might be wrong.

That is a nice language feature to enforce, I love it :) This is however basically the same thing that V8 / most Javascript runtimes do under the hood[1]. Essentially, as you construct objects and add/remove keys, you back them in the engine with more-efficient shared objects that have the same structure. Add a key D and it moves from <shared structure with A,B,C> to <shared structure with A,B,C,D>. Since most code ends up doing the same kind of operations on a bunch of similar bits of data, you can save a lot of compute-time by assuming that and having a slower fallback when it fails (like using an actual dictionary instead of a Struct-like thing).

That said, Javascript has zero enforcement for this, and the runtimes may have already shifted to something different. They are quite different beasts. Just pointing out that the idea has been around.

[1] it has been a while since I've looked closely, and I may be mistaken. Call it 95% certainty.

leakybucket• 12 years ago

I've got a passing interest in Erlang, and also saw the existence of Elixir http://elixir-lang.org/ , which uses the Erlang VM. Could someone from either language's community comment on if they see a strong future for Elixir?

peregrine• 12 years ago

Think of elixir as a nicer and more approachable erlang. It compiles to the same byte code. It's very young but I think once it hits v1 it will have a very bright future I think.

lostcolony4• 12 years ago

Well, how about Joe's thoughts? - http://joearms.github.io/2013/05/31/a-week-with-elixir.html

rdtsc9• 12 years ago

It runs on Erlang's VM. So there are chances if you pick one up or another you'll be able to at some level inter-operate.

It is up to you, try one and another one. Some people have big issue with Erlang's syntax. I actually kind of like it, so I prefer Erlang.

Others find Elixir more approachable, it reminds them of Ruby for example. Elixir has macros so you can do some nifty things with them. Those kind of blow my mind when I read them so I am afraid I would get too "clever" with them.

You can call Erlang from Elixir fairly easily so you can take advantage of libraries you find for Erlang.

So kind of up to you. Whatever you like or whatever gets you more productive and interested.

nodivbyzero• 12 years ago

Way to go, Erlang! As a developer, I really appreciate it.

pointernil3• 12 years ago

Kudos to all ppl involved! Thanks for the effort.

Compared with maps adding a fun-call chaining operator like |> should be child's play, right? ;)

Seriously, what are the functional programmer's tricks to handle long function call chains in languages without the |> or similar operators?

Anyway, maps are a more important addition to the environment for sure. So thanks again.

momerath• 12 years ago

https://github.com/rabbitmq/erlando

nox_6• 12 years ago

An F#-like operator |> would be useless without a convenient partial application syntax. Macro-like operators aren't well received in Erlang, certainly not when they obfuscate what really is going on.

pointernil3• 12 years ago

really useless?

http://joearms.github.io/2013/05/31/a-week-with-elixir.html -> "The Pipe operator"

I really get several arguments against such "syntax sugar". I would not call it useless, though.

So how to handle/structure/avoid code with function call chains in absence of such operators?

oinksoft• 12 years ago

Thanks to Anthony Ramine for his hard work on the named `fun' feature, and to the OTP guys as always :^)

nox_6• 12 years ago

You're welcome.

riquito• 12 years ago

The EEP (Erlang Enhancement Proposal) about maps is really neat and a superb documentation.

http://www.erlang.org/eeps/eep-0043.html

minikomi• 12 years ago

This was a great read .. A very considered document.

kabdib• 12 years ago

The templating of the key descriptors is neat, but not original. NewtonScript did this in the early 90s (mostly in response to the severe memory pressure on the Apple Newton). I believe that Self did it, too (but those guys were on Sun workstations, and just showing off as far as I'm concerned :-) )

sbierwagen• 12 years ago

  code {background: #F7F7F9; color: #999999}

Egad, the color scheme he uses in the code blocks is horrible. The snook.ca contrast calculator says that's only a 2.66:1 contrast ratio, which is terribly unreadable. It needs to be at least 7:1.

(CSS edited for clarity)

Big changes to Erlang

Related Stories