Data munging in Perl 6 vs. Perl 5

http://perl6advent.wordpress.com/2014/12/09/day-9-data-munging-in-perl-6-vs-perl-5/

by lelf • 12 years ago

84 72 12 years ago

perl6advent.wordpress.com

Mithaldu3• 12 years ago

To be clear here, that Perl 5 code is not production grade code and i'd send anyone trying to commit that to a repo back to the hash mines.

bglazer• 12 years ago

Could you elaborate a little? I ask because the Perl 5 code looks a lot like the Perl that I write. I'd like to improve, but the existing Perl code that I'm writing on top of is really bad. Like really, really bad. I don't have much quality Perl to compare against.

Mithaldu3• 12 years ago

Two examples:

    $freq{substr $grade{$_}, 0, 1}++ for keys %grade;

better:

    $freq{$_}++ for map { substr $_, 0, 1 } values %grade;

best:

    my @letters = map { substr $_, 0, 1 } values %grade;
    $freq{$_}++ for @letters;

----------------------------

and

    open my $fh, '<:utf8', "grades.txt"
        or die "Failed to open file: $!";
    
    my %grade;
    while (<$fh>) {
        m/^(\w+) \s+ ([A-F][+-]?)$/x
        or die "Can't parse line '$_'";
        $grade{$1} = $2;
    };

better:

    use 5.010;
    use IO::All -binary, -utf8;
    
    my $line_format = qr/
        ^               	    # line start
        (?<student>\w+)         # name
        \s+                     # gap
        (?<grade>[A-F][+-]?)    # grade
        $                       # line end
    /x;
    
    my $fh = io->file("grades.txt");
    my %grade;
    while (my $line = $io->getline) {
        die "Can't parse line '$line'" if $line !~ $line_format;
        $grade{$+{student}} = $+{grade};
    }

peteretep• 12 years ago

Exactly. Perl's critics often completely overlook the degree to which liberal use of the map/reduce paradigm can simplify the graphical representation of your code and allow the powerful list processing features to be brought to bear. Consider:

    use strict; use warnings; use List::Util

    qw/reduce/;
    map { print } join "\n", "Distribution of grades by letter:",
    map { sprintf(  "\t%s: %s student%s",   @$_,   $_ -> [1] == 1 ? '' : 's'  )    }
    map { my   $c   =   $_   ;    sort   {   $a  ->  [0]   cmp    $b   ->   [0]    }
    map { [ $_, $c->{$_} ] } keys %$c } reduce { $a->{chr(ord($b->[1]))}++; $a } {},
    map { printf(  "List of students with failing grades:\n\t%s\n",   join  ', '   ,
    map { $_->[0] } grep {ord $_->[1] >68} @$_); @$_} reduce {push(@$a, $b); $a} [],
    map { printf( "${^A}'s grade: %s\n"  ,  $_ -> [1] )  if  $_->[0]  eq  $^A;  $_ }
    map { $^A="Zsófia"; [split(/\s+/)] } <STDIN>

The use of strict and warnings making this particularly Modern Perl, you could almost be forgiven for thinking you were reading Haskell...

Ultimatt6• 12 years ago

needs use feature 'say'

mercurial3• 12 years ago

It's definitely leaning toward the functional side, but some things look very odd (like calling "filename".IO...). The new operators leave me dubious (eg, '*' to create a one-argument lambda function). And '»'? At least the equivalent '>>' is apparently available, but damn...

delluminatus• 12 years ago

The * operator made me laugh out loud. Only Perl would have an operator called a "Whatever star".

Although, I actually think a lot of the stuff Perl does is really cool. I wouldn't want to have to maintain a codebase of it, but it never fails to be interesting. Like the whatever star. "*.value" basically saying, "Whatever value is here, gimme the value of it". It's different from a typical lambda declaration, but intuitive in its own way.

munificent• 12 years ago

It's different from a typical lambda declaration, but intuitive in its own way.

Isn't it pretty similar to Scala's use of "_"? I think _.value creates a one-parameter lambda in Scala.

muraiki2• 12 years ago

Clojure has %:

(map #(/ % 2) [2 4 6 8]) => (1 2 3 4)

(map #(% 2) [#(- 2 %) #(* 2 %)]) => (0 4)

(#(+ %1 %2 %3) 2 4 6) => 12

Ultimatt6• 12 years ago

It's not just one argument! I't consumes as many arguments as you like, there is also the HYPER WHATEVER which is two asterisc in a row * * (without white space a limitation of HN!) which is a dimensioned form of whatever for dealing with higher dimension data structures.

The following are identical with and without whatever:

  perl6> "ACAGATAAATTA".comb.map: * ~ * ~ * ;
  ACA GAT AAA TTA

  perl6> "ACAGATAAATTA".comb.map({$^first ~ $^second ~ $^third});
  ACA GAT AAA TTA

It's worth pointing out we're lucky that 'first' is lexically before 'second' in the second one though. The $^ consumes parameters in the lexical order of the variable names. So it's used normally for $^a $^b in sorting. So .sort({$^a cmp $^b}) versus .sort({$^b cmp $^a}). But a general feature was made rather than an obscure exception. In general Perl6 is very consistent. It might also just be a bit too crazy at times, some of the stuff you can do like the (* * *)(2,3) below you obviously would just not do in real life.

So an example where the lexical ordering is kind of neat:

  perl6> "ACAGATAAATTA".comb.map({$^third ~ $^second ~ $^first});
  ACA TAG AAA ATT

Contrived since you could just split the string by three characters and flip it :P But hopefully you get the idea!

discardorama2• 12 years ago

"and is joined by the new < > variant for literal strings."

... why? What was wrong with {} or [] , as most other languages do? Sigh.

masak• 12 years ago

The postcircumfix {} is still there, and used. It just unambiguously takes an expression, and doesn't try to intuit string quotes in some cases. So you use {} for indexing with a general expression, and <> for string indexing.

discardorama2• 12 years ago

I dunno, this sounds like syntactic bloat to me.

sigzero• 12 years ago

I don't think the P6 is more readable myself.

petercooper• 12 years ago

Perl was my go-to language for 8 years before Ruby so I fancied a comparison. And.. Ruby is rarely as brief, but even with my Perl hat on, it seems much easier to read, especially if you didn't write it:

Perl 6:

    say "  {.key}: {+.value} student{"s" if .value != 1}"
    for %grade.classify(*.value.comb[0]).sort(*.key);

Ruby:

    grades.group_by { |name, grade| grade[0] }
          .sort
          .each do |grade, names| 
      puts "  #{grade}: #{names.length} student#{'s' if names.length != 1}"
    end

Or if it had to be as 'functional' as the Perl..

    puts grades.group_by { |name, grade| grade[0] }
               .map { |grade, names| "  #{grade}: #{names.length} student#{'s' if names.length != 1}" }
               .sort.join("\n")

Both can be improved, of course! :-) I'd be keen to see a Python version actually because I know so few of its idioms but it often does well with these sorts of things as well..

WayneS• 12 years ago

Agreed, the new perl looks like Ruby. And I am willing to bet Ruby is more self consistent and easier to learn. (Says the perl 5 expert)

kbenson6• 12 years ago

In what ways do you not find Perl self consistent? There's lots or criticisms that can be leveled at Perl, but I'm not sure I follow that one.

Skunkleton2• 12 years ago

I threw up a python version, and stole your group_by for the frequency portion :)

cletus• 12 years ago

I don't mean to be snarky but... Ok, sorry I'm going to be snarky.

In all honesty, does anyone even care about Perl 6 at this point? It's become the Duke Nukem Forever of programming languages as far as I can tell.

Back in the 90s I remember the first CGI scripts I wrote were in Perl but really... the world has moved on.

cant-be-serious• 12 years ago

Kool kid won't care, they're already builds next big thing with nodejs anyway..

mercurial3• 12 years ago

Minus the snarkiness, that's not a bad question. The linked article doesn't showcase most of the interesting stuff in Perl 6 like optional typing, junctions or macros, but there are some cool things about it.

That said, I think Python has eaten Perl's lunch, and who knows how things will evolve with the proliferation of statically-typed, non-verbose, fast functional languages? Personally, I'm not going back to dynamic languages as my primary workhorse.

cygx3• 12 years ago

Personally, the features I'm most interested in are ones you didn't actually list. They are

- grammars

- grapheme-based Unicode

- implicit parallelism

This is why Perl6 is still relevant today. Larry is hopeful that Rakudo can deliver sometime next year (not all of these features are quite there yet), but even if it doesn't, the language will remain of interest as long as there is no serious contender in sight.

agumonkey• 12 years ago

I'm not following the Perl community, but I loved seeing some of Damian Conway's talk (mostly on very very fine and inspiring first class grammar) on P6, do you know some good articles about Perl6 new/weird ideas ?

hyp03• 12 years ago

Haskell and ocaml fit the bill, but aren't new... which ones do you mean?

wwweston• 12 years ago

I check in with what's going on in the Perl community every so often because I find I usually learn a lot more from them than, say, from the Ruby or Python community.

Perl 6 does seem to have become a perpetual playground for language experimentation, but that hardly makes it Duke Nukem Forever -- you can use it for that today.

Perl 5 is alive and well, seeing regular updates, and seems to have a future, even if its glory days as far as mindshare are over.

mhd• 12 years ago

Has it? If we're talking about Python or Ruby, I'd regard that as a small move sideways, not onwards. It's pretty much the same family of languages, and Perl6 almost seems like a new member of it, yet another alternative for Perl5, not necessarily the most likely upgrade path.

If we're talking about the world having moved on to either Haskell, Erlang or Go, Rust, well, I don't really see that being as pervasive as it often appears on trend-seeking sites like this.

And I really hope we're not talking about JavaScript…

hyp03• 12 years ago

Maybe, the alternative perl, "Perlb"? I'd also agree with the "perpetual playground" comment, and note that Perl6 originated not as something needed, but as a way to keep the developer band together.

Plus, I've always noticed a tendency in Perl in the last few years to be interesting/entertaining... sort of like it was designed by science fiction writers.

bane• 12 years ago

You should, even if you don't use it. It's highly likely that ideas that come from Perl 6 will be adopted in some form in most other major languages at some point.

Perl's weird, it just takes a bit of everything (and not in a C++ kind of "be" everything), tosses it in a blender and then suddenly everybody has or wants smoothies in their language.

A short list of what I think we'll see percolating over to other places:

- some of the better syntactic sugar ideas, like blocks and closures, comma as a list constructor, chained comparisons

- what will probably be best-in-class Unicode support

- autothreading

- further improvements to the way regexes are used

- grammars

- junctions could become the next "thing" in lots of languages

- elements of the module system, like using two versions at once without conflict

- the different parameter passing modes

- the current Perl culture around testing

- bags and sets

- the continued relevance and model of CPAN

themckman• 12 years ago

Gosh...I want my next job to be in Perl...it just looks like such a different language. It makes me smile every time I read it and read about it.

Skunkleton2• 12 years ago

I feel like python provides a more readable solution.

    #!/usr/bin/env python3
    with open('data.txt', 'r') as f:
        data = dict(x.split() for x in f if x.strip())
        print("Zsófia's grade:", data["Zsófia"])
        print("List of students with a failing grade:\n  ",end="")
        print(*[x for x in data.keys() if data[x] == 'F'], sep=", ")
        print("Distribution of grades by letter:")
        for k,g in groupby(sorted(data.values()), lambda x: x[0]):
            print("  {}:".format(k), sum(1 for _ in g))

viraptor• 12 years ago

A working version with some stuff improved - groupby implemented, len() instead of sum(), a less squished version of printing, "student(s)" added:

    #!/usr/bin/env python3

    from collections import defaultdict

    def invert_dict(data):
        res = defaultdict(list)
        for k, v in data.items():
            res[v[0]].append(k)
        return res

    with open('data.txt', 'r') as f:
        data = dict(x.split() for x in f if x.strip())

    print("Zsófia's grade:", data["Zsófia"])
    print("List of students with a failing grade:")
    print("  " ,end="")
    print(*[x for x in data.keys() if data[x] == 'F'], sep=", ")
    print("Distribution of grades by letter:")
    grouped = invert_dict(data)
    for k, g in sorted(grouped.items()):
        print("  {}: {} student{}".format(k, len(g), 's' if len(g) > 1 else ''))

Veedrac• 12 years ago

No need for `invert_dict`; you're just counting so `collections.Counter` would work fine:

    from collections import Counter

    with open('data.txt', 'r') as datafile:
        students = dict(line.rsplit(maxsplit=1) for line in datafile)

    failing = [name for name, grade in students.items() if grade >= 'E']
    grade_counts = Counter(grade[0] for grade in students.values())

    print("Zsófia's grade:", students["Zsófia"])

    print("List of students with a failing grade:")
    print("  " + ", ".join(failing))

    print("Distribution of grades by letter:")
    for grade, count in sorted(grade_counts.items()):
        print("  {}: {} student{}".format(grade, count, 's' * (count != 1)))

codygman• 12 years ago

A Haskell solution, though I'm sure someone can do better ;)

Perhaps I'll clean it up using printf/format[0] later.

    {-# LANGUAGE OverloadedStrings #-}
    import           Control.Monad     (liftM)
    import qualified Data.HashMap.Lazy as H
    import           Data.List         (sort)
    import           Data.Maybe        (fromMaybe)
    import           Data.Monoid       ((<>))
    import qualified Data.Text         as T
    import qualified Data.Text.IO      as TIO
    import           Prelude
    
    main :: IO ()
    main = do
      gradesMap <- liftM
                   (H.fromList . map ((\(x:y:_) -> (x,y)) . T.splitOn "\t") . T.lines)
                   (TIO.readFile "example.txt")
      let failingNames = H.keys $ H.filter ((== F) . T.take 1) gradesMap
          gradeTotals = foldl
                        (\m k -> H.insertWith (+) (T.take 1 k) 1 m)
                        H.empty
                        (H.elems gradesMap)
      TIO.putStrLn $ "Zsófia's grades: " <>
        fromMaybe "N/A" (H.lookup "Zsófia" gradesMap)
      TIO.putStrLn "List of students with a failing grade:"
      TIO.putStrLn $ "  " <> T.intercalate ", " failingNames
      TIO.putStrLn "Distribution of grades by letter:"
      mapM_ (\(k,v) -> TIO.putStrLn $ "  " <> k <> ": " <> (T.pack . show $ v) <> " students")
        (sort . H.toList $ gradeTotals)

0: http://hackage.haskell.org/package/formatting

jrochkind12• 12 years ago

Question which is meant to be an honest question and not a troll, I really want to know and accept that some might have reasonable answers:

Why would you use Perl6 instead of ruby?

zzzcpan6• 12 years ago

It depends. Why would you use Ruby instead of Perl 5?

jrochkind12• 12 years ago

Ruby instead of Perl5 (not Perl6)? Because I found Perl5 really hard to write maintainable comprehensible code in, requiring weird magic boilerplate, and full of gotchas. Could just be me, maybe could have been overcome, was just my subjective opinion. And I like writing OO code in Ruby style. Also, like many others, what brought me to ruby was wanting to check out Rails (although now I'd say I have fonder feelings for ruby the language than I do for Rails the framework). All just my own opinions and experiences.

Those sorts of answers from other's perspectives is what i'm interested in for Perl6 (not 5) over ruby. I think I know many reasons to prefer perl5 over ruby (although I don't), but once you leave perl5 for perl6... I'm not sure why you wouldn't just ruby (or python). But could be just cause you're used to Perl and like Perl. Or could be because there's something about Perl6 you especially like over ruby/python. Etc. I think it's a reasonable question?

zaphar• 12 years ago

I don't currently use either but Perl6 grammars[0] are pretty dang awesome. If I were writing parsing code and my choices were Ruby or Perl6 I would choose Perl6 just for the grammars.

[0] http://doc.perl6.org/language/grammars

muraiki2• 12 years ago

I think Perl 6's killer feature is actually concurrency. P6 provides a number of ways to do reactive programming, channels, actor model semantics, etc. It even offers some new ways to approach modeling concurrency and asynchronous logic.

Here's a quick intro to supplies: http://perl6advent.wordpress.com/2013/12/19/perl-6-supplies-...

Here's a more in depth exploration of Perl 6's concurrency features: http://jnthn.net/papers/2014-nlpw-reactive.pdf

If you want to use a dynamic language (albeit also offering optional static typing) with powerful concurrency options that doesn't also involve s-expressions, I think P6 might be one of the few options available. I'm not familiar with Ruby, but from what I've heard the default runtime has some limitations when it comes to concurrency, and while it's been a while since I used Python, I recall hearing that it also has some problems with handling concurrency.

debacle• 12 years ago

Speed. Always speed. Perl is still faster than every other scripting language out there.

cygx3• 12 years ago

Perl is still faster than every other scripting language out there.

Wait, what? There are are specific use cases where this might still be true, but as a general rule, this hasn't been the case for some years now.

Ultimatt6• 12 years ago

Not true for Perl6 at the moment. Perl6 is an entirely different language with its own implementation. At the moment it will be either on par or slower than Ruby. For some select numerical operations Perl6 is faster than Perl5. But for the rest it's between 100x and 10x slower than Perl5. This is what is being worked on now up to the official release of 6.0.0 though.

esaym2• 12 years ago

I thought of a few troll-worthy comments, but then moved on to others. Few things came to me though later that are less abrasive so I guess I'll share these nicer sides of my opinion:

I don't know if it will be the same for perl6, but at least with perl5, backwards compatibility was rarely broken. Every time my debian-testing box upgraded a ruby version, it broke something. From 1.8,1.9,2.0, and I think 2.1 actually worked without any issues...

This is never an issue with perl5. And before you say "perl is dead it never changes", go look on the perl5porters mailing list. There are a ton of changes going on. I had every intent this year to join the dev team, but the commits come in so quick on that mailing list, I can never find a place to start. When perl5 updates to a new version (which has been happening once a year for like 5 years now), it is not just a few bugs that are fixed, but a whole lot of new features are added and plenty of new syntax to scare the heck out of anyone. So an argument of the ruby dev cycle is 'slower' probably won't hold up.

Onto more perl6 specifics, I think one of the original goals was perl6 being able to compile a script into a raw binary. Don't know if that feature will be implemented. Ruby can't do that, though it does have the JIT compile stuff. So I guess that could be one reason..

Perl6 is new, so I guess it is full of new "technology and stuff" right?

I personally never did much with ruby. I tried to use it as my first language to learn with. I signed up on some hacker forum wanting to learn to hack with my new found ruby skills... They just booted me with the message of "ruby aint really programming". This was back in 2007, but I never could get over that...

From then on I only used Ruby when I couldn't figure out how to do something in perl. So none of my ruby scripts have ever been seen by another pair of eyes. I keep them all hidden, knowing that they are cool and all in their own right, but knowing if I were just a little smarter... I could have written it in perl...

kamaal• 12 years ago

A full answer will look like a book.

The short answer, is that the comparison isn't fair to either communities. Perl 6 has many killer features. Concurrency, Macros, Grammars, Regex goodness, functional programming features, OO features, Strong typing support, user defined operators, multiple VM backends, Continuations to name a few. But there are many many others.

Perhaps the most killer feature of Perl 6 is like lisp, you can use the language to grow itself without breaking backwards compatibility. This alone will ensure that the community can grow without worrying about breakage, regression and backwards compatibility.

Read more here: http://perlcabal.org/syn/

Ruby and Perl 5 definitely have more libraries, users and a far more mature ecosystem of things. But I guess the sheer awesomeness of Perl 6 will attract a lot of developers and that will have a healthy community of its own.

In short I think there are plenty of reason why one would chose Perl 6 above Ruby.

Data munging in Perl 6 vs. Perl 5

Related Stories