Smart Perl

[Spanish original source]
In the previous article we saw an example of modern Perl, today we'll delve a bit more into Perl 5.10 smart matching, and how this combined with the dynamic nature of the language lead us to a ridiculously small program, which is also easier to understand and maintain.
I once read (I think from Paul Graham) that when sections of code seems very similar, it usually means that a level of abstraction is required, of course  he is a Lisp programmer, and has defmacro. However Perl also has its own means, and in this case our first solution could be based on a hash that includes the functions allowed in our calculator:

 1 #!/usr/bin/perl
 2 
 3 use Modern::Perl;
 4 use Scalar::Util qw( looks_like_number );
 5 use Statistics::Descriptive;
 6 
 7 use constant SYNTAX_ERROR => "Error: tipee 'help' para ayuda";
 8 
 9 my %FUNCS = (
10     sum                => 0,
11     mean               => 0,
12     count              => 0,
13     variance           => 0,
14     standard_deviation => 0,
15     min                => 0,
16     mindex             => 0,
17     max                => 0,
18     maxdex             => 0,
19     sample_range       => 0,
20     median             => 0,
21     harmonic_mean      => 0,
22     geometric_mean     => 0,
23     mode               => 0,
24     trimmed_mean       => 0,
25 );
26 
27 my $s = Statistics::Descriptive::Full->new();
28 while (1) {
29     print "Listo> ";
30     my $command = readline(STDIN) // last;
31     $command =~ s/^\s+//; $command =~ s/\s+$//;
32     given ($command) {
33         when ( looks_like_number($_) ) { $s->add_data($command) }
34         when (/^(exit|quit)$/)         {last}
35         default {
36             if   ( exists $FUNCS{$command} ) { ... }
37             else                             { say SYNTAX_ERROR}
38         }
39     }
40 }

This is a good step forward, because we are simplifying code in the complicated part of our program, and replacing it with a simple declaration of a hash, where including new functions is as simple as adding a line.
Of course any astute reader already noticed that I cheated because the program is incomplete and line 36 requires an action, our problem is how do we invoke the right method for the operation, and as usual there is more than one way to do it, the worst could have the hash filled with references to the methods, like this:

10     sum  => \&Statistics::Descriptive::sum,

which allows to invoke the methods as:

36     if ( exists $FUNCS{$command} ) { say "$command = " . $FUNCS{$command}($s) }

That is the worst way because you have to know a lot of Perl to understand how it works, and Perl has the ability to dispatch methods symbolically allow us to make our intention perfectly clear with code easier to understand:

36     if ( exists $FUNCS{$command} ) { say "$command = " . $s->$command }

The runtime cost involved in the later is higher than the former, but it is a price that I am willing to pay gladly, because it makes the program much easier to understand, and lately people is much more expensive than machines.
Finally, if laziness is one of your principles, you can rewrite the definition of the hash like this:

 9 my %FUNCS = map { $_ => 0 } qw( sum mean count variance standard_deviation
10     min mindex max maxdex sample_range median harmonic_mean geometric_mean
11     mode trimmed_mean );

which I like, because it saves me some punctuation (which seems to overwhelm many people) and I have less probability of making a syntax error.
Basically I'm building a list of words (the names of the methods) with "qw", from this list I build another (using map) that contains each element of the original list ($ _) followed by 0, perl automatically converts this list into a hash where each name is associated with 0 as its value.
If you think the above explanation was complicated or incomprehensible, you can see the Perl documentation for map, what will be great because you can also learn something about functional programming, and that will be very helpful for sure.
Now I'm going to get rid of the "if", I prefer multi-way conditionals because they are linear, they look better and are easier to follow, that's why I think that the given/when is the best thing that happened to Perl in a long time, besides I am also got rid of the regular expression in line 34 for something that makes more sense for a Perl outsider:

 1 #!/usr/bin/perl
 2 
 3 use Modern::Perl;
 4 use Scalar::Util qw( looks_like_number );
 5 use Statistics::Descriptive;
 6 
 7 use constant SYNTAX_ERROR => "Error: tipee 'help' para ayuda";
 8 
 9 my %FUNCS = map { $_ => 1 } qw( sum mean count variance standard_deviation
10     min mindex max maxdex sample_range median harmonic_mean geometric_mean
11     mode trimmed_mean );
12 
13 my $s = Statistics::Descriptive::Full->new();
14 while (1) {
15     print "Listo> ";
16     my $command = readline(STDIN) // last;
17     $command =~ s/^\s+//; $command =~ s/\s+$//;
18     given ($command) {
19         when ( looks_like_number($_) ) { $s->add_data($command) }
20         when ( ["exit", "quit"] )      {last}
21         when (%FUNCS)                  { say "$command = " . $s->$command }
22         default                        { say SYNTAX_ERROR }
23     }
24 }

Now it looks a lot better (it even resembles Erlang).
I am using some functions of the smart matching that will explain below.
In line 20 there is an array matching:

$command ~~ ["exit", "quit"]

Matching a scalar (to left) against an array (to the right) is equivalent to:

sub match_scalar_arrayref {
    my ($scalar, $arrayref) = @_;
    for my $item ( @$arrayref ) {
        return 1 if $scalar eq $item;
    }
    return undef;
}

I can't remember how many times I've wrote code like that, or this:

if ( grep { $scalar eq $_ } @$arrayref ) ...

which I may be able to write clearly and with less work:

if ( $scalar ~~ $arrayref ) ...

Probably you've already guessed that match at line 21 is equivalent to:

if ( exists $hash{$scalar} ) ...

A little temptation

I received a reader's suggestion that make our program shorter and easier to maintain, the idea was to change the line:

21    when (%FUNCS) { say "$command = " . $s->$command }

by:

21    when ($s->can($command)) { say "$command = " . $s->$command }

The "can" method is provided by the UNIVERSAL class, from which all objects ultimate derive in Perl, and the purpose of this method is to determine if an object or class has a particular method.
By using this we could delete %FUNCS completly, our interpreter will be automatically updated with new commands as Statistics:: Descriptive evolves, which sounds very good from the standpoint of maintainability, however, this has a fatal flaw for me: is not safe.
The problem is that I lose control over what Perl runs automatically, which may not be critical in this case, but it could be extremely dangerous. So I prefer the security and keep the hash as a mechanism of dispatch (and authorization of use).
The moral is to be careful when using dynamic execution control mechanisms, especially when using data from external unreliable sources to be injected into this execution control mechanisms.

Finishing the program

Line 22 gives an error when a command is unknown, the message says to use "help" for help, but the command "help" is not implemented yet, a quick way to implement it is:

23         when ("help") {
24             say "Los comandos válidos son: "
25                 . join( ", ", qw(exit quit help), keys %FUNCS )
26         }

Wow, that was easy, and the best is that it is also consistent because it uses the same data structure to report, select and authorize the commands.
One command that I forgot to include in the calculator in the previous article was "clear", adding this function now is as simple as putting a new word in the definition of %FUNCS:

 9 my %FUNCS = map { $_ => 1 } qw( sum mean count variance standard_deviation
10     min mindex max maxdex sample_range median harmonic_mean geometric_mean
11     mode trimmed_mean clear ); 

It was easy, right?. The best thing is that the new command appears automatically in the help because the program is consistent.
Lets  recap today's accomplishments, we have a program:

• Very compact
  
• Easy to understand
  
• Easy to mantain
  
• Consistent
• Safe
  

Perl is as good or as any other language at many fronts including good design a quality, but few languages offer mechanisms like the ones used here to develop this program with such little work.
Next time I will improve the example with a manual of statistic functions with very little effort.
I will say goodbye with the final version of the program.

 1 #!/usr/bin/perl
 2 
 3 use Modern::Perl;
 4 use Scalar::Util qw( looks_like_number );
 5 use Statistics::Descriptive;
 6 
 7 use constant SYNTAX_ERROR => "Error: tipee 'help' para ayuda";
 8 
 9 my %FUNCS = map { $_ => 1 } qw( sum mean count variance standard_deviation
10     min mindex max maxdex sample_range median harmonic_mean geometric_mean
11     mode trimmed_mean clear );
12 
13 my $s = Statistics::Descriptive::Full->new();
14 while (1) {
15     print "Listo> ";
16     my $command = readline(STDIN) // last;
17     $command =~ s/^\s+//;
18     $command =~ s/\s+$//;
19     given ($command) {
20         when ( looks_like_number($_) ) { $s->add_data($command) }
21         when (%FUNCS)                  { say "$command = " . $s->$command }
22         when ( [ "exit", "quit" ] )    {last}
23         when ("help") {
24             say "Los comandos válidos son: "
25                 . join( ", ", qw(exit quit help), keys %FUNCS )
26         }
27         default { say SYNTAX_ERROR };
28     }
29 }

Using Modern Perl

[Source article in spanish]
I will try to write a series of articles about Perl, showing how easy and quick is to make solutions based on this platform.
For this I chose a simple design that allows me to illustrate a number of techniques and best practices, with an algorithm accessible to any developer even to a rookie.
The example program will be a statistical calculator which at first will be written in a traditional style, but will gradually become more flexible and easier to maintain, while applying some unique mechanisms of language and some libraries from CPAN.
The grand finale is to make the calculator as a web application using a suprising mechanism available for Perl. Having said that, I will start using modern Perl now.
Giving honor to the title of the article, the first thing our program does is to use the module Modern::Perl, which is a shortcut to say:

use feature ':5.10';
use strict;
use warnings;
use mro 'c3';

That is, turns on all the features introduced in Perl 5.10, also activates the strict and warnings, and finally set the method resolution order to the C3 algorithm. As expected all the examples we will see throughout this series of articles, will only work in Perl 5.10, because I'm trying to promote as many new features as possible, so: install Perl 5.10 now.
Modern Perl advocates strongly recommended the use of strict because it captures many common errors, including accidental use of symbolic references, and typographical errors in variable names, at the cost of declaring them with our (globals) or my (lexicals) before use.
Perl warnings inform us about possible errors in coding. In Perl 5.10 strict is more strict and warnings gives many new warnings, so, they catch more problems than before, which usually improves the overall quality of code and save debugging time.
In my case, when I wanted to read a command or finish the cycle in case of an end of file, so I wrote:

my $comando = readline(STDIN) or last;

Perl immediately warned me that in some cases undef (which signals the EOF) could be confused with "0" (zero) coming from the file, because perl interprets "0" and undef as false values. One way to correct the instruction would be:

defined (my $comando = readline(STDIN)) or last;

But I rather use the new operator // (defined or), that simplifies the statement:

my $comando = readline(STDIN) // last;

The C3 method resolution order, solves some problems with the original resolution order of Perl, and it is advisable to always use it in new code, this is not entirely new, there are modules that use this resolution order for some 4 years now, beacause of a CPAN module (Class:: C3) but now C3 has native support in the language.
So the first tip is to use Modern:: Perl everywhere, because it activates a number of useful and recommended features of Perl in one shot.
Returning to the program, after using Modern:: Perl, it imports the subroutine looks_like_number() of Scalar:: Util, which saved me the trouble of writing regular expressions to recognize numbers, and also saves a lot of panic from readers that can freeze just by looking at those regular expressions.
The last module in use is the main ingredient of the calculator, it never crossed my mind to write statistical algorithms, that's the pupose of CPAN, which has almost everything in it. I choose to use Statistics:: Descriptive, which serves my purpose perfectly.
Line 7 declares a constant with an error message and line 9 defines a variable with an object of class Statistics::Descriptive::Full which will be the state of the calculator during the main loop.
The main loop is simple: read a command or terminate (last) if reached end of file [line 12], then remove the spaces from the left and right of the command [line 13], if the command is a number add it to the dataset [line 15] and if not, select and execute a command.
The selection is done with the new control structure of Perl 5.10 given/when [lines 18-36] that performs smart matching between the given value and the when clauses. As the matching is "smart" depends on the operands, and generally works as expected, however there are some oddities and it never hurts to read the manual.
Finally, the new say operator is just a print which puts a newline at the end of the string, avoiding a lot of concatenations with "\n" and therefore contributing to code clarity.

 1 #!/usr/bin/perl
 2 
 3 use Modern::Perl;
 4 use Scalar::Util qw( looks_like_number );
 5 use Statistics::Descriptive;
 6 
 7 use constant SYNTAX_ERROR => "Error: tipee 'help' para ayuda";
 8 
 9 my $s = Statistics::Descriptive::Full->new();
10 while (1) {
11     print "Listo> ";
12     my $command = readline(STDIN) // last;
13     $command =~ s/^\s+//; $command =~ s/\s+$//;
14     if ( looks_like_number($command) ) {
15         $s->add_data($command);
16     }
17     else {
18         given ($command) {
19             when ("sum")                { say "$command = " . $s->sum() }
20             when ("mean")               { say "$command = " . $s->mean() }
21             when ("count")              { say "$command = " . $s->count() }
22             when ("variance")           { say "$command = " . $s->variance() }
23             when ("standard_deviation") { say "$command = " . $s->standard_deviation() }
24             when ("min")                { say "$command = " . $s->min() }
25             when ("mindex")             { say "$command = " . $s->mindex() }
26             when ("max")                { say "$command = " . $s->max() }
27             when ("maxdex")             { say "$command = " . $s->maxdex() }
28             when ("sample_range")       { say "$command = " . $s->sample_range() }
29             when ("median")             { say "$command = " . $s->median() }
30             when ("harmonic_mean")      { say "$command = " . $s->harmonic_mean() }
31             when ("geometric_mean")     { say "$command = " . $s->geometric_mean() }
32             when ("mode")               { say "$command = " . $s->mode() }
33             when ("trimmed_mean")       { say "$command = " . $s->trimmed_mean() }
34             when (/^(exit|quit)$/)      {last}
35             default                     { say SYNTAX_ERROR }
36         }
37     }
38 }

To use the calculator simply execute the file, below is a test run:

opr@toshi$ perl stat.pl

Listo> 19
Listo> 45
Listo> 24
Listo> 15
Listo> 39
Listo> 48
Listo> 36
Listo> count
count = 7
Listo> 10
Listo> 28
Listo> 30
Listo> count
count = 10
Listo> mean
mean = 29.4
Listo> standard_deviation
standard_deviation = 12.685950233756
Listo> salir
Error: tipee 'help' para ayuda
Listo> help
Error: tipee 'help' para ayuda
Listo> exit
opr@toshi$

A simple improvement

A better way to write the program would be to delete the if statement at line 15 and make a new "when" clause, this also allows me to show that given topicalizes $_ to the given value and when clauses not only compare strings (using eq) and regular expressions (using =~) but also allow, among others, to write boolean expressions using $_ as an alias to the value being matched.

 1 #!/usr/bin/perl
 2 
 3 use Modern::Perl;
 4 use Scalar::Util qw( looks_like_number );
 5 use Statistics::Descriptive;
 6 
 7 use constant SYNTAX_ERROR => "Error: tipee 'help' para ayuda";
 8 
 9 my $s = Statistics::Descriptive::Full->new();
10 while (1) {
11     print "Listo> ";
12     my $command = readline(STDIN) // last;
13     $command =~ s/^\s+//; $command =~ s/\s+$//;
14     given ($command) {
15         when ( looks_like_number($_) ) { $s->add_data($command) }
16         when ("sum")                   { say "$command = " . $s->sum() }
17         when ("mean")                  { say "$command = " . $s->mean() }
18         when ("count")                 { say "$command = " . $s->count() }
19         when ("variance")              { say "$command = " . $s->variance() }
20         when ("standard_deviation")    { say "$command = " . $s->standard_deviation() }
21         when ("min")                   { say "$command = " . $s->min() }
22         when ("mindex")                { say "$command = " . $s->mindex() }
23         when ("max")                   { say "$command = " . $s->max() }
24         when ("maxdex")                { say "$command = " . $s->maxdex() }
25         when ("sample_range")          { say "$command = " . $s->sample_range() }
26         when ("median")                { say "$command = " . $s->median() }
27         when ("harmonic_mean")         { say "$command = " . $s->harmonic_mean() }
28         when ("geometric_mean")        { say "$command = " . $s->geometric_mean() }
29         when ("mode")                  { say "$command = " . $s->mode() }
30         when ("trimmed_mean")          { say "$command = " . $s->trimmed_mean() }
31         when (/^(exit|quit)$/)         {last}
32         default                        { say SYNTAX_ERROR }
33     }
34 }

I think that almost any programmer used to dynamic languages like Python or Ruby can readily understand code in Modern Perl and even be comfortable working with it.
The programmers of languages like C, C++, C# or Java, after getting used to some basic principles should feel a kind of liberating experience, because writing a program such this in those languages is certanly more difficult.
In the next article we'll see some dynamic features of Perl that make the program shorter, more flexible and easier to maintain.

Archaic Perl

A couple of days ago I had to attend a vendor who came to offer their services for the development of a web application.
As one of the participants of the organization had to handle some unexpected event, I took the opportunity to start a small research during casual conversation: "What development tools do you use at company?". In an ideal world the answer would have been: "Perl", but they told me that they work mainly in Python, but can work in other environments, including Perl. After informing them that the organization prefer Perl for the development of our applications, and after a micro religious debate, one of them (Juan) concluded:

In the end anything that can be done in one language that can be done in the other, but Perl programming is just more archaic.

that nearly upset me, but given that the missing guy arrived and the important issue was the meeting, I remained calm.
Now in retrospect I wonder: what he meant by saying that Perl is archaic? Perhaps John was referring to Perl 1 (1987), which was a kind of Shell Script with grep, sed and awk included, he could even think  that until Perl 4 (1991, soon after Python 1.0), however the current age is Perl 5 (1994) and viewing the subject young age, I think that Juan couldn't find a word to describe the mythical defects of Perl, so he ended up using the wrong word.
If Perl is archaic, then probably the object-oriented and functional programming are too, however those are the two technologies with most momentum at the present, and given that Perl's own object system was copied from Python, I will assume he meant some of the following:

1. Perl is ugly
2. Perl is messy
3. Perl is unreadable
4. Perl is incomprehensible

I will briefly address these prejudices that have been widely circulated on the Internet and for which there is no real support, much less after the rebirth of Perl (which I will discuss in another article).

Perl is ugly

As this is a matter of taste, things which are ugly to some one may be very attractive to other. But assuming that Perl is one of the ugliest languages, it has features that make it a practial language to solve a lot of problems easily.
One of the features that make Perl syntax leaning (not necessarily ugly) are the sigils indicating the type of each variable, however this feature facilitates extensibility and allows the interpolation in strings, and when I say Perl is extensible I mean that we can intervene in the compilation process to change its original syntax, a very high level feature shared with few languages, and the basis of the domain specific languages (also called DSL) that are very useful and popular. Perl offers at least three different mechanisms to achieve this goal.
Another feature is the practical integration of regular expressions within the language, therefore making extensive use of them, unfortunately, these expressions are ugly no matter what language are you using.
Take for example the parsing of a specific instruction from LaTeX:

\begin{document}

In Perl it would look something like:

if ( $latex =~ m/\\begin\{[az]+\}/ ) ...

In Python it would look like:

pattent = re.compile(r'\\begin\{[az]+\}')
if pattern.match(latex):
...

In fact neither is nice, but  Perl is really more succinct and easier to understand, and I will not show the hassle to use them from Java.
Finally there is the holy argument about coding style and all that nonsense about the compiler enforcing to write the code in a good style. I say this because even when I think you get used to "the right coding style", it is also true that sometimes this is a nuisance and gets in your way, and in such cases there is no remedy. Below there is an example of Python code hard to format because of inflexible language syntax.
When code requires a particular style, you should use formatting tools, for example I use indent for C, and perltidy for Perl, lately I write my Perl code in the following style:

perltidy -l=99 -sbl

No matter how I get code delivered or if I type it myself, because I can convert it into my standard style with a single command, even before saving it (I use vim).

Perl is messy

Languages are not messy, people are.
However, there are languages that have more features than others to organize a project, Perl provides several ways to organize the code to suit many needs, ranging from programming on a single line (command) to the construction of large and complex applications.
The language allows the creation of procedural modules with their own namespaces, which may even be organized into multiple files to be loaded "on demand", very enterprise, isn't it?.
While most languages only have a fixed way of handling objects, Perl has a basic object system that allows to implement OOP in many ways, like the language motto says: "there is more than one way to do it".
Perl avoids to force the programmer to follow a particular structure, whether it fits the needs of a specific program or not, otherwise it would be like Java.

Perl is unreadable

This is just a particular combination of the two myths that preceded it, but it is also argued that you can not read the code wrote after 15 min., and that may be good some times, because it allows you to write programs quickly even if they are dirty, after all nobody wants to design and document code following the principles of software engineering to understand the last program I wrote a few hours ago, only because I needed a hint on the character frequencies in a dozen files:

perl -MYAML -ne '$c{$_}++for split//;END{print Dump\%c}' data.txt

It is easier to write this again than trying to understand it, of course this is easy to make in Perl because it has some magical constructs, do not try something like this in another language, because your best scenario is to make it work, but I guarantee that it will be much longer and difficult.
But Perl also allows you to write nicer code if it were necessary:

use YAML;
use IO::File;
use strict;

my %counts;
my $fd = new IO::File $ARGV[0], "r";
while ( my $line = readline($fd) ) {
    for my $letter ( split( //, $line ) ) {
        $counts{$letter}++;
    }
}
print YAML::Dump( \%counts )

I'm sure this is as easy to understand as Python or Ruby for a programmer, isn't it?. Then the issue is not if the Perl language is unreadable, but the motivation to write the program, and the programmers expertice to write code easily readable or maintainable even by less experienced colleagues.
Finally, the least of my worries while making the last example was formating the code, since my editor did most of that work automatically, but just in case, I used perltidy on it, for you to see how well it looks.

Perl is incomprehensible

For whom?, Japanese is incomprehensible to me, but I doubt that to be the case for most people living in Tokyo, Perl is equally incomprehensible to someone who is not trained to understand it. In the previous section I showed you that Perl code may be as clear as Java or Python counterparts.
Certainly there is a lot of Perl code that is virtually incomprehensible except for the language gurus, but that doesn't mean that the programs should be written that way.
The shorthand form that allows to obfuscate Perl code also makes perl not just an interpreter, but a very useful tool in the command line, also allowing the expression of the genious and expertise over the language. There are very few languages that allow yo do JAPHs as Perl.
The fallacy appears when people start saying things like: "Perl is evil because it allows those things", or my favorite: "It is impossible to write unreadable programs in Python", really?, lets see who understands this little program written in Python:

for n in range(12):
    exec("abcdefghijkl"[n]+"=lambda x=0,y=0: "+filter(
        lambda x:x not in "\n$\r","""(x*y#x/x!range(x,
y#x+y!b(1,1#d(e~,e~#d(f~,f~#c(e~,e~+d(g~,d(g~,g~))#"%4
d" % a(x,y#map(lambda y:i(x,y),h~#" ".join(j(x)#"\\n".
join(map(k,h~))""".replace("~","()").replace("#",")!")
        ).split("!")[n])
print l()

and I can find much nastier things written in Java, which means you can write bad and incomprehensible programs in any language, and probably rookies can do it only because they are naive.
However, full potential of the developers may not be unleashed unless the language provide the highest level abstraction mechanisms, and in this case languages like Java and PHP are pretty bad while Perl outperforms Python and Ruby easily, disqualifying any possibility to label Perl as archaic.
Perl can be as corporate as any other language, and mastering this language in an organization is an investment where the code can be used and reused in many ways: from systems administrators to developers, via the database managers etc., and in solutions ranging from simple command line operation to the development a corporate application.