Saturday, October 24, 2009

Language evolution

[Spanish source]
Perhaps you have notice that I have not written very much lately, the reason was that I had to devote full time to an urgent migration.

The system in question belongs to the client who hired me to develop a web application, which incidentally was my first application of this style back in 1998.

At that time I was giving my first steps with Perl, but somehow people convinced me that the application should be developed in PHP, I slowly realized that PHP was not flexible enough, and end up doing several Perl programs that implement the most of the functionality, yet the entire application interface remained in PHP.

During this project, I learned that Perl was much more versatile, powerful and fun than PHP (3), and if I had not been persuaded, would probably have found, the entire application would be in Perl and I would not have a story to tell.

At the start of the migration process, it occurred to me that it would be easier to do interface migration first, because I thought that being the simplest part of the application could be easily migrated to PHP5, this was far from reality, because nothing worked as it should.

I understand that everything evolves, but programs didn't even compile, some APIs have changed enough to warrant a complete review of all source code, and of course the application was a mess, what else could it be?, At the time there wasn't available a library of templates, access to the database was done with the horrible library functions for PHP3, and the entire application is a monument to the ASP style programming, where the view, model and controller were completely integrated, as if they had gone through a blender.

The truth is that it was easier to compile PHP3 for the new operating system than trying to migrate the code to PHP5, in that matter I wish to thank Debian and particularly, because it may be one of the few places on the planet that still keep PHP3 source code, because the language's community has a policy of withdrawing old sources from the Internet.

Having overcome the problem (ie PHP), I had nightmares about the pile of Perl 5 code that would not work at all in the new Perl 5.10, so I resigned and started to copy and run all files.
The first thing broken was DBI, in fact the connection string to the database was using a format that I did not even remember it had existed:

  DBI-> connect ( "dbi: Pg: dbname =", "user", "pass");  
which I changed to the current syntax:

  DBI-> connect ( "dbi: Pg: dbname = mydb, host =", "user", "pass");  
I tried the program again and it worked, I proceeded to try another program and it worked, and the trend continued, program by program and couldn't find any faults.
A decade of language and the CPAN evolution and almost everithing worked at once, I was amazed (and of course pleased).

By contrasting the Perl and PHP evolution I can appreciate that while the former has acquired a wealth of features such as new language constructs, several concurrency implementations, event-driven, threads and corrutines, new abstractions to facilitate and improve OOP which allow the programmer to choose between several object-oriented systems, significant advances in the area of meta programming, and some others, the later has achieved basically a system of OOP, and still it is more incompatible than Perl.

The question is: why this radical difference?, especially when it is said that there are many more solutions ready in PHP than in Perl and then there should be more development effort in former that the later.

You may have the impression that I am bashing about PHP, but I am not, PHP was just the trigger that made me think about the problem, but if we look to other languages, will probably reach similar conclusions, for example: how much did Python language evolved in the last decade?, would a 10 year old program still work in the current envirnment?, what about Java?. My bet is that Perl beats them both on these fronts.

Although Perl is a New Jersey solution, the language itself is heavily influenced by Lisp, a MIT solution, although one can argue about the complexity of the syntax of Perl, the fact is that the semantics of their operations is consistent enough and extremely versatile, allowing modules extending the language, so you can experiment with new features without  introducing them into the interpreter.

Other programming environments have very good tools that greatly facilitate the implementation of compilers, but the complexity in the implementation of any language extension by preprocessing the language limits the development of such extensions.

 Moreover, the lack of integration with the compiler, forces the user to introduce complexity in the construction of software, execution of pre-compilers, handling of temporary files, and others, that become a nightmare particularly in dynamic environments, imagine that to load a Moose module, you should need to run the Moose preprocessor, and then load the module resulting from the compilation, it would be really annoying, Perl has mechanisms that allow the Moose compiler to run automatically, accessing the source code and returning the modified code to the Perl compiler for the final stage of compilation, and you all you need to do is: use Moose; at the program start.

The problems described above limit the adoption of external extensions to the language, restricting the evolution through extensions, which is the easiest way to evolve a language. In this sense Perl is a lot like Lisp, and most of the evolution of language is achieved by adding external modules from the CPAN. Today there are extensions that implement various structures such as try/catch, adding the ability to declare the parameters in the declaration of subroutines, and many other goodies that are not part of language.

With respect to compatibility, there are two factors to take into account: the compatibility of the libraries and the language itself.
The language compatibility is maintained using pragmas, which for Perl are almost the same as any CPAN module (or it seems so, from the perspective of a programmer), so when the language changes new pragmas are added. For example the new features of Perl 5.10 are activated using a pragma.

The greatest Perl incompatibility occurred in the last decade is the elimination of pseudohashes, and to minimize the problems mechanisms were created to facilitate the migration of existing code (use fields) and the deprecated pseudohash feature was keep around for about 5 years. This illustrates how the community is committed to maintain compatibility with existing code.

The community has recently adopted the term DarkPAN, which is all the code is written in Perl, that is not publicly available, but lies hidden in thousands of systems that people do not even know have been written in Perl (the one in this story appears to be written in PHP), and although many people prefer to modernize the Perl platform obviating the DarkPAN, community generally thinks that this is not the best policy.

In the universe dark matter accounts for most of the mass, and we do not know if DarkPAN outweighs the CPAN, but we can see some side effects, for example, though there seems more systems written in PHP, there are many more job offerings for Perl, and DarkPAN may be the cause.

The truth is that all that code which drives many business is there because it works, and because it can be maintained over time with less effort than similar code developed on other platforms.

The most impressive fact is that today we have a very modern platform compatible with a decade old code, with libraries so easy to use, modern and powerful such as Moose, Devel:: Declare and Catalyst, which are the envy of other languages' developers.

So if you want to program in a simple, modern, powerful and enduring, plataform the choice is clear: Perl.


  1. This comment has been removed by the author.

  2. Nice story, but to be completely honest, there exists an archive page with sources of earlier versions:

  3. I am glad to hear that, I couldn't find it with google, thanks for the link.

  4. Cool story. However, I am not sure you make a good job praising perl. Why not?

    Because you ignore languages like python and ruby. I mean, we all know that php can not win a beauty contest with perl, but both python and ruby can. I dare to claim this. The CPAN is so often touted as what sets perl apart from the other languages - but why is it that all three languages were such a success despite perl being around?

    I really think that in the long run some languages will tend to be easier to maintain. One has to have a solid structure underlying the whole language, as a conceptual framework.

    Decade old code can not be a good motivation. We still see Ada being used, even though it seems to finally fade off slowly - but could decade old ada code be a sign of it being great? Or was it rather that it was able to snatch a niche successfully?

  5. shevegen: Python also has serious compatibility problems (Python 3.0). So may have Ruby as I heard. As for maintainability, Python even has no lexical variables and no separate operators for string/number comparison (same problems as in PHP).

    PHP had success because it was marketed as "you can program in it without knowing programming". Ruby - because of hype with Rails. Python - due to anti-Perl FUD. But still, Perl is more popular than Python and Ruby.

  6. I see the Python 3.0 intentional break of compatibility as lost opportunity to add some important missing features in this language.

    As a mater of taste I like ruby a lot, but I don't see anything interesting in python, besides a clean syntax, so they rather keep syntax clean than add useful features to the language, as an example think about one line closures.

    Lack of lexical variables is a very missed feature because they simplify the code, saving lots of bugs and making programs much easier to write, understand and maintain.

    Maybe perl is snatching some niches, like system administration and web (should I say network?) programming, if so I am happy with those two ;-)