Another piece of old research. It’s so interesting, though, I can’t help putting a note up about it. In a piece of research released in 2000, Lutz Prechelt compared C, C++, Java, Perl, Python, Rexx, and Tcl.  (It’s gotten a fair bit of attention before, so it’s not new material)

What’s so good is the fairly rigorous and natural approach – instead of leaning exclusively on local students or one company’s employees performing some unnatural task, the researchers solicited for solutions online for the scripting languages (Perl, Python, Rexx and Tcl) and got 80 different implementations of a simple dictionary task. With these tasks they ran a lot of metrics and generated solid, comparable empirical data. Read it for the details (there is also a prettified version available).

I liked this observation, however:

For all program aspects investigated, the performance variability due to different programmers (as described by the bad/good ratios) is on average about as large or even larger than the variability due to different languages.

It’s quite profound – and even if the study itself is a bit dated and has some minor flaws – this observation probably still holds regardless!

Also, for Perl programmers, this statistic is quite neat:

The variability in execution speed for Perl programs were far lower than for the other languages except Tcl. It might be over-interpreting, but it indicates Perl is a bit more predictable – or Perl programmers are – than most of the other languages. Also, speed-wise is holding up quite good, and even the higher end of the range of execution speed is better than the higher end for all other languages.

The curious thing is that this indicates that the variability between programmers matters less with Perl (and Tcl) than the other languages. Or at least speed-wise, but the low variability also seems to be a tendency in the other metrics in the paper.

What’s so interesting, though, is how this demonstrates in hard numbers the importance of human variables in programming. It’s almost like framework matters less than the people using it.. funny, eh?

Also see this page of language comparison links.

9 thoughts on “Never mind the language, the programmer is what matters

  1. Tom DeGisi says:

    It might just mean that there is particular implementation for that problem in Perl and TCL which is obvious to to competent programmers in the language.

  2. Steven Obua says:

    Well, you could also argue that the language DOES matter most. In particular, you should only program in TCL or perl.

    Of course this is doubtful, just as these kind of experiments are 🙂

  3. bernz says:

    Cool, this suggests that most perl programmers reach an approximately equal (and respectable) level of insanity. 😀

  4. JGM says:

    Is it possible that PERL has some common library or built in set of functions/objects that perform a core part of the dictionary task requested? I would think that in that case you would see the low variation, as all or most of the submissions would be running identical code.

  5. admin says:

    DeGisi and JGM, your suggestion that this is caused by some particularly helpful implementation in Perl + Tcl is probably correct. I haven’t looked at the exact task, but a dictionary task sounds like a simple hash-implentation in Perl to me.

    But does that invalidate the data? Not at all, actually it is probably getting us closer to the bigger picture: The more common, important or useful tasks are made a part of the programming language the less it relies on the programmer to get it right. The Java,C++ and C-coders might all have had to rely on creating their own hashing algorithm while the Perl-coders could just use the hash algorithm built-into and optimized in Perl.

    That might be the reason for the lower variability, but it doesn’t make the observation more trivial. Rather it indicates that you might want to look at a higher-level language even for perfomance-demanding tasks, if it has built-in support for it, since statistics predict you might not be able to re-implement the algorithm in a better way anyways.

    Not exactly new thoughts, I guess… It’s just that they got some real observations of this that is cool.

  6. admin says:

    For the 0.001% of readers who *didn’t* get here through Reddit, there is also a lively discussion about this article here:

  7. Chris Blanc says:

    Perl is a general purpose language that excels at common tasks based on file systems, text processing, and any kind of data conformity or fixing task. With its built in regular expression powerhouse, it lends itself to the kind of thinking that produces quick and reliable solutions. As a result, it reminds me of other “glue” languages like Microsoft BASIC on 1980s computers and HyperCard on the Mac. I feel naked using a machine that does not have it installed.

  8. Emilio Wuerges says:

    My guess is that it is a matter of culture surrounding the programmers.
    Perl programmers often follow mailing lists where people post their problems and solutions. They often have a close solution to recur to, before starting from a totally new Idea. I don’t know about TCL, thought.

Leave a Reply

Your email address will not be published. Required fields are marked *