The CPAN Testers game

Never saw a person using a microscope to read code, but it could happen!

For those who don’t know, one of the greatest features of the Perl programming language is the centralized repository of Perl modules known as CPAN (Comprehensive Perl Archive Network).

CPAN includes a vast number of different solutions for a vast range of applications. For almost everything you will be able to find a match (or more) of Perl modules (and maybe using different programming paradigms) for you.

CPAN is not just that: it is highly organized and this includes proper documentation about the standards to be used to create useful Perl modules and make them available.

Those standards include how to add automated tests for your released modules. Those tests will be executed by:

1. the people who wants to use your modules and installed them using CPAN directly.
2. people who created smokers to test everything at CPAN automatically.

This post is specially about the later.

Anyone can became a tester for CPAN and I’ll try to illustrate the steps to become one.

Configuration to submit those distributions installed manually

In fact, this is so simple that anybody that uses CPAN should do it, including system administrators.

This wiki have almost all the details related.The tutorial is highly based on UNIX-like operation systems. You may need to do some adaptations (like, for example, using something else instead chmod to give execution permissions).

I also would add a step to upgrade the CPAN module (which is the default shell to interface with CPAN repository) before installing Task::CPAN::Reporter, following the steps below:

• Usually reloading CPAN is enough, but I already have issues by reloading it only, so exiting cpan(the CLI part of CPAN module) and entering again is more an additional guarantee than a necessary step.
• Installing Perl modules through cpan is quite easy and it will make your life easier to install modules, specially because that are additional features that you can add to it (like colorized output and command history).
• CPAN can work behind a HTTP/HTTPS proxy if you need to use one, as CPAN-Reporter does. CPAN-Reporter uses HTTPS access to send the tests results, but that can be changed to HTTP (in the case you’re in a host where openSSL/libreSSL is broken) by editing the configuration file at $HOME/.cpanreporter/config.ini, changing the protocol transport of the URL metabase.cpantesters.org/api/v1 from HTTPS to HTTP.

Creating a smoker machine

For software testing practice, smoke tests is a term to designate the possibility to execute automated tests without need of human intervention.

CPAN::Reporter::Smoker will let you do that but it’s configuration is a lot more involving.

First of all, doing smoke tests can be a dangerous thing, specially because you are downloading code from people you don’t know and don’t have the slightest idea what the code does. I never heard about a case where someone uploaded malicious code to CPAN, but while that could happen, it will probably be bugs that can damage the smoker machine.

So, to avoid that, the best is to create a Virtual Machine (VM) and there are several options to create one. I myself found that using Virtualbox is a good option (easy to setup and good enough performance) for Linux. You can use any OS that supports Perl for that.

Which OS is better? Well, Perl runs better in UNIX-like OS’s, that is not exactly a secret. Doing smoke tests are far easier at a UNIX-like box. Tests will not usually hang like in Microsoft Windows or Cygwin (I’ve tried both). Of course there are some modules that runs exclusively in some OS, and you can do some good to add more tests in those “weirdos” instead of a UNIX-like OS.

After that, consider limiting the smoker machine access to your LAN and to the Internet. Smoking tests are also possible by using an HTTP/HTTPS proxy.

This wiki has a detailed description about setting up the smoker, but again is highly based on UNIX-like OS.

There are some steps that I recommend doing before following the wiki instructions:

1. Install a custom perl (or use local::lib) to avoid root/administrator privileges: you can do it manually or use Perlbrew. I specially like using Perlbrew because I can get a local, customized perlinstallation without messing with the (probably) already perlsetup available. You can get some performance improvement by compiling perl in your machine, specially if you setup it without some unnecessary features. You can find a good tip about doing a faster Perlbrew setup here.
2. Install the CPAN::SQLite module: this will make CPAN to run faster by using a SQLite database as a back-end. Installing it requires you to have additional libraries depending on your OS of choice. Be sure to check it’s environment variables too.
3. Install CPAN::Mini: be sure to read the related Pod to get the details, specially regarding the minicpan command line program.

After following the instruction in the wiki, be sure to check the hints documentation of CPAN::Reporter::Smoker.

Smokers on OpenBSD

If you’re into OpenBSD OS and would like to run a smoker from it, I strongly recommend you to check out my project cpan-openbsd-smoker: many of the discussed steps here are applied in an automated fashion, which will allow you to get a smoker running in a few minutes without breaking a sweat.

What a heck, even if you’re not looking to use OpenBSD, looking at the available configurations for Packer, Ansible and Vagrant will probably give you some ideas to configure you’re preferred OS as an smoker. Feel free to copy them!

What about the game (in the post title)?

Well, there is a little bit of game about who submit more tests to CPAN: it is just a funny way to incentive “competition” between the testers. There is even a website to give you the details here.

The last time that I checked I was at position 16 (my user id is ARFREITAS). To give you a hint how serious people get about creating smoker machines to submit their tests I created the two following charts below, being the Y axis the number of submissions and the X axis the ranking position:

Actually I would create a single chart, but the difference between the ten first testers to the rest is so big that you wouldn’t be able to see it nicely. As you can see, getting the first place from BINGOS is really a hard task. I hope you got a cluster for that.

Conclusion

Becoming one of the greatest CPAN testers can be fun, but the more important thing is helping the Perl community doing all those tests, even if you’re not a developer. Alas, you can learn a lot about tests automation with the experience (I did myself). Also, you can consider that TAP is a big reference about software testing automation, since other languages is implementing the concept.

Anybody that released a Perl distribution on CPAN will get an e-mail about your testing report, and thus have a change to correct bugs that would be totally unexpected when the distribution was initially tested.

After all, in the spirit of open source, everybody wins by getting better code from CPAN.

Acknowledgments

Thanks to Breno (breno at rio.pm.org) about giving the details about configuring protocol transport for CPAN::Reporter.

Originally published in 2013.