Tests analysis in Pharo with DrTests


Is a project intended to replace the existing TestRunner. DrTests aims to provide a plugin-based UI to deal with tests in Pharo. It will provide the same features as the actual SUnit UI (i.e., running tests, profiling tests and computing code-coverage) but will allow to plug additional analysis on unit tests.

Installation and requirements

Since we have DrTest written in Spec (framework in Pharo for describing user interfaces) It works on Pharo 8.0 and Pharo 7.0 (download the Spec new version is required in Pharo 7.0)

Load DrTests: Using the playground write the following code if you want the stable or development version and then execute it. Load stable version:

Metacello new
	repository: 'github://juliendelplanque/DrTests/src';
	baseline: 'DrTests';

Load development version:

Metacello new
	repository: 'github://juliendelplanque/DrTests:dev/src';
	baseline: 'DrTests';

DrTest UI and different plugins

Now that we have one of the two versions installed in our image we can start using DrTest, write on the playground and execute:

DrTests open

You will open the main UI of DrTest :

Every plugin can define:

  • The input ( E.g. packages, clases, methods, etc)
  • The action (run button)
  • The result/Output ( E.g. test results, coverage percent, test cases, etc)

Now, let’s talk about the plugins we have working in DrTests:

Tests Runner Plugin

As we can see in the image above, this plugin has the same behavior with the old Test Runner, execute tests. This plugin defines:

Input: Test packages + Test cases Action: Executes the selected tests cases Output: Test results by groups (i.e. errors, failures, skipped and passed tests)

But with DrTest we have an extra UI called miniDrTests. Some plugins can use this minimal interface. With the defined input we can switch to miniDrTests, it has a button to re-run this configuration and update the UI with a color and some text giving us the necessary feedback, lets see:

MiniDrTest Interface running and result

The colors mean: Green: All the tests passed without errors. Yellow: At least one failure (a failing test). Red: At least one error. Gray: all are skipped tests or they are not executed yet.

Test Coverage Plugin

Allows you to see the percent of code coverage your tests have in a package.

Input: Test packages + Package under analysis Action: Executes the test cases inside the Test packages selected and verify if this test cases execute the methods inside the package under analysis Output: Percent of code coverage (0.0%–1.0%) + List of uncovered methods MiniDrTests: Available in Development version

Tests Profiler Plugin

Shows the time (in milliseconds) it takes for each test to be executed.

Input: Test packages + Test cases Action: Executes and benchmarks the selected test cases Output: Test results by group (i.e. errors, failures, skipped and passed tests) + Time by group and time for each test MiniDrTests: Disable

Comment to test Plugin

Executable comments: In Pharo we usually use comments to add a description for packages, classes and methods, In this last case we can include an executable comment as an example of the method itself.

Method in String Class with an executable comment (the second comment)

The left side has a piece of code between parentheses. The right side has the expected result. Both divided by '>>>'.

Executable examples in comments allow the reader execute the code and understand the parameters. In addition the documentation is always synchronized because tools such as DrTests can check that examples are correct.

Input: Test packages with executable comments + classes with executable comments Action: Turns an executable comment into a test case and executes it. Output: Test results by group (i.e. errors, failures, skipped and passed tests) MiniDrTests: Disable

Rotten Tests Finder Plugin

Rotten Tests:Tests that pass and contain assertions, but whose assertions are not executed. Such tests are worse than no tests at all, because they give developers false confidence in the system under tests.”

This plugin uses the implementation in Rotten Test finder Project by Julien Delplanque and finds the rotten tests inside a test package.

Input: Test packages + test cases Action: Finds the rotten green tests Output: List of rotten green tests MiniDrTests: Disable

Last, rotten green tests are not smoke tests (its simplest form, a smoke test may contain no assertions at all). They are tests that in fact have an assertion, but this assertion is never called. You can find this kind of tests not only in Pharo, but also in other languages. for more information you can read the paper in: https://hal.archives-ouvertes.fr/hal-01819302v2

Future work

DrTests is working now in spec new version, but there are some features that we want to add:

  • Re-run a group in results (E.g. only failures)
  • Add new plugins
  • Release an alpha version incorporated in Pharo 8.0

Finally you are free to try DrTests and give us feedback! And why not? try to implement your own plugin for tests analysis too, the next blog post will explain it better ;) Stay alert! Don’t forget check other GSoC projects in Pharo.

Dayne Lorena Guerra Calle

Written by

@DayneStorm : 🎮TWITCH partner, programmer and Open Source enthusiast.