Tests analysis in Pharo with DrTests
Is a project intended to replace the existing TestRunner.
DrTests aims to provide a plugin-based UI to deal with tests in Pharo. It will provide the same features as the actual SUnit UI (i.e., running tests, profiling tests and computing code-coverage) but will allow to plug additional analysis on unit tests.
Installation and requirements
Load DrTests: Using the playground write the following code if you want the stable or development version and then execute it.
Load stable version:
Load development version:
DrTest UI and different plugins
Now that we have one of the two versions installed in our image we can start using DrTest, write on the playground and execute:
You will open the main UI of DrTest :
Every plugin can define:
- The input ( E.g. packages, clases, methods, etc)
- The action (run button)
- The result/Output ( E.g. test results, coverage percent, test cases, etc)
Now, let’s talk about the plugins we have working in DrTests:
Tests Runner Plugin
As we can see in the image above, this plugin has the same behavior with the old Test Runner, execute tests. This plugin defines:
Input: Test packages + Test cases
Action: Executes the selected tests cases
Output: Test results by groups (i.e. errors, failures, skipped and passed tests)
But with DrTest we have an extra UI called miniDrTests. Some plugins can use this minimal interface. With the defined input we can switch to miniDrTests, it has a button to re-run this configuration and update the UI with a color and some text giving us the necessary feedback, lets see:
The colors mean:
Green: All the tests passed without errors.
Yellow: At least one failure (a failing test).
Red: At least one error.
Gray: all are skipped tests or they are not executed yet.
Test Coverage Plugin
Allows you to see the percent of code coverage your tests have in a package.
Input: Test packages + Package under analysis
Action: Executes the test cases inside the Test packages selected and verify if this test cases execute the methods inside the package under analysis
Output: Percent of code coverage (0.0%–1.0%) + List of uncovered methods
MiniDrTests: Available in Development version
Tests Profiler Plugin
Shows the time (in milliseconds) it takes for each test to be executed.
Input: Test packages + Test cases
Action: Executes and benchmarks the selected test cases
Output: Test results by group (i.e. errors, failures, skipped and passed tests) + Time by group and time for each test
Comment to test Plugin
Executable comments: In Pharo we usually use comments to add a description for packages, classes and methods, In this last case we can include an executable comment as an example of the method itself.
The left side has a piece of code between parentheses.
The right side has the expected result.
Both divided by '>>>'.
Executable examples in comments allow the reader execute the code and understand the parameters. In addition the documentation is always synchronized because tools such as DrTests can check that examples are correct.
Input: Test packages with executable comments + classes with executable comments
Action: Turns an executable comment into a test case and executes it.
Output: Test results by group (i.e. errors, failures, skipped and passed tests)
Rotten Tests Finder Plugin
Rotten Tests: “Tests that pass and contain assertions, but whose assertions are not executed. Such tests are worse than no tests at all, because they give developers false confidence in the system under tests.”
This plugin uses the implementation in Rotten Test finder Project by Julien Delplanque and finds the rotten tests inside a test package.
Input: Test packages + test cases
Action: Finds the rotten green tests
Output: List of rotten green tests
Last, rotten green tests are not smoke tests (its simplest form, a smoke test may contain no assertions at all). They are tests that in fact have an assertion, but this assertion is never called. You can find this kind of tests not only in Pharo, but also in other languages.
for more information you can read the paper in: https://hal.archives-ouvertes.fr/hal-01819302v2
DrTests is working now in spec new version, but there are some features that we want to add:
- Re-run a group in results (E.g. only failures)
- Add new plugins
- Release an alpha version incorporated in Pharo 8.0
Finally you are free to try DrTests and give us feedback! And why not? try to implement your own plugin for tests analysis too, the next blog post will explain it better ;) Stay alert!
Don’t forget check other GSoC projects in Pharo.