Testing in WordPress plugin development
Testing is part of the software development basics and a way to make sure the code you’re writing does what it’s supposed to do. There are many ways to test. I’m focusing here in unit testing and integration testing.
Unit testing means individual components (a single function) while integration testing is about testing larger parts of software to see how well everything works together. In WordPress context these two are often unnecessary and even difficult to tell apart. A third form of testing is regression testing, which basically means running old tests again to ensure new developments haven’t broken anything that used to work before.
I’m writing from a Mac OS user perspective, assuming familiarity with Homebrew and Composer. I’m sure Linux users can use their own package managers when necessary, Windows users I have to leave on their own devices unfortunately.
I’m a plugin developer, so that’s my context (I’m the author of Relevanssi and Relevanssi Premium search plugins). These testing methods can likely be applied in other kinds of projects, including site development where lots of custom code is involved — and often implemented as a custom plugin.
Testing tools
PHPUnit. The basic tool for testing WordPress is PHPUnit. The latest version is 8, but at the moment you can’t use the version 8 for testing WordPress, so start with installing the version 7 of PHPUnit.
Wp-test-framework. In order to run the tests, we need a testing framework. Otto Rask has developed a neat framework that will take of care of everything for us. wp-test-framework is easy to include as a Composer dependency:
composer require --dev rask/wp-test-framework
Once you’ve defined the dependency in your composer.json, you can download and install the framework with:
composer updateWordPress. Of course we also need a copy of WordPress. You don’t have to actually install it, the test framework takes care of that, we just need the files available somewhere. With WordPress CLI, downloading the files is this easy:
wp core downloadWithout WP CLI, it’s almost as simple, as long as you have wget installed:
wget https://wordpress.org/latest.zip
unzip latest.zipNow all you need is to add a wp-tests-config.php file in the WordPress directory. The file sets the necessary configuration. You can find a sample config file from the WordPress repository. All you need to change in the sample file is the database settings.
The environmental variable WP_TESTS_INSTALLATION is then set to point to the directory where you downloaded WordPress, like this:
export WP_TESTS_INSTALLATION=/Users/msaari/wordpress/MySQL database. Now only one piece is missing, the database. The tests require a database server. If you already have one running (for example you’re running MAMP), use that, otherwise MariaDB is easy to install:
brew install mariadb
mysql.server startCheck that wp-tests-config.php has the correct database settings. Note that the tests must run in a separate database, because the first thing the tests do is to wipe out all the tables in the database.
PHPUnit settings
Now that we all the pieces of a working test environment together, we can start building the actual tests. Create a phpunit.xml file in your project root that defines where the tests are. The file from Relevanssi looks like this:
The bootstrap setting is important, because it defines where PHPUnit will look for bootstrapping instructions to start up the test environment.
The testsuite configuration defines that the tests are found in directories tests in files with names beginning with test-.
Bootstrap file
The tests/bootstrap.php looks like this:
require_once './vendor/autoload.php';
\rask\WpTestFramework\Framework::load();require_once dirname( __DIR__ ) . '/relevanssi.php';
The first two lines include and start up the wp-test-framework and the third line includes Relevanssi. This is how you install plugins in the test environment.
Writing the tests
The test files are classes that extend the WP_UnitTestCase class. The test setup is done in a public static function wpSetupBeforeClass(), the clean up is done in wpTearDownAfterClass() and in between PHPUnit runs all public functions with names beginning with test.
In the case of Relevanssi, each class starts by running the Relevanssi install function relevanssi_install() and then the setup function relevanssi_init(). The setup function would normally be triggered by the init hook in WordPress, but here that hook doesn’t run, so the function needs to be called manually. In the tear down function, Relevanssi is uninstalled with relevanssi_uninstall(). This makes sure the different tests don’t interfere with each other.
The classes are split up based on what they test. They roughly correspond to different files in Relevanssi: one class tests searching, one class tests indexing, one class tests excerpts and highlights and so on.
A single test function uses Relevanssi functions to do something and then makes assertions on return values. The most common assertion is $this->assertEquals(), which complains if the actual value doesn’t match the expected value. In unit testing you run a single function that does something. You know the expected return value with different outputs, and then you just compare to see if that’s true.
Tips for writing tests
If everything is built up from small enough functions, writing unit tests for them is fairly straightforward. Testing encourages you to split up big functions into smaller parts, which are then easier to test. In these cases, all sorts of preparation work isn’t as necessary — all you need is to give the function the right kind of input.
When testing WordPress plugins, there’s usually some need for bigger integration tests. With Relevanssi, it’s good and useful to test single functions, but it’s also good to test that the whole search process goes through, or that the database indexing works as expected. This requires preparations and creating posts that can be searched and indexed.
You can use WordPress functions in the tests to create posts and other content. Using wp_insert_post() to create posts works well. The WordPress test package contains some factories and helper functions that make creating content even easier. For example, to create ten posts, you can use
$post_ids = $this->factory->post->create_many( 10 );Now $post_ids contains the IDs of ten new posts. There isn’t much documentation for these functions, so you need to go to the source code to get to know these.
Test functions can depend on each other. If you define a dependancy by adding @depends test_another_function() in the function comment block, this function will be run after the test_another_function() and it will receive the return value from that function as the parameter.
The tests may also accidentally affect each other. Make sure you spell out your expectations regarding plugin settings and reset the settings you adjust for tests, otherwise you’ll end up with cryptic problems when a test passes when run alone, but fails when you run the whole test suite.
Here’s an example test function that tests image attachment indexing setting in Relevanssi:
The test first deletes all posts with the $this->delete_all_posts() helper function, then creates two posts. One post is an image attachment, the other a PDF attachment — the actual attachment files are not included, because they are not necessary for the test. Relevanssi is set up to index the attachment post type and the image attachment file indexing is disabled.
The index is now built with relevanssi_build_index() function. If everything works well, $return['indexed'] will have the correct number of indexed posts, 1. Then image attachment file indexing is enabled, index is rebuilt and now the value should be 2.
What to test?
When I started, I didn’t quite know what to test. I started by building tests for the most common functionality and for bugs I came across. Then I learnt about code coverage, which means looking at which parts of the code are covered by the tests, and after that my goal was to reach 100% code coverage.
It’s not a completely meaningful goal, but it has proven to be useful. To reach 100% code coverage, I’ve been going through the Relevanssi codebase and have refactored it a lot, and while at it, I’ve found lots of bugs and faulty logic — which is not a surprise when you consider that the Relevanssi codebase has existed for a decade and has lots of old code still in it.
PHPUnit can generate code coverage reports. There are couple of different methods, I use this one:
phpunit --coverage-html coverageThis generates a HTML report in the coverage directory. The report includes the Change Risk Anti-Patterns index (the CRAP index), which is based on cyclomatic complexity and code coverage and gives one view on how difficult the code is to understand. I’m sure it’s not the only truth, but based on my experience, code with a smaller CRAP index is much more readable and easier to maintain — and more pleasant to look at.
What the Relevanssi test suite doesn’t work at all right now is the compatibility with other plugins. I may get there at some point, but including other plugins in the test suite is difficult enough that I’m not there yet.
If you want to take a closer look at the Relevanssi test suite, you can find in the GitHub repo, which includes all the tests in the free version. The Premium version is also shipped with a full test suite that includes more tests that cover Premium features.
The testing process
When I work at Relevanssi, I run tests constantly. If I create something new or modify something old, I’ll run phpunit to see that nothing breaks. When I create new functionality, I also write tests for it. When I refactor old code to improve the code coverage, I run phpunit constantly.
Releasing new versions also requires testing. Before I release a new version, I will run a test that installs the package and then runs tests on it, to make sure the install package is functional. In the Premium release process, this is done by releasing the new version with a version number of 0 — a version 0 can be installed with a magic API key. With the free version, I tag the new release at GitHub, which lets me install it with Composer.
I have a test directory on my computer with a composer.json that looks like this:
With this file, I can then run this script:
#!/bin/shrm -rf vendor
composer clear-cache
composer updatecd vendor/relevanssi/relevanssi-premium/
composer update
sh multi-version-test.shcd ../../msaari/relevanssi
composer update
sh multi-version-test.sh
This script first deletes any old versions by removing the vendor directory, then clears the Composer caches and downloads the new versions. Then the script moves on to the Relevanssi Premium directory, runs composer update to install the wp-test-framework and then runs a test script that runs the tests with multiple WordPress versions. Then the same is repeated for the free version.
If the tests pass, the new versions can be released: Premium by changing the version number 0 to the actual version number and the free version by pushing it to the WordPress.org SVN repo.
Testing with multiple WordPress versions
I’ll elaborate this step a bit more. I used to test with just a single WordPress version — the latest one, if I remembered to update the test WordPress — but then I realized I could test against multiple versions with just a little more effort. If the different WordPress versions are in separate directories, all it takes is changing the value of the WP_TESTS_INSTALLATION environmental variable between the tests.
I ended up writing a fairly complicated script in order to run the tests and ended up with this:
This script assumes you have a test directory where the WordPress versions will be set up and where you already have a configured wp-tests-config.php file available. Then all you need is to define the lowest WP version you want to test.
The script reads the currently available versions from the WordPress API and returns a list of the latest version in each series — right now it would be 5.3, 5.2.4, 5.1.3, 5.0.7, 4.9.12, and so on.
For each version the script checks if it has been downloaded already. If the directory — say wordpress-5.2.4 — does not exist, the script fetches the installation package, unzips it, moves it to the right place and adds in the wp-tests-config.php file.
If the version existed or could be installed, the tests are then run. If the multisite testing config exists (which means we’re testing Relevanssi Premium) and the WP version is at least 5.1, the multisite tests are run, otherwise the single site tests are run. Tests are run as --stop-on-failure which makes the process faster.
The oldest version that can be tested is 5.0, because wp-test-framework makes expectations that were included in 5.0.0. It would probably make sense to test the 4.9 versions, but I assume 5.0 generally matches 4.9 well enough, and people running 4.9 should really move on to version 5 and Classic Editor already.
Automatic tests
It’s of course possible to make things even better. Now I have to run all the tests manually. It would be better if the tests could be run automatically whenever a new version is pushed to the version control system. There are tools for this, but so far building a working WordPress and PHPUnit testing setup in GitLab CI pipeline has proven to be too difficult for me.
If someone has done that, I’m all ears, but until that, I’m happy with the current process. It’s simple and fast enough to make testing so easy that I’m willing to do it as an integral part of the development process.
