Intro to BDD style E2E testing with CodeceptJS, Docker Compose, and Semaphore CI

Challenges of UI test automation

Throughout the last few months, I have worked on a very exciting project where we used Domain Driven Design and BDD (Behaviour Driven Development) paired with a test-first approach to build our product. The greatest code coverage in the solution has been on the integration tests level with a sprinkle of unit tests for a more complicated domain in our rich models and app services. The arrangement of tests has been very much like Gleb Bahmutov’s testing trapezoid which aligns well with my philosophy on testing. The funny thing is, that the team is attempting to use the DDD approach in Python / Django projects, so compromises are to be expected — ex. lack of static types, value objects, proper encapsulation is impossible and so on… However, that is material for a separate essay.

Our approach to testing has worked well. We’ve had proper feedback on regression errors and with the introduction of a continuous integration server (Semaphore CI in our case) our workflow has greatly improved. Despite that, we lacked an additional layer that would help detect regressions on the UI level. We’re still using seasoned jQuery code and we don’t have the benefits that modern front-end frameworks and libraries offer in terms of testability, so view testing has been an area our team has avoided like the plague.

The first attempt at introducing testing on the UI level (so-called end-to-end tests or e2e) was made shortly after I joined the team. I’ve had the most experience in that area, having previously tested an Ionic 1 app with Protractor, standalone Selenium on a Jenkins build server hosted on a Mac Mini. I knew the disadvantages of that approach — an approach we have forgone for poor predictability and lack of consistent feedback (tests failing for no apparent reason while the actual UI worked). For that reason, I have been tasked with doing research on our ambitious endeavor to improve our feedback loop on regressions with E2E testing.

At that time we looked at the following libraries/approaches:

• Puppeteer
• NightmareJS + xvfb
• Cypress
• Protractor + Selenium + Headless Chrome or Chrome with xvfb
• Behave + Splinter + Selenium
• Behave + Selenium Python driver

After a lengthy analysis (including reading a great article on E2E testing with Django and BDD) and inside discussions we’ve decided to go with Behave + Splinter + Selenium. Our most important reason was to avoid additional complexity by introducing more JavaScript into the code base of our project. We’ve also hoped to reuse the specification of our integration tests in line with this quote from behave Django project documentation:

“If you want to test/exercise also the “user interface”, it may be a good idea to reuse the feature files, that test the model layer, by just replacing the test automation layer (meaning mostly the step implementations). This approach ensures that your feature files are technology-agnostic, meaning they are independent how you interact with “system under test” (SUT) or “application under test” (AUT).”

The history of our failure

We’ve encountered problems from the start. We wanted to run our tests in Google Chrome because it is used by the vast majority of our users. This made sense for testing on our local development environments, but running on a CI server required containerization. Our Docker Compose experience at that time was not yet impressive, so it was a challenge to set the services so that dockerized selenium would run HTTP requests against our web app container.

When tests started to work on local macOS environments they were still failing on Windows. When we managed to run tests on both dev OS setups, then they would give inconsistent results when running on our CI server.

We cared for reliable, predictable and automated feedback on introduced regressions and we did not want to add tests only to add more maintenance to our codebase and the tests themselves. If we had inconsistent results we would simply skip running the tests and having them would have become pointless.

After this failed attempt we have temporarily foregone our aspirations to test the front-end and continued with our focus on integration tests. The regressions in our UI remained detectable only by manual testing, which has been done for the core user scenarios every time we wanted to deploy a new version to production. Simply stated: We were defeated.

The second attempt to E2E testing

Recently I have stumbled upon an interesting project named CodeceptJS. I was intrigued by its trivial API, which is very close to the natural language and therefore follows the same idea behind Gherkin syntax and BDD. I reckoned, that even a moderately-technical person will be able to read this code and at least have a basic understanding of what is happening.

The fact that the library is written with JavaScript didn’t hurt so much anymore, cause I knew I could deploy our web app to a docker container registry and separate the codebases. This way I wanted to constrain the complexity of E2E within a separate project. Additionally, CodeceptJS provides a great abstraction layer on top of existing browser automation tools (Selenium, NightmareJS, Protractor, Puppeteer or Appium) making interacting with them consistently and pleasurable.

Toughened by our past failure and having gained more Docker experience we thought we’d give E2E testing another try.

This article will show our attempt of setting up the project based on an example of the popular open-source project Wordpress. My goal is to show how CodeceptJS with Docker Compose helped us get more predictable test results and enjoy more calm when introducing new features and new releases.

Requirements and assumptions

I assume that you have enough skill with Node and NPM to follow the instructions. If you’re an alien to this runtime, I suggest you get the code accompanying this article and follow instructions in the README file. The code is hosted on GitHub.

Configuring Docker Compose

Before we’re able to test our Wordpress site we have to configure the Docker Compose environment, so that the Wordpress service is accessible both by the host’s network as well by other containers. Firstly you’ll need to install Docker Compose. The instruction on how to do this can be found here:

https://docs.docker.com/compose/install/

When you have Docker Compose installed you’ll need to create a docker-compose.yml file, that will define our service along with all dependencies:

Below I will describe all service defined in this Docker Compose configuration and explain some of the important settings:

Chrome

This service contains the Selenium server with a Chrome browser included. This browser will by performing our tests. For our host’s network we’re opening two ports (4445, 5901), which are different than the defaults to avoid conflicts with any local selenium installations — along with one we’ll make to conveniently run E2E test in our local Chrome browser. I’m also creating a network name of selenium that can be used as an alias when connecting to this service from other containers (it can be used with CodeceptJS running in a docker container, but is not covered by this article).

DB

This is our Wordpress database running on MySQL. An important change from the default config is the mapping of our root project directory to /code/ inside of the database container:

volumes:
      ...
      - .:/code

This will later be used for restoring DB backups before starting the test.

WordPress

This is the service for our Wordpress site. In case you’re testing a different application, that can be written using any language or technology (ex. Python / Django, RoR, NodeJS) this is where you’ll provide the name of your image in a docker container registry (ex. Docker Hub) or use a local Dockerfile.

For the purpose of this tutorial I have exported port 80 on the host’s network because we want Wordpress to have the same URL when called from the local network as well as from the isolated network used by containers:

ports:
      - "80:80"

If you have a different service listening on port 80, then you’ll have to turn it off while experimenting with this tutorial. I have not been able to find how to change the exposed port in the docker image documentation for Wordpress, otherwise, I’d expose it under a different port. In your application, you’re likely to have an impact on the port you’re exposing internally in the container as well as externally. In that case, I recommend using a port different than 80, but the same for both host’s and container networks, ex. "8080:8080".

The most important however is to set the hostname under which Wordpress will be accessed by other containers — in particular, the browser running in the selenium service:

networks:
      default:
        aliases:
          - wptests

We could simply use the name of the service, but I like to keep that meaningful.

Since we want to be able to run the tests also in our local Chrome browser and Wordpress is not capable to the server from two different hosts/ports without some changes, we’ll have to map the hostname wptests in our hosts file. On a macOS system you can do it by running the following command in your terminal:

sudo echo '127.0.0.1 wptests' | sudo tee -a /etc/hosts

You’ll have to handle other operating systems yourself ;)

It would also be nice if we didn’t have to go through installing Wordpress manually every time we run the tests on a completely new virtual machine in a CI environment, so I prepared a backup.sql file, which you can find here:

https://github.com/maksymilian-majer/codeceptjs-docker-tutorial/blob/master/backup.sql

You’ll have to download it and save it in the root directory of your project.

After creating the docker-compose.yml and adding your host's file mapping you are ready to test our Docker Compose environment is ready for testing. You’ll have to run this command:

docker-compose up -d

This command will start all our services with their dependencies. All that’s left is to restore the database from the backup.sql file. You can do it by running the following command:

docker-compose exec db /bin/bash -c "mysql -uroot -psomepsw wordpress < /code/backup.sql"

After you’ve gone through all these steps you can open your web browser and make sure that it all worked properly by opening this URL:

http://wptests

When you’re running tests in a Continuous Integration environment you will want to run the two commands above at the very beginning and then you can run your E2E tests. At the end of the article, you’ll find how to do it on Semaphore CI.

In the real world

If you want to test a real WP installation you not only need to restore the database. You’ll also need all your plugins, uploads, etc. I won’t go into details, because it’s not the scope of this article and there’s an already a good resource explaining this on the web:

Easy WordPress Migration with Docker | Josh Mobley

Working with CodeceptJS

Running on your local environment and in the container

Now you should already have a working Docker Compose Wordpress service ready to be tested with any browser automation tool. It’s time to install and configure our weapon of choice: CodeceptJS. The framework has very good docs, including a QuickStart, but I want to show some more advanced configuration possibilities, that will make it easier to use in a more complex real-life project.

In order to continue, you’ll need NodeJS > 9 and NPM. I recommend using NVM (Node Version Manager) and the possibility to automatically switch node version based on the value stored in .npmrc file in the root of your project. I will be using NodeJS version > 9.

My goal is to always minimize the global dependencies to make starting the work on a project as frictionless as possible. That’s why I installed both CodeceptJS and Selenium as local dependencies:

npm i -D codeceptjs codeceptjs-webdriverio selenium-standalone

After you have installed those, you can initialize your project with the CLI that comes with CodeceptJS, however, you’d need to have it installed as global NPM packages. In our case we can add an NPM script, that will allow us to run CLI commands:

"scripts": {
   ...
   "cjs": "codeceptjs",
   ...
}

This small trick will help us call CLI commands using the following instruction:

npm run cjs [command_name]

For example, to initialize the project you’ll run:

npm run cjs init

Using the CLI is optional and it does not support more complex configuration or usage (ex. test files nested in subdirectories), so I prefer to add the tests and configuration manually to have a better structure for complex projects with many subdomains, that you’ll be testing..

My entire package.json looks like this:

I have also defined a few of useful NPM scripts, that I will explain further down in the article.

When initializing the project the CLI will create a configuration file, but the framework allows a more flexible form of configuration using a codecept.conf.js file. You should remember to only have one of these files.

In my case the codecept.conf.js file looks like this:

An interesting technique I used here is separating the configuration of helpers into files. CodeceptJS allows running tests in parallel with multiple different browser automation tools. I’m only using one, but if you wanted to add others it would soon clutter your main configuration file. For readability, it is worth the effort to separate the configuration as I did in the example.

It’s also worth paying attention to these two additional lines I added to help me run tests both using local selenium and browser as well as selenium inside of a docker container:

webDriverConfig.host = process.env.SELENIUM_HOST || 'localhost';
webDriverConfig.port = process.env.SELENIUM_PORT || 4445;

In order to avoid conflicts with our local selenium server, our container will export port 4445, so when we want to run the tests using a local selenium server and browse we need to change the port to 4444. If you wanted to use the CodeceptJS docker container to run the tests I have also added the option to configure the host of selenium server. I have forgone running CodeceptJS from a container, because most of C servers have Node and NPM, so they are capable of installing and running CodeceptJS CLI. Running tests in a container required more resources, the tests were running slowly and we often experienced timeouts. If you’re interested I can share a docker-compose.yml file, that allows to additionally run tests in the docker container.

There are to more important settings, that I’ll explain later in the article, but I’ll mention them here already:

include: {
        "I": "./steps_file.js",
        "Security": "./tests/01_security/steps.js" // allows adding commonly used steps – like login
    }

The includes section allows us to define additional parameters that will be injected into your scenarios using Dependency Injection. The first dependency is created by the CLI when you want to customize the API of the actor with custom steps. I have defined the second one to include some common steps with regards to security — in my case it’s logging in, that needs to happen before each scenario.

Once you have the basic config, you’ll need to add selenium settings in webdriver.conf.js:

One of the settings is worth noting:

url: "http://wptests"

In our Docker Compose network configuration, we’ve added a wptests alias for our Wordpress service, so that the selenium container can access it with this hostname.

The last part of our boilerplate is to add the file for custom steps definitions for I scenario parameter, i.e. steps_file.js file:

Now your project should have the following structure:

codeceptjs-docker-tutorial/
├── backup.sql
├── codecept.conf.js
├── docker-compose.yml
├── package.json
├── package-lock.json
└── webdriver.conf.js

It’s time for our first test scenario. Firstly add an empty module for the Security steps in tests/01_security/steps.js file:

module.exports = {};

Later we’ll add code to it, but it’s sufficient for now to run the other tests.

First test

This is the time to write your first test scenario in the path tests/01_security/01_login_test.js:

Feature('Login');
Scenario('Login page has Username and Password labels', (I) => {
    I.amOnPage('/wp-login.php');
    I.see('Username or Email Address');
    I.see('Password');
});

We’re starting the test from visiting the login page and checking the expected text is shown there. As you can see the API of Codecept’s actor is very intuitive and it will not pose trouble even for people with little technical skills. I highly recommend it to manual testers who want to make their first steps in automation. Ok, we got the first scenario, so let’s try to run the test suite.

We’ll start by running it in a local browser automated with Selenium. I assume, that you have followed the steps to set up Docker Compose and your WP service is up and running. It’s time to install Selenium locally:

npm run selenium:install

Next, start the Selenium server. I recommend you open another tab in your terminal app. You can also run the command in detached mode (add the -d switch), but remember to kill the process once you’re done testing. Here’s the command:

npm run selenium:start

Once Selenium server is running you can start the tests with another one our helper NPM commands from package.json:

npm run test:selenium-standalone

If you see the Chrome browser pop up and then the command outputs the following result to the console, it means we have our first success :)

Time for another trial — running the tests using the Selenium server with Chrome running in a container. You can even stop the local selenium by hitting Ctrl+C. Run this instruction in the terminal:

npm test

The result should be very similar, but you’ll notice that the Selenium server port is 4445, which is expected. If everything went as planned, then you already have the boilerplate to conveniently write and run tests locally and then launch them in Docker containers, which will come in handy when automating your tests on a CI server.

Having this basic configuration and a working E2E testing environment we can move into more interesting cases and possibilities of CodeceptJS. Let’s start by checking the validation on the login form as well as logging in with proper credentials. Your tests should look like this after adding validation scenarios:

These scenarios cover our basics as far as the functionality of logging in. It’s worth remembering, that E2E tests are costly and run for a long time. You should only write them for key scenarios and usage flows of your application. I won’t necessarily adhere to this recommendation in further examples of this article.

Don’t Repeat Yourself — extracting common steps

The login functionality we tested above is a repeatable action we will run before testing every other functionality of the WP admin panel. It makes it a good candidate to move to a common step definition, which is called PageObjects in CodeceptJS.

We’ll start by moving common code to the previously created file tests/01_security/steps.js:

If you remember well, this file has already been added to includes section of codecept.conf.js. It means, that we can inject it as a parameter to our scenarios like that:

Scenario('Invalid username or password', (I, Security) => {
    // now we can use Security.loginWith('user', 'pass');
});

After refactoring our tests with the help of PageObjects our tests will look like that:

The whole source code representing the current state of this article can be found in this commit:

https://github.com/maksymilian-majer/codeceptjs-docker-tutorial/tree/ab01c2d9e22c59664bf3c5502eda39fc872d0b69

Test preparation — Before hooks

After testing login I want to test one of the main features of WP, which is posting. For that purpose I will create another test file: tests/02_posts/01_posts_test.js.
Before I can create new posts I need to log in. I want this precondition to be fulfilled before each scenario I run.

Warning: Previously I have enabled the option to keep browser state and cookies between scenarios. Now it will no longer be necessary, so I commented this out of the configuration in webdriver.conf.js:

    // keepCookies: true,
    // keepBrowserState: true,

Now to log in before each scenario I’ll use the Before hook and our tests will look like that:

Running only specific tests

You’ll often want to work with a single scenario or only scenarios in a single file. To narrow down the scope of tests you can use the--grep parameter with its many options when running the tests using CodeceptJS CLI. For convenience, I have used the possibility to tag specific scripts and I have created a helper NPM script to only run these tagged tests.

In your package.json see these lines for details:

"scripts": {
    ...
    "test:current": "SELENIUM_PORT=4444 codeceptjs run --steps --debug --grep @current",
    ...
  },

Inside of your test, that you want to mark as the current you can add the @current tag to your feature name like that:

Feature('Posting @current');

Then you just need to fire off another of our helper commands:

npm run test:current

Debugging

One of the weakly documented features of CodeceptJS is debugging with the use of an IDE like WebStorm or VS Code. In my package.json I have configured a help command to run tests in debug mode with Node so that we can attach to it in our IDE.

You can run this command like this:

npm run test:debug

I also added a launch.json configuration for VS Code, that will allow you to attach to your Node process and hit breakpoints in the code. Once you’ve run the above command Node will wait with proceeding to your tests until you hook up your debugger.

However, I see IDE debugging as far less useful than running CodeceptJS with --debug switch and adding pause() in the code. You can read about it in the docs: https://codecept.io/advanced/#debug

Troubleshooting

In this section, I will add problems reported by readers and do my best to try and solve them.

Invalid credentials when logging in

If you use your local browser to log into WP as admin, then you can get an Invalid username or password error in your tests during a proper login attempt. I recommend logging out of all other browser sessions.

When you’re running the tests in a CI server it’s not going to happen, because they are usually launched from isolated virtual machines.

Next steps

I won’t dig deeper into the CodeceptJS API, because they are nicely documented and I have already covered the least documented sages. I wanted to show you a real-life solution you can use as a skeleton to test your own complex application.

I recommend you continue your reading through the CodeceptJS documentation: https://codecept.io/basics/

Configuring the build project in Semaphore CI

One of the main goals I mentioned at the beginning of this article is automating testing for regressions when introducing new features. That’s why I want to show you how we can configure Semaphore CI to run our tests:

1. Start by registering an account with Semaphore and connecting Github.
2. Next, add a new project by going to https://semaphoreci.com/new and picking Github:

3. Find and select your repository by clicking on it:

4. Pick your branch:

5. Select an owner for the project:

6. Wait for Semaphore to detect project type:

7. Semaphore should recommend choosing a platform based on Docker. Go ahead and select it:

8. You can skip the next step of choosing a deployment platform by clicking Skip this step link at the bottom:

9. The last step is the configuration of build parameters:

From our perspective it’s important to kill MySQL and apache2 services, so we don’t experience conflicts when listening to ports from our services in docker-compose.yml.
Here’s the whole content of the build scripts for you to use the most powerful programming pattern (Copy > Paste):

Setup

sudo service mysql stop
sudo service apache2 stop
docker-compose up -d && sleep 10
npm install
docker-compose exec db /bin/bash -c "mysql -uroot -psomepsw wordpress < /code/backup.sql"

Job #1

npm test

10. The last thing left for you to do is to hit the Build with this settings button.

11. And Voilà — your project will build on Semaphore in less than two minutes :)

Wrap-up

In this walkthrough, you have learned how to set up Wordpress and Selenium with Docker Compose services. Then you’ve learned how to test the Wordpress service using the great CodeceptJS framework. Finally, you’ve seen how to automate your tests with Semaphore CI.

You will find accompanying the code for GitHub.

If you share the love for test automation like me please leave a mark in the comments and do give some claps! 👏

Want to work with amazing startups and great teams on similar or more exciting problems? Check out ITCraftship’s Jobs Page!

Finally, if you’re looking for skilled developers, who will help you with challenges like this, check out our recruitment services at ITCraftship — candidate screening exclusively by seasoned developers like myself 😉

BTW. If you are an IONIC fan you can check out my other article on The Benefits of Continuous Delivery of Ionic Applications.