SIMgroep was thinking on improving the release process and started writing a simple plan that prescribes the way software is developed, tested en released. Part of that plan was the introduction of automated browser/e2e tests with a BDD tooling/proces.
BDD is a second-generation, outside-in, pull-based, multiple-stakeholder, multiple-scale, high-automation, agile methodology. It describes a cycle of interactions with well-defined outputs, resulting in the delivery of working, tested software that matters.
The process looks something like this:
1. Product Owner (PO) and business analyst define features in Gherkin. Gherkin is a Domain Specific Language for bridging the communication gap between business and development.
2. PO and business analyst together with the team enrich features with scenarios in Gherkin.
3. Team agrees on a user story by its feature definition.
4. Feature definition is run against a tool parsing and executing the Gherkin (in our case Cucumber.js) and will fail.
5. The features are build and test steps implemented so that the feature passes.
So why Cucumber?
I cannot describe this better than “the why” on cucumber.io. Why should you use Cucumber? Because “Cucumber makes your team amazing”:
A single source of truth
Cucumber merges specification and test documentation into one cohesive whole.
Because they’re automatically tested by Cucumber, your specifications are always bang up-to-date.
Focus on the customer
Business and IT don’t always understand each other. Cucumber’s executable specifications encourage closer collaboration, helping teams keep the business goal in mind at all times.
When automated testing is this much fun, teams can easily protect themselves from costly regressions.
BDD with Cucumber.js, the basics
So we knew we wanted to use Cucumber.js to run our Gherkin scenarios. That would mean we would write the scenarios, run them, and validate our outcomes with an assertion library like Chai.
First off, let’s start with a scenario:
The feature part describes globally what the feature is about and is most likely the same as you use in the user story. Each feature consists of multiple scenarios. The scenarios are what’s being parsed and executed by Cucumber.js. Let’s start with implementing the first scenario:
As you can see we abstracted our step logic behind a helper service (DigidLogin) to help us login and the DashboardPage object. Making use of this kind of encapsulation is called the page object pattern and it’s used to keep the steps unaware of the underlying UI and html. This will make your tests less brittle.
End-to-end testing with Protractor
Cucumber.js only parses the scenarios but we want to test the scenarios against our app, our front-end web-application written in AngularJS. So we need something to bridge the gap. Enter Protractor. So before I’ll show the DashboardPage we’ll start with the Protractor setup.
Protractor is an end-to-end test framework for Angular and AngularJS applications. Protractor runs tests against your application running in a real browser, interacting with it as a user would.
And even better, as we found out, somebody already built a bridge between Protractor and Cucumber.js: protractor-cucumber-framework. So add the protractor-cucumber-framework to your dev dependencies
$ npm install --save-dev protractor-cucumber-framework
Next step is to create the
protractor.conf.js in the root of your project and this is what the protractor config looks like:
This will setup and configure Protractor so it uses Cucumber.js to find and execute our feature files.
Next step is writing the DashboardPage object. We use Protractor for navigating directly to a page via its browser and selecting the name from the returned html by using its element locator. Protractor has a very rich API which you can use for browsing, interact with forms and reading the responses: http://www.protractortest.org/#/api
Our DashboardPage looks like this:
Ok ready to go? No. One more thing. You might have noticed
seleniumAddress in the
protractor.conf.js. We configured our docker setup to use the default selenium hub and chrome debug node:
Chapeau, that’s all. After these steps you just need to spin up the docker boxes and run protractor. Victory.
Some things we learned
The above setup provides us with a lot of value and we learned a lot in our journey.
It takes some time
Tests are slow and since we run them in our Gitlab CI pipeline we do have to wait some time before the entire build and test process is done. With 109 scenarios and 623 steps we currently are at 12m35s. We use GrumPHP locally to force our code quality rules on every commit but we excluded running protractor because the feedback loop takes too much time.
Retry with Protractor flake
There is something called protractor-flake which makes it possible to re-run failed tests. We use it in our Gitlab CI setup to retry failed tests because occasionally there will be race conditions that will solve themselves after a retry. It’s a shame if you have to retry the entire suite so Protractor flake can save you some time.
BDD best practices prescribe to have a purely business scenario and no dependency on implementation but we end-to-end test an application. And as you could see in our scenario we use words like “visit dashboard”. That’s not really business value driven but more output/interface driven. Because of our reasons for testing we choose to have a dependency on UI although a bit smaller because of the page object pattern.
Get the entire team aboard
This entire blogpost is about implementing the tooling for executing Gherkin scenarios with Cucumber.js and Protractor, but that’s mostly a technical effort. To get the most value out of the BDD process you need to center around the Gherkin scenarios and discuss it with customers, PO, testers, developers, designers and all other possible stakeholders. Don’t make it a tech party please. And if you use Gherkin to wrap end-to-end testing on the front-end like we do, this makes even more sense because if only the developers are writing the Gherkin scenario’s it only adds another layer of complexity.
Headless Chrome came, PhantomJS discontinued. In the world of testing things are changing. New tools arrived with Cypress and Puppeteer being strong contenders. Although they provide Chrome(-ish) support only, they provide possibilities to do more powerful and much faster end-to-end testing compared to the traditional selenium/webdriver/.. etc. like setup. With the Pareto principles (80/20) in mind I believe Chrome Headless testing will gain a lot traction. One of our teams is currently experimenting with Cypress for a new project so you might read our experiences on that matter in a future blogpost.
I started out with some background on why we (and possibly you) want to use this BDD tooling setup: Gherkin parsed by Cucumber.js, executing Protractor to test our front-end. Why? Gherkin provides a single source of truth and a form of living documentation. And we can focus on customer and stakeholder business outcomes more and should do less rework.
The second part is about implementing the different tools and some example code. Following these steps should make implementation fairly easy.
And finally I tried to share some lessons learned. For me the most important lesson is to get the entire team aboard.