BDD in Testing

In this article, we will share our experience with the implementation of BDD (Behavior-driven development) for a test automation project. This article will:

• Describe what BDD is and how to use it
• Clarify when it is an appropriate approach for a project (but not all)
• Explain why we chose it
• Indicate best practices to adhere to
• Provide practical examples of what a BDD approach looks like

We hope that you will gain a better understanding of whether this approach is suited to your project.

What is BDD, and how does it add value to testing projects

BDD (Behavior-driven development) uses human-readable descriptions of software requirements as the basis for software tests. These tests are both reusable and highly customizable. Users can tailor the tests for functional components and then re-use them to speed up the testing workflow for global components.

The most common use for BDD is for UAT tests (User Acceptance Tests). These tests are based on a user story plus business requirements and are written in a formally specified, human-readable, ubiquitous language. When we say ubiquitous language, we refer to a very limited context, with clearly defined terms and concepts that have no ambiguity. The skeleton for the tests can be written by the product owner or by a business analyst.

BDD is written in a way that it is human-readable

One of the first steps that a team must take when deploying BDD in a company is creating common definitions shared between stakeholders, domain experts, and engineers (PO/BA/QA/DEV). This process involves the definition of entities, events, and outputs that the user(s) care about and creating a standard set of definitions that everybody can agree on. This step is crucial as it impacts future development and usage of BDD.

An added benefit of using a BDD approach is that every team member (technical/non-technical) is able to understand the test record and can read and easily interpret the output of the tests.

Commonly used BDD frameworks include:

• Cucumber
• Quantum (Java Android/iOS)
• SpecFlow (.NET)
• Jbehave (Groovy/Kotlin/Scala)
• Codeception (PHP)

When is it a good idea to choose this approach

When starting a new project or when you are presented with the opportunity to switch between testing methodologies you have to consider the fact that the initial setup/deployment of a BDD project is slower and requires more effort than the use of an existing method. This is simply due to the fact that when you work on an existing test automation project, you already know the system and you can quickly get started by; downloading a test automation framework, setting up the web driver or an API gateway, selecting your existing programing language and writing your initial tests.

With a new BDD project these initial steps require more time and effort. Yes, you still need to choose a programming language/building tool and do some simple pre-setup, but some extra work needs to be done in the BDD project. For example, you need to create a shared vocabulary which is understood by each and every team member, you need to create the general functions/steps that you will start with. Based on my own experience investing time in establishing this foundation is especially important, and cannot be skipped.

The reality is that this type of project is incredibly well suited for long-term or white-label projects. With these types of projects, you will clearly benefit from all of the advantages of this solution (such as the option to re-use existing steps). However, if your project duration is roughly 1–2 years in length you can always choose a cheaper and faster solution. For the front-end part you can use existing testing tools for tests that require clicking actions on web elements (browserstack+ selenium or mabl). You can also use existing testing tools for back-end tests (such as postman or smartbear).

Indeed, the use of BDD is most suited for white-label projects that tend to re-use components. The strength of BDD is most apparent when you can re-use steps and functions. If global elements are not commonly used, then this approach is not cost-effective. BDD is not suitable for heterogenous projects and can be very costly in terms of time when applied to existing software as it is necessary to refactor existing tests. In this type of situation, it does not make sense to deploy a BDD approach, as it is simply too costly to deploy.

Advantages

• Simple English / 70+ supported languages
• Understood by non-technical team members
• Sharing of system knowledge and testing requirements across team/teams
• Cost savings over time
• Shared vocabulary

Disadvantages

• Slower deployment and challenging to deploy
• Time intensive during deployment
• Many unnecessary classes / functions / steps
• Communication is required across teams
• Increasing project complexity requires refactoring of existing steps
• Documentation

Gherkin format

Gherkin is a business readable language that helps the user describe the business behavior (test skeleton) without going into the details of the actual implementation. It uses plain language to describe use cases and allows users to extract logical details from the behavior tests. The Gherkin format is based on Treetop Grammar, which exists in 70+ languages. Therefore, your team can write Gherkin-based tests in 70+ spoken languages. For this reason, it is highly suited to distributed teams who speak multiple languages.

Important terms used in Gherkin

• Feature
• Background
• Scenario
• Scenario Outline Examples
• Given
• When
• Then
• And
• But

An example of Gherkin syntax:

Feature: Title of the Scenario
Given [Preconditions or Initial Context]
When  [Action/Event or Trigger]
And   [Join]
But   [Join]
Then  [Expected output/ Result]

Why did we choose to use this approach

Our DEV team we did not have any experience working with BDD. During previous projects, this approach was overlooked as we were not able to quantify the benefits. However, at some point in time, there was a company-wide presentation by another team who presented their use of this approach. During the presentation, our colleagues demonstrated just how effective they were when creating new tests using a BDD approach.

Based on the presentation, we knew that we had to take a closer look at this approach. The colleague had presented how they were (as a single QA user) able to create an average of 10 back-end tests per day. This was exactly what we wanted to achieve in our own team, but this was not the only reason we chose this approach.

In addition to what our colleague had presented, we chose this approach because of its practicality. At that time, we had just started working on a new project that required coverage of all possible types of automated tests (back-end/front-end/performance/security). We decided to use Cucumber as a BDD framework, as it is the most popular framework, provides free updates and has a large user support network.
We chose a Serenity framework for test management and test reports, and Kotlin as the programming language because we would be deploying automated testing for both FE and BE components (functional/integration/acceptance/e2e).

To give you some background on the complexity of the situation and to provide some rationale for our final decision to use a BDD approach, we will explain what the project looked like and how it was rolled out. The particular project involved working with teams from both Indonesia and Slovakia, and the project specification required the creation of a general white-label solution for internet banking for banks, corporations and clients.

During the first few months of this particular project, we had only one and a half QA’s on the team, and as you may know, it is quite difficult to hire people with experience in test automation. Due to this constraint, we had to be as efficient as possible in our work. As there was a lack of experienced staff we ended up hiring people with little or no experience with test automation! Here the BDD proved to be a great help as we were able to hire people to the Indonesian division who had experience with writing feature files (the skeleton for the actual tests). This helped us a great deal as once prepared, other team members were able to easily create and automate the required tests.

Business analysts were also able to help with the skeleton, especially when we talk about acceptance tests. We found that communication and efficiency improved when the teams were able to communicate in simple English, something supported by our chosen approach. Many team members were in Indonesia, and although they were able to communicate fluidly with manual testers working locally on these tests, the communication between locations was often difficult. If you have ever worked with remotely teams, distributed in far-flung countries you probably know what kind of problems arise in communication. In short, you can’t expect that all team members will speak English like a native speaker. For example, you can have an excellent developer, but can only do their job correctly if they fully understand why they should do it and fully understand all the requirements. Similarly, manual testing by QA’s located in Indonesia was often problematic as they were not able to ask the same questions as they might have been able to ask in their native language. We found that BDD is great for this type of international team because it supports the use of simple English to explain what exactly needs to be tested and why.

To summarize, all of the following points pushed us towards the use of BDD:

• New project (we started with a clean slate)
• General white-label solution
• Quick response was required for changes / new requirements
• Few QA team members
• Many business analysts.
• Non-technical people and manual testers
• Language barrier

Project structure with Cucumber

As a final point, we want to provide a basic overview of what the project structure of the automated tests look like and clarify some of the core differences between a BDD project and a non-BDD project.

In a non-BDD project you usually have one Test Class (test body) and one Step Class (logic part) but when using a BDD solution you also need to consider that these two classes are further divided into additional classes. Dare to imagine that in a BDD project we further split the Test Class and the Step Class into the following: the Test Class would be split into the Feature file + Test, and the Step Class would be split into the Step Definition + Steps.

Below you will find that we have prepared examples of how this appears in a project in practice.

Feature file -> Test assignment written in Gherkin format

Scenario Outline: Creating requests with absence of mandatory fields
When I create POST request for ACCOUNT without mandatory <param>
Then expected status code for request is 400
Then I should get status code ERROR
Examples:
| param |
| ID |
| NAME |
| CODE |

Where ACCOUNT + 400 + ERROR are parameters of the steps and param is used to repeat the same steps, omitting the ID first in the request then NAME and CODE (in the final test report, you will see it annotated as 3 TC)

Step Definition -> Functions with annotation / comprehension of text in the feature file (no logic)

Link text to code via annotation (in this example we use regex annotation)

@When(“^I create POST request for (.*) without mandatory (\\<?\\w+\\>?)$”)
fun i_create_post_without_param(endpoint : Endpoints, param : String) {
baseSteps.createPOSTRequest(URL.API_GATEWAY, path = endpoint,
bodyContent = JsonGenerator().generateJSON(accountGroup),
authorization = baseSteps.token)
}
@Then(“^expected status code for request is (\d+)$”)
fun status_code_for_request(statusCode : Int) = baseSteps.checkStatusCode(statusCode)
@And(“^I should get status code (.*)$”)
fun get_status_code(statusCode : ResponseMessages) = baseSteps.validateStatusCode(statusCode)

Steps -> Test the logic of the step definition

Test -> Composing feature file + step definition

@RunWith(CucumberWithSerenity::class)
@CucumberOptions(
     features = [“src/test/resources/features/film/create.feature”]
     plugin = [“pretty”],
     glue = [“com.name.stepdefinitions.film”,  
             “com.name.stepdefinitions.common”])
class Create

You can use more steps / definitions in the glue parameter but only one feature can be in a feature file.

Cucumber “Features” + examples

Finally, we will show you a few tricks and tips that will make it easier for you to write tests. Often these situations are overlooked at the beginning of the project. We hope these 3 things will make it easier for you to write new tests.

Background

Occasionally you will find yourself repeating some steps within all scenarios for a Feature.

Since the steps repeat in every scenario, this indicates that those steps are not essential to describe the scenarios; they are incidental details. You can literally move steps to the background, by grouping them under a Background section.

A Background allows you to add some context to the scenarios that follow it. It can contain one or more steps, which are run before each scenario. A Background is placed before the first Scenario/Example, at the same level of indentation:

Background:
Given I get token for admin from NETFLIX_EU app
Scenario: Searching fantasy film
When ……
Then ……
And  ……
Scenario: Searching anime from JAPAN for each region
When ……
Scenario: Searching anime from JAPAN for ASIA region
When ……

Regex in annotation

Cucumber allows for an alternative to Regular Expressions (https://github.com/cucumber/cucumber-expressions#readme), but in our experience, using simple regular expressions is more effective because you have much more freedom when writing global definition:

@And(“^I have (\d+) cucumbers in my belly$”)
fun countCucumbers(value : Int) = baseSteps.getCucumber(value)

(\d+) allows you to write only an integer parameter in the feature file and when the user enters the number 2.5 they are automatically notified about the mistake in the feature file.

@And(“^in each entry of (.*) array in response (was|wasn’t) displayed parameters:$”)

(.*) serves as a joker (in the feature file you can write any character/number/sign)

(was| wasn’t) after a word response you can write only was or wasn’t, another word will be evaluated as incorrect, this logic prevents this type of annotation from executing a function.

@And(“^I should get status code (\\<?\\w+\\>?)$”)

(\\<?\\w+\\>?) regex when 2 different types of string are allowed, the first is a normal string (for example: „test“), the second is if this type of parameter looks like „<test>“ as you see symbols. < and > isn’t mandatory.

Data tables

Commonly used data tables in Cucumber include the Scenario Outline. This is when the test scenario repeats itself several times. In this case, you do not initialize the data tables, Cucumber does everything for you.

Scenario Outline: Testing endpoint accessibility for update film
Given I get token for <user> from <app> app
When I create PUT request for FILM with all parameters
Then expected status code for request is 400
And I should get status code ERROR
Examples:
| user | app |
| maker | NETFLIX_EU |
| admin | NETFLIX_AS |
| regular | NETFLIX_NA |

But sometimes, you need to send additional parameters in TC where the step will not be repeated.

And in response was displayed parameters:
| name | start_date | type | episodes |
@And(“^in response (was|wasn’t) displayed parameters?$”)
fun response_display_parameters(visibility : String, 
                                data : DataTable) = baseSteps.getParams(data)
@Step
fun getParams(parameterTable : DataTable) : Map<String,Any> {
   val tableMaps : MutableList<MutableMap<String,String>> = parameterTable.asMaps()
   val loadedValues = tableMaps[0]
   val transforms = tableMaps[1]
     return loadedValues.entries.map { (param,valueKey) ->
         val loadedValue = if (valueKey.isNullOrEmpty()) "" else DataLoader.getValue(valueKey)
    val transform = transforms[param]
    val finalValue = processValueTransform(transform.toString(),loadedValue)
           Pair<String,Any>(param,finalValue)
    }.toMap()
}

In this case, as you see, you need to map this parameter into the map and introduce the type DataTable to the step definition. Yes, this takes much more effort, but again it saves time and effort in the future through the application of more general steps.

For more examples please visit page https://github.com/cucumber/cucumber-jvm/tree/main/datatable