Continuous Behavior Driven Testing

Behavior-driven testing (or BDT) is an agile software testing technique that encourages collaboration between developers, QA and non-technical or business participants in a software project. We are using here python behave.

Plan > Code > Build > Testing > Release > Deploy > Operate > Monitor

CI/CD flow using BDT

Background:
In Behavior-driven testing, an application is documented and designed around the behavior-driven a user expects to experience when interacting with it. Behavior-Driven Testing (BDT) is an uncommon term in software testing/development compared to Behavior Driven Development (BDD). The main purpose of Behavior-driven testing is that every time a new feature is ready for testing, all current steps have to be checked to see if they can be used. The reason we end up releasing a buggy product is that we don’t focus on how an end-user or customer will use it, instead, we are worried about testing methods or individual APIs.

Aim and Objectives:
The main aim of this research is to propose continuous behavior-driven testing in the CI/CD pipeline.
The objectives are formulated based on the aim of this study which is as follows:
1. To Increase the frequency of product releases or updates.
2. To automate continuous behavior-driven tests in CI/CD pipeline.
3. To generate test reports which should be readable by everyone.

Python-style feature file:

Behave is behavior-driven development, Python style. Here is an example of a Python-style feature.

Visit this page to find out more about how Python behaves: https://behave.readthedocs.io/en/latest/

@UserAPI
Feature: Create a user using  RESTful API
    As a user
    I want to use user API for user CRUD operations
    So that I can create, retrieve, update, and delete users using HTTP requests.
Background:
        Given user API endpoint
        And a customer's first name
        And a customer's last name
        And a customer's gender
@CreateUserPositive
    Scenario Outline: 001 Create a user with a valid email and password
        When I enter user email id <email>
        And I enter the user phone number <phone>
        And I enter password <password> for the user
        And I send a POST request to the user API
        Then I should not see the error message
        And I should get a status code of "<status>"
        And I should be able to request GET for the newly created user
        And the response value of "email" should equal <email>
        Examples:
            |email       |phone      |password    |status |
            |"xxx@xx.com"|1234567899 |"abc12345"  | 200   |
@CreateUserNegative01
    Scenario Outline: 002 Create a user with an invalid email and password
        When I enter user email id <email>
        And I enter the user phone number <phone>
        And I enter password <password> for the user
        And I send a POST request to the user API
        And I should get a status code of "<status>"
        Examples:
           |email        |phone      |password   |status |
           |"xxx@xxx.com"|1234567899 |"abc12345" | 406   |

Different ways to trigger BDT:

1. To trigger all tests at once.

behave

2. To trigger tests for specific features.

behave features/user_api.feature

3. To trigger a test for a specific tag in a scenario.

behave features/user_api.feature --tags=CreateUser

4. To trigger tests for multiple tags in a feature.

behave features/user_api.feature --tags=CreateUser,UpdateUser

5. The command shall not execute the Scenario which is tagged with UpdateUser.

behave features/user_api.feature --tags=~UpdateUser

Jenkin CI server BDT integration:

In order to fully automate the process, Jenkins offers us a variety of interfaces and tools.

To test the CI/CD pipeline, we need to create a Jenkinfile based on Groovy, and we can configure all stages of the pipeline.

you can see the stages of the pipelines are as follows:
Checkout => Build & unit test => Deployment => BDT test run and Report generation

Jenkins for BDT

BDT success test report:

Let’s start by reviewing BDT test reports on the areas where BDT will be successful. Here you can see both successful and unsuccessful test cases where a user was created with all correct inputs and a user was created unsuccessfully with invalid inputs. We also verified that the failure should occur with status code 406. It shows that our feature is prepared for end-user distribution.

CI/CD Pipeline Output

BDT failure test report:

If the report quality score is low, then the content is not publishable. We can’t ship this new feature because it breaks a core requirement. Let’s take a moment to deconstruct how this BDT test report will help us to decide not to release this new feature. Before we release a new feature, we always run old tests to make sure that our basic feature is working. If it’s not, then we don’t release new features, because we don’t want to impact the performance of our end users.

Conclusion:

Automating behavior-driven testing (BDT) with the continuous delivery pipeline (CD) will boost the DevOps process. It has historically been challenging for our DevOps teams to keep up with rapidly changing user scenarios and requirements in the fast-paced environment of DevOps and Product releases.

The final benefit of including BDT in our software release approach is creating a more inclusive environment where all project stakeholders are engaged more frequently and meaningfully.