How to Automate BDD Testing in OutSystems, Part 1: An Introduction to the BDDFramework
We’ll be introducing the BDDFramework, an OutSystems Forge component in this blog post. It’s the first of a three-part series into testing OutSystems applications using the BDDFramework component. BDDFramework is a test-automation framework where tests are specified using the Gherkin syntax. Gherkin is a human-readable language for structuring and describing an application’s expected behaviors. The resulting scenario is then used to build test-automation for your OutSystems Applications.
The main purpose of BDD frameworks is to support Behavior-Driven Development (BDD), where all technical (for example, developers) and non-technical (for example, business analysts) participants in a software project collaborate to define a common understanding of how the software should behave.
Organizations that don’t follow the BDD processes, can also take advantage of BDD Frameworks. During software development practices like continuous integration/continuous delivery (CI/CD), people analyze a variety of test results on a daily basis. An efficient development and quality practice requires that everyone involved understands the test results. Many quality professionals and developers find the structural nature of the Gherkin syntax very appealing. It provides a standard for everyone to follow when setting-up certain levels of test automation. The tests serve as their own living documentation, as a consequence, BDD is sometimes called specification by example (SBE).
The BDDFramework component can be used for Web and Service Applications, the Server component of Mobile apps, as well as REST and SOAP APIs. It’s comparable to frameworks in other technologies, such as Cucumber, JBehave or SpecFlow. You can use the BDDFramework with different purposes and goals in mind, depending on what you need for your specific contexts. In this article we will show you how to build your first automated test scenario in a Web Application available in the OutSystems Forge.
Laying the Test Groundwork
Step 1: Set up Your Test Environment
To get the BDDFramework component, you can go to Service Studio, select the Forge tab, search for the component and install it.
Now that you’ve installed the BDDFramework, it’s time to create our first test application. In this tutorial, we’re going to create tests for the eCommerce application. Install the application from the Forge, as you did for the component.
Now we need to create a new application which will contain the automated tests. We’ll call it TestECommerce. We recommend that you implement the tests in a separate application. This will give you more flexibility when deploying applications between different environments. For instance, it will allow you to avoid running test code in a production environment.
Step 2: Understand the BDD Test Scenario
Before diving into building an automated test scenario using the BDDFramework, let’s understand it from a user’s perspective.
Open the eCommerce Web Application in a browser. You’ll see that it’s an online wine store. The test scenario is simple: adding a product to the cart will correctly update the cart with the added item and its price.
In the homepage, the user adds a bottle of Prosecco Armani DOC to the cart. The user then goes to the cart to verify that the correct wine, quantity and price have been added.
Step 3: Translate the BDD Test Scenario to Gherkin
This specific interaction can be translated into Gherkin, and used as a test scenario in a test-automation framework. With Gherkin, we’ll structure the interaction (Scenario) into groups of steps using the following Gherkin syntax: Given, When, and Then.
Here’s a translation of what we’re seeing above:
Scenario — describes the specific scenario that illustrates a business rule (adding a product to the cart.)
Given — describes the initial context of the scenario — the required pre-conditions we need in place before conducting the action/event that we are testing (in this case, we should have a virtual shopping cart and a specific product to add.)
When — describes the specific action/event — in many scenarios there should only be one such step (for example, adding the product to the cart). If you find yourself having to add more than one step here, you should consider if you need to break up the scenario into two or more.
Then — describes the expected outcomes of conducting the action/event in the system. These steps commonly contain various assertions that verify everything we want to check as a result of this test.
In this syntax, the scenario should be clear to anyone who reads it, whether they are technical or non-technical participants. This clarity is a highly valued characteristic that the BDDFramework maintains throughout the process of designing, implementing and running test scenarios.
Implementing the Test With BDDFramework
We’ve defined the scenario, and now we can start creating an automated test that can be automatically run using the BDDFramework. In this framework, the scenarios are defined in web screens using web blocks, and the logic for each group of steps is implemented in the screen actions.
Step 1: Create the Scenario and Steps with Web Blocks
In Service Studio, start by using the Manage Dependencies window, to include the BDDFramework public resources into the test eSpace (eCommerceTests).
The framework includes four web blocks you can use to build your tests:
- BDDScenario — each scenario is represented by a BDDScenario web block.
- BDDStep — each group of steps is represented by a BDDStep web block.
- FinalResult — returns stats about all scenarios run on the web screen (count successful tests, count failed tests, and so on.) It should always be included at the end.
- SetupOrTeardownStep — is a special kind of step that can be included in scenarios (before or after the actual test logic) to perform setup/cleanup operations of data that is outside of the scope of the scenario from a functional/business perspective (for instance, delete test data that was generated during the test.)
Add a BDDScenario to the screen and fill out the placeholder with the Scenario description.
Next, add BDDSteps (includes the Given, When and Then steps) and fill out each step description. In the end, you should have something like this.
The entire scenario is now implemented inside a BDDScenario from the framework. We also added the FinalResult block at the end of the page (the “No Scenarios Executed” block), because as previously mentioned, you should always include this web block for each screen that’s tested.
Step 2: Implement the BDDStep blocks as Screen Actions
It’s time to implement each BDDStep as a Screen Action. In this test, we will be interacting with the eCommerce application and its eSpace through public actions and entities. Regardless of our intention to put in place the automated tests, we should always follow the best practices for OutSystems application development. Therefore, the application’s core functionality should be implemented in clearly defined steps and not, for instance, in Screen Actions. Or, we can also have BDDFramework tests that interact with REST or SOAP APIs from an OutSystems application, and even Service Actions.
To import the Screen Actions into the TestECommerce eSpace, we need to make them public in the eCommerce eSpace.
Now we’ll import the eCommerce Entities and because we’ll need to access them from TestECommerce later on, we need to make them public before importing them.
We’re ready to implement the Screen Actions for all groups of steps: Given, When, Then.
The first step of the Given group is “That I have a cart”. We start implementing the step by right-clicking on the corresponding step in Service Studio. We then choose “Select OnNotify destination”, followed by creating a new screen action.
We are calling the Cart_CreateNew action from eCommerce to create whatever the application needs to have a cart. The output of that action is a CookieID. Now we perform an assertion (AssertTrue from the BDDFramework) that validates if a non-empty CookieID is being returned, thus validating that the operation was successful.
If the operation fails, and we hadn’t created this assertion, we won’t get a valid CookieID as output, and it will be harder to pinpoint where in the test steps the failure has occurred.
The BDDFramework provides a set of assertion actions we can use, each one has a different purpose (true statements, false statements, values, etc) — it’s important that we select the appropriate one for each situation to improve code legibility and maintenance.
You probably noticed that we store the CookieID output from the Cart_CreateNew action in a web screen variable. This is because we’ll need this ID for other operations in other BDDStep actions and we pass data between them by using this sort of variables or records in the database.
Now we’ll implement the second step of the Given group “And there is a product called ‘Prosecco Armani DOC’ ”.
The first step is to create the following Screen Action:
We must validate that there’s a product in the database called “Prosecco Armani DOC” by performing a query to search for it, followed by using an assertion (AssertTrue from the BDDFramework) to validate that the result is not empty. We also store the ProductID of that record because we will need it further along.
Now we’ll implement the “I add the product to the cart” step of the When group.
In this step, we call the Cart_AddProduct action of the eCommerce eSpace — which holds the functional logic that is the main focus of this test. We pass the parameters that define which product we are adding, the quantity (1) and the cart we’re using.
Next we store the results of that call: whether it was successful, the output message and the FinalQuantity — all of these are stored in variables so that we can proceed to the Then steps.
An important note about Cart_AddProduct:
There is a testability issue with this specific action in the eCommerce, in the sense that it always uses the cart ID that is stored in the current session. This ID is managed internally in eCommerce for each session, which makes it hard to test the cart mechanisms. As such, we must make a minor change to Cart_AddProduct in eCommerce for it to alternatively receive an optional CartCookieID that overrides the current session’s cart ID:
In the first Then step “The operation should be successful”, we assert that the output data from the previous action was correct:
As for the last Then step “And the cart should have been correctly updated”, we also want to validate that the cart data in the database has been correctly updated.
We start by performing a query that includes all the data for purchase orders in the cart with the CartCookieID we stored and then conduct three assertions. In the first one we validate that there is only one product in the cart. In the second assertion, we check that the name of that product is the expected one: “Prosecco Armani DOC”. Finally, we verify that there is only one item of that product ordered.
With this last action implemented our test is finished and we can publish TestECommerce.
Step 3: Run The Test Scenario
The test scenario we just created runs when rendering the web screen where it was created. If you open it through a browser, this is what you’ll obtain.
As you can see, the output is very similar to what you see in Service Studio when building/changing the BDDScenario. This makes it very easy for someone to jump from the test run report (the rendered web screen) into the implementation of the test.
You’ll probably notice in the above image that there are several green check marks along the BDDSteps. like this:
The BDDFramework will output one of these symbols for each assertion that is performed in the step — this is why the two last steps have 3 marks each. If there are no assertions in the step, the framework will always output one green check mark nevertheless, just to signal that the step was successful.
Let’s now check out what happens when one of the assertions fails. Imagine we edit the “Operation should be successful” Then step and change the AssertOperationMessage to expect the wrong message:
Running the test now in the browser, here is what we get:
The test now clearly indicates that in that step there were 3 assertions, but the second one failed. It also will inform you that it expected a specific value for “Operation Message” but obtained a different one. Once a step fails in a BDDScenario, all following steps will be skipped, as you can see in the image above.
In our experience, having this sort of information regarding the execution of the test really speeds up its troubleshooting, because anyone who is confronted with this output will be able to quickly:
- Understand the full test scenario.
- Identify in which step it is failing.
- Get information on what were the expectations that were not met by the system behavior.
- Jump into the portion of the test code where it is failing.
Finally, let’s also see how BDDFramework tests behave when there are unhandled exceptions. Imagine now that we trigger an exception in the same point of the code where we put the failed assertion:
As you can see, the framework indicates that the first assertion was successful (there is a green check mark) but then there is another symbol indicating there was an unhandled exception and what the exception message was. Similarly to a failing BDDStep, in this case the framework won’t run any steps after this one.
Step 4: Teardown the Test Data
Going back to the BDDScenario block, you may also notice there are two grey areas stating “Click to add Setup/ Teardown” when you hover them with the mouse pointer. You can drag a SetupOrTeardownStep into these ones. One very common use case for such a step is to include it at the end of the scenario to perform a teardown (deletion) of data that was generated during the test.
In fact, in the example we’ve been building, we create data in the eCommerce database for a shopping cart used in the test and it is highly recommended that we delete it in the teardown step. Here’s an example of how the action associated with that step could look like:
As you can see from this simple logic, if a CartCookieID was saved during the test, we access all records of the PurchaseOrder entity that are identified with that ID and delete them (should only be one such record). This ensures us that the test won’t leave data behind and puts the system back into the state where it was before it was run — a best practice when building automated tests.
Running Several Tests Inside one Screen
To wrap-up this introduction to the BDDFramework, let’s see how it is possible to run several test scenarios (a test suite) inside the same web screen. When setting up a test suite, it’s best practice to build each BDDScenario in a separate web block inside your test eSpace — this allows you to better organize different scenarios and have the corresponding implementations (the Screen Actions) grouped together. As you can see in the following animation, we’ve copied the test into a Web Block. We also created a new test, similar to the first one but where we try to add another product called “Barca Velha” to the cart:
Once you have the two test scenarios in the same screen, you can run them on the same page.
The two scenarios are run in sequence and at the end the FinalResult web block indicates that all scenarios were successful. This is useful when you want to group tests in one test suite screen performing the tests on the same feature or action. So there we have it, in this article we’ve looked at Gherkin, written a test scenario with Gherkin, implemented that test scenario using the BDD framework and ran several test scenarios inside a single screen. In the second article of this series about testing OutSystems using the BDDFramework, we will look at how to run tests with the BDDFramework’s REST API.
