How to Automate BDD Testing in OutSystems, Part 2: The BDDFrameworks Test-Execution REST API

This article is an introduction to running tests using REST APIs in the BDDFramework, an OutSystems Forge component. It’s the second in a three-part series about testing OutSystems applications using the BDDFramework component. If you’re just getting started with Behavior-Driven Development (BDD) testing, and you’re not familiar with the BDDFramework component, it’s a good idea to read part 1 of the series, an introduction to BDDFramework.

In this article, we’re going to look at how to use the BDDFramework’s Test-Execution REST API.

In the first part of the series, we set out to test our eCommerce web application, an online wine store. We set up a test environment, created a test scenario (adding a product to the cart will correctly update the cart with the added item and its price) and implemented it in OutSystems using the BDDFramework.

How to Trigger Your Tests

To run these two test scenarios, all we have to do is open the TestECommerce web screen (it contains the tests) in a web browser. The tests are run when the page is rendered. The final state of the TestECommerce web screen looks like this:

If you want to trigger your tests, such as by an orchestration process (for instance: a Continuous Integration/Deployment pipeline,) you’ll need a way to programmatically run these tests.

Step 1: Access the BDDFramework REST API Documentation

You can use the BDDFramework REST API to run your tests. Look through the API documentation, by accessing the following endpoint, in the OutSystems environment where you’re developing the tests:

https://<YOUR_ENVIRONMENT_HOST>/BDDFramework/rest/v1/#!/v1/

Step 2: Call the Test Suite Through the REST API

You can start off by making a simple HTTP GET request to call an API method named BDDTestRunner. It passes the name of the eSpace that contains the tests and the corresponding test suite screen. The method looks like this:

rest/v1/BDDTestRunner/{TestESpace}/{TestSuiteScreen}

We can test our specific scenario where the TestESpace is TestECommerce, and the TestSuiteScreen is CartScenarios, by opening a browser and inserting the following URL:

https://<YOUR_ENVIRONMENT_HOST>/BDDFramework/rest/v1/BDDTestRunner/TestECommerce/CartScenarios

Note: When calling a TestSuiteScreen, always make sure that it can be accessed through non-authenticated requests (in Service Studio, set the Anonymous Role in the Web Screen properties.) When the BDDFramework makes a request to get the screen, it will be without authentication and as a result, it will not work if not set up this way. This is a common issue we see people running into when using the API for the first time.

If you’ve caused the API to successfully run a single test contained in a test suite screen, your result should be a JSON response file that looks like this:

{  
   "SuiteSuccess":true,
   "SuccessfulScenarios":2,
   "FailedScenarios":0,<
   "FailureReports":[  ],
   "ErrorMessage":""
}

In this example, the SuiteSuccess output is set to true. This means all the tests in the suite passed. Then, we have 2 SuccessfulScenarios and 0 FailedScenarios. And, naturally, there are no FailureReports, and since the BDDFramework was able to call this TestSuiteScreencorrectly, there is no ErrorMessage.

If we had made a mistake, for instance using a TestSuiteScreen that doesn’t exist in the eSpace, we would get an error that looks something like:

“Could not find TestSuite screen ‘WrongScreen’ for eSpace ‘TestECommerce’. Exception: The remote server returned an error: (404) Not Found.”

What Happens if the Test Fails?

To see what happens, we’ll intentionally cause one of the tests to fail by swapping “Barca Velha” for a product called “Pera Manca.” We know that when we try to add it to our cart, the test will fail because we know the store doesn’t carry this product.

As you can see, when opened in a browser and running the second scenario, the test suite fails in the BDDStep where we search for the product in the database. Let’s test the same scenario, but this time, we’ll call the test suite through the API:

{  
   "SuiteSuccess":false,
   "SuccessfulScenarios":1,
   "FailedScenarios":1,
   "FailureReports":[  
      "------------------------------------\r\nScenario\r\nAdding a product to the cart\r\n------------------------------------\r\nGiven\r\nI have a cart [P] \r\nAnd there is a product called \"Pera Manca\" [F] \r\n    [F] Condition \"Pera Manca found\" is false\r\n\r\nWhen\r\nI add the product to the cart [Skipped]\r\n\r\nThen\r\nThe operation should be successful [Skipped]\r\nAnd the cart should have been correctly updated
[Skipped]\r\n------------------------------------\r\n\r\n"
   ],
   "ErrorMessage":""
}

This time around we have 1 FailedScenarios and 1 FailureReport with the stripped down BDD output from the test that failed. Once you unescape (decode) that report, you get:

------------------------------------
Scenario
Adding a product to the cart
------------------------------------
Given
I have a cart [P] 
And there is a product called "Pera Manca" [F] 
    [F] Condition "Pera Manca found" is false
When
I add the product to the cart [Skipped]
Then
The operation should be successful [Skipped]
And the cart should have been correctly updated [Skipped]
------------------------------------

You get the exact same BDDScenario data for the failed test, as you got when running it through a browser. If you have more than one test failing in your suite, you’ll see a failure report for each failed test. We have found this to be useful when integrating with other tools, since you can include the full test output of each test report, accelerating the analysis of test failures.

The BDDFramework test-execution REST API is fairly simple to use, but as you can see, it’s also powerful enough to integrate with other tools you may use to manage your tests. In the third article of this series about testing in OutSystems using the BDDFramework, we will look at how to run data-driven tests — testing a public API — to also showcase how we can use the framework for such a scenario.