Smoke testing APIs using MuleSoft Blackbox Automated Testing (BAT)

Running smoke tests after deployment is a common practice of API testing that focuses on quickly and superficially checking basic API functionality.

Smoke Tests

The term “smoke test” originates from electronics testing, where a device is turned on for the first time, and if it doesn’t emit smoke (indicating a major failure), it’s considered a basic test pass.

In the context of API testing, smoke tests serve as a preliminary check to ensure that the API is operational and that critical functions are working as expected. These are usually executed right after a new build or deployment, to catch issues (e.g. connectivity to downstream systems, databases, etc.) before more extensive testing is performed.

Typically smoke tests involve checks related to endpoint availability, HTTP status codes and responses (e.g. 200 OK, 404 Not Found, 500 Internal Server Error), authentication and authorisation, data validation, and basic functional scenarios. Other checks, related to non-functional requirements can also be performed such as response time checks, error handling, resource cleanup (when API creates temporary resources), integration points, and logging and monitoring.

The goal of smoke testing is to quickly identify major issues that could prevent further testing or development progress. If smoke tests pass, it’s an indication that the API’s fundamental functionality is intact, and more comprehensive testing can proceed, such as functional testing, integration testing, and performance testing. If smoke tests fail, development teams can address these critical issues before spending additional resources on in-depth testing.

Automated Testing

There are lots of specialised and well-known tools that perform automated API testing, some of which are open-source and free, some others that are license-based, more os less expensive, etc, mostly capable of being utilised in scenarios of continuous deployments (aka CI/CD pipelines).

A few common pain-points across these is that they all have their own way of writing tests and assertions, dependencies, and requirements for installation and making it difficult (or almost impossible) to export/import to a different one, and also it forces architects and developers to go through the learning curves on each, becoming specialists on this or that tool.

In MuleSoft’s Anypoint Platform, at the very core of its API Functional Monitoring is the Blackbox Automated Testing (aka BAT) CLI, which is the focus of this article, and a great choice for smoke testing MuleSoft applications that expose REST APIs.

Blackbox Automated Testing (BAT)

Using BAT CLI, MuleSoft architects and developers can automate testing and get test reports. These automated tests can be embedded as part of a continuous deployment pipelines, and executed after deployment of APIs in order to verify functionality and validate the results that the API generates.

BAT tests are written based on Behaviour Driven Development (BDD) syntax, which is an embedded domain specific language (EDSL), very similar to and processed in DataWeave, making them particularly friendly and comfortable for MuleSoft architects and developers.

Structure

The structure of BAT test suites look as the below:

.
├── bat.yaml
├── config
│   ├── default.dwl
│   ├── devx.dwl
│   ├── qax.dwl
│   └── stgx.dwl
└── tests
    └── HelloWorld.dwl

File bat.yaml is the main in a test suite, where its configuration is declared, including result reporters, which can be set to a number of different outputs, ranging from from HTML files and emails to more specific ones such as Slack WebHooks, New Relic, and Pager Duty.

Directory config contains configuration files used by the tests, where architects and developers can have different endpoints, credentials, etc., separated by environment or any other classification needed.

Directory tests contains one or more files that contain the actual sets of tests and assertions.

For illustration purpose, consider the following RAML snippet of a Daily Tasks API, which has a single resource endpoint /tasks with a GET method that returns a simple list of tasks containing a description and week day:

/tasks:
  get:
    responses:
      200:
        body:
          application/json:
            example: |
              "results": [{
                  "description": "Specify API and Implement API",
                  "weekday": "Monday"
                }, {
                  "description": "Deploy and Smoke Test API",
                  "weekday": "Tuesday"
                }
              ] 

Assuming that the API runs on localhost and is reached by pointing to http://localhost:8081/api/v1/tasks, which always works fine and always returns an array with 5 elements (1 for each working day of the week), the following exemplar smoke test could be performed to ensure the API is running and its basic functions work:

import * from bat::BDD
import * from bat::Assertions
---
suite("Daily Tasks Test Suite") in [
  it must 'Get 200 response and a Collection of 7 daily tasks' in [
    GET `http://localhost:8181/api/v1/tasks` with {} assert [
      $.response.status mustEqual 200,
      (sizeOf($.response.body.results) == 7) mustEqual true
    ]
  ]  
]

As seen above, the syntax is very similar to other frameworks such as Mocha and Jasmine and very much familiar to DataWeave model.

MuleSoft has an extensive documentation on API Functional Monitoring and more specifically around BAT and BDD syntax, which can be found at: https://docs.mulesoft.com/api-functional-monitoring/bat-top

Practical Example

Putting all of that in practice, let’s implement an exemplar solution comprised by 3 APIs and some BAT tests to validate that the APIs are up, running and behave as expected. For that, consider a very simplistic solution, where an experience API named Industry Programming Languages API makes calls to and combines data from an API named Programming Languages System API with another API named Industries System API. The experience API has 2 simple resource endpoints, both accepting GET method:

• /industries
• /industries/{industry}

These respectively return a list of industries, or one specific industry and their most used programming languages including details such as file extension and paradigm. For example, in financial and big data applications, Python is most likely the language of choice, whilst in the automotive industry, C-like languages such as C++ reign supreme.

The below diagram illustrates the solution from a very high-level view:

Design and Implementation of APIs

Designing the system APIs first, we would have the following RAML specs:

Programming Languages System API

#%RAML 1.0
title: Programming Languages System API
description: This demonstration API lists programming languages
version: v1
types:
  programmingLanguage:
    properties:
      name:
        type: string
        example: "C"
      paradigms:
        type: string
        example: "procedural, structured"
      fileExtensions:
        type: string
        example: ".c, .h"
    example:
      name: "C"
      paradigms: "procedural, structured"
      fileExtensions: ".c, .h"
/programming-languages:
  displayName: Programming Languages
  description: Programming Languages
  get:
    description: Get a Collection of Programming Languages
    displayName: Get a Collection of Programming Languages
    responses:
      200:
        description: Collection of Programming Languages
        body:
          application/json:
            type: programmingLanguage[]
            example:
              - 
                name: "C"
                paradigms: "procedural, structured"
                fileExtensions: ".c, .h"
              - 
                name: "BASIC"
                paradigms: "non-structured, later procedural, later object-oriented"
                fileExtensions: ".bas"             
/programming-languages/{programmingLanguage}:
  displayName: Programming Language
  description: Programming Language
  uriParameters:
    programmingLanguage:
      type: string
      example: "Clipper"
  get:
    description: Get a Programming Language
    displayName: Get a Programming Language
    responses:
      200:
        description: programmingLanguage
        body:
          application/json:
            type: programmingLanguage
            example:
              name: "COBOL"
              paradigms: "procedural, structured"
              fileExtensions: ".cbl, .cob"

Industries System API

#%RAML 1.0
title: Industries System API
description: This demonstration API lists industries
version: v1
types:
  industry:
    properties:
      name:
        type: string
        example: "fishing"
      description:
        type: string
        example: "description"        
      related:
        type: array
        items:
          type: string
        required: false  
        example: ["fish farming", fish processing"]
    example:
      name: "fishing"
      description: "Industry description"
      related: ["fish farming", "fish processing"]          
/industries:
  displayName: Industries
  description: Industries
  get:
    description: Get a Collection of industries
    displayName: Get a Collection of industries
    responses:
      200:
        description: Collection of industries
        body:
          application/json:
            type: industry[]
            example:
              - 
                name: "fishing"
                description: "Description of this industry"
                related: ["fish farming", "fish processing"]
              - 
                name: "mining"
                description: "Description of this industry"
                related: ["mining engineering", "automated mining"]                 
/industries/{industry}:
  displayName: Industry
  description: Industry
  uriParameters:
    industry:
      type: string
      example: "mining"
  get:
    description: Get an Industry
    displayName: Get an Industry
    responses:
      200:
        description: Industry
        body:
          application/json:
            type: industry
            example:
              name: "mining"
              description: "Description of this industry"
              related: ["mining engineering", "automated mining"]

And last, but not least, our experience API:

Industries Programming Languages Experience API

#%RAML 1.0
title: Industries Programming Languages Experience API
description: This demonstration API lists the top programming languages by industry
version: v1
types:
  industry:
    properties:
      industryName:
        type: string
        example: "automotive"
      industryDescription:
        type: string
        example: "Industry description"      
      programmingLanguagesDescription: 
        type: string
        example: "C and C++ are the most widely used coding languages for the car manufacturers"        
      languages:
        type: array
        items:
          type: object
          properties:
            language: 
              type: string
              example: "C"
            paradigms:
              type: string
              example: "object-oriented, procedural, structured"
            extensions: 
              type: string
              example: ".cpp, .h"
          example:
            language: "C++"
            paradigms: "object-oriented, procedural, structured"
            extensions: ".cpp, .h"              
    example:
      industryName: "automotive"
      industryDescription: "Industry description"
      programmingLanguagesDescription: "C and C++ are the most widely used coding languages for the car manufacturers"
      languages:
        -
          language: "C++"
          paradigms: "object-oriented, procedural, structured"
          extensions: ".cpp, .h"
/industries:
  displayName: Industries and most used programming languages
  description: Industries and most used programming languages
  get:
    description: Get a Collection of industries and most used programming languages
    displayName: Get a Collection of industries and most used programming languages
    responses:
      200:
        description: Collection of industries and most used programming languages
        body:
          application/json:
            type: industry[]
            example:
              -
                industryName: "automotive"
                industryDescription: "Industry description"
                programmingLanguagesDescription: "C and C++ are the most widely used coding languages for the car manufacturers"
                languages:
                  -
                    language: "C++"
                    paradigms: "object-oriented, procedural, structured"
                    extensions: ".cpp, .h"
/industries/{industry}:
  displayName: Industry and its most used programming languages
  description: Industry and its most used programming languages
  uriParameters:
    industry:
      type: string
      example: "cybersecurity"
  get:
    description: Get an Industry and its most used programming languages
    displayName: Get an Industry and its most used programming languages
    responses:
      200:
        description: Industry and its most used programming languages
        body:
          application/json:
            type: industry
            example:
              industryName: "automotive"
              industryDescription: "Industry description"
              programmingLanguagesDescription: "C and C++ are the most widely used coding languages for the car manufacturers"
              languages:
              -
                language: "C++"
                paradigms: "object-oriented, procedural, structured"
                extensions: ".cpp, .h"

The implementation of these is also very straightforward and simple, as the 2 system APIs return static industries and programming languages sourced from DataWeave files and the experience API simply makes calls to the system APIs and combine their results.

Smoke Testing Approach

Provided the experience API is the only one being externally consumed, and given this is a demonstration solution only, we will deploy all of the 3 APIs, but our approach to smoke testing will be to only smoke the experience API by executing these tests and assertions:

1. Send GET request to /api/industries and assert that
   Response status code is HTTP 200 (Ok).
   Response payload contains a collection of 1 or more industries.
2. Send GET request to /api/industries/e-commerce and assert that
   Response status code is HTTP 200 (Ok).
   Response payload contains industry with name “e-commerce” and programming language “Java”.
3. Send GET request to /api/industries/none and assert that
   Response status code is HTTP 404 (Not Found).

BAT Test Suite

Having determined our approach, including what and how to test, we can utilise BAT CLI and generate a test suite based on the bat.yaml below:

suite:
  name: "Industries Programming Languages Experience API - BAT Test Suite"
files:
  - file: tests/industries-programming-languages-exp-api.dwl
reporters:
    - type: HTML
      outFile: results/results.html

Next step is to implement the actual tests and assertions described previously in our file industries-programmin-languages-exp-api.dwl:

import * from bat::BDD
import * from bat::Assertions
---
suite("Industry Programming Languages API Test Suite") in [
  it must 'Get 200 response and a Collection of Industry Programming Languages' in [
    GET `http://localhost:9090/api/industries` with {} assert [
      $.response.status mustEqual 200,
      (sizeOf($.response.body) > 0) mustEqual true, 
      sizeOf($.response.body filter ((value, index) -> value.industryName == "e-commerce")) mustEqual 1,
      ($.response.body filter ((value, index) -> value.industryName == "e-commerce")).languages[0].language[0] mustEqual "Java"
    ]
  ],
  it must 'Get 200 response and e-commerce industry' in [
    GET `http://localhost:9090/api/industries/e-commerce` with {} assert [
      $.response.status mustEqual 200,
   (sizeOf($.response.body.languages) == 1) mustEqual true,
      ($.response.body.languages[0].language) mustEqual "Java",
      ($.response.body.industryName) mustEqual "e-commerce",
      ($.response.body.industryDescription) mustEqual "Description of this industry"
    ]
  ],
  it must 'Get 404 response When Industry is Equal to none' in [
    GET `http://localhost:9090/api/industries/none` with {} assert [
      ("HTTP Status " ++ $.response.status) mustEqual ("HTTP Status 404")
    ]
  ]  
]

With all the 3 APIs deployed, and up and running, we can execute BAT CLI from command line:

And the following results output HTML file is produced:

Next Steps and Conclusion

As mentioned before, BAT CLI can be installed, and utilised as part of any CI/CD pipeline, being embedded and executed after the successful deployment of MuleSoft APIs, whether deployed in Cloudhub, on-premises, RTF, etc. That is a natural next step from this, and in return, teams can realise reduction of the sprawl of testing tools and frameworks, while utilising an out-of-the-box, readily available, and Muley friendly tool.

Happy integrations and enjoy!

The Author

Eduardo Ponzoni is an experienced integration developer, architect, and manager with 19+ years of experience. Well versed public speaker and author of dozens of articles on all things systems and data integration. Leading from the front and by example, Eduardo is a MuleSoft Ambassador, holds several certifications and loves sales and delivery. He has led multi-disciplinary teams in digital transformation programs with MuleSoft, Kafka, Azure & Boomi. Currently at Deloitte AU as Director of Cloud and IT Engineering, responsible to establishing and growing the integration business, from strategy, roadmap, vision, sales/pre-sales to hands-on execution & delivery as a people leader and a technology specialist.