Photo by Sigmund on Unsplash

TESTING

Testing with BDDfy

A short introduction to behaviour-driven development using TestStack’s BDDfy

Using Behaviour-Driven Development (BDD) is a way to verify system behaviour using terminology that is approachable and understandable by both technical and less-technical individuals. It can, therefore, form a bridge of understanding between the people that require a software solution and the people they engage to build the solution.

Many tools, including cucumber, gauge, and BDDfy, can be used to write and execute BDD tests. This blog provides a short overview of BDD and illustrates how BDDfy can be used to write BDD tests.

Overview of BDD

BDD emerged from TDD; however, the test cases can either be written prior to implementation or after. An effective approach is to express requirements in the given/when/then form, and these requirements can then be turned into both the solution and automated tests (as illustrated below). Alternatively, if a system has already been implemented, BDD tests can still be applied after the fact, however it is then necessary to translate the existing requirements into the BDD form.

BDD should be used in conjunction with other forms of testing, including unit tests. In this way, it forms another layer in your test pyramid.

Example layers of a test pyramid — your actual layers might vary depending upon circumstances

BDD tests tend to take longer to run than unit tests and require an environment in which to run, so it’s impractical to cover all permutations of scenarios, such as handling unexpected errors. Unit tests are better suited to this, as they execute faster. BDD should be used to broadly determine that requirements have been met, whilst unit tests should cover requirements and also delve into the nitty gritty, for example, can the solution cope if the downstream API returns XHTML rather than JSON?

It is also possible to organise BDD test cases into groups to address a specific purpose. For example, you might have a narrow set of “smoke tests” running after a deployment to verify the system is still up and running. Then, you might have a wider set of “regression tests” that ensure all requirements have been met, and these would run prior to merging changes and possibly prior to promoting changes towards production. (The latter might also be called “acceptance tests”.)

BDD tests are commonly used to verify user interfaces, e.g. web applications. However, they can also be used to verify backend components that expose an API, and considering that a lot of business logic resides in these components, there’s often a lot of benefit to applying BDD to them. Writing BDD tests that interact with web applications has often been time-consuming and troublesome due to difficulties getting the browser(s) to do what you intend and timing issues between invoked actions and results being rendered on the browser. Testing against an API is often simpler and avoids these browser complexities, so it is often better to verify the bulk of your business requirements at the API level rather than at the browser level.

Using BDDfy

Many tools can be used to write BDD test cases. One such tool is BDDfy, and a brief introduction is provided below.

With this tool, you can write BDD test cases using C# and .NET. The tests are run with common test runners, including NUnit and XUnit.

It does necessitate the C# language, and the .NET framework, however despite this, it remains a viable option for testing any system, regardless of the language/technology used to build the system under test. Today, the .NET framework can be run on Windows, Mac and Linux, and is supported by the popular clouds.

Using a completely different language to write the tests will ensure that when testing, the code under test is not used to verify the code under test, i.e. the tests are completely independent from the code under test.

The tool itself is very lightweight and flexible. The learning curve is shallow, so it’s possible to get some tests up and running quickly.

Let’s look at an example, starting at the end. The example code that follows will output a report like this.

Story: Guitar Shopping Use Case
  As a guitar retailer
  I want to send surveys to customers
  So that they can provide feedback about their shopping experience.
  
  Given guitars are available, when a customer places an order, then a survey is sent to the customer
    Given guitar inventory is available
      And a registered customer C12345
    When an order is placed
    Then a survey is sent to the customer

As the tool is flexible, there are different ways you can use it. A basic approach is to write tests by implementing classes in C#, where each class represents a story (or use case).

using TestStack.BDDfy;

namespace My.Tests;

[Story(
        Title = "Guitar Shopping Use Case",
        AsA = "As a guitar retailer",
        IWant = "to send surveys to customers",
        SoThat = "they can provide feedback about their shopping experience.")]
public class GuitarSurveyTests(InjectionFixture injection) : IClassFixture<InjectionFixture>
{
  private readonly InjectionFixture _injection = injection;

  // TODO implement test scenarios
}

In the example above, we have a class that will encapsulate test scenarios for a guitar shopping site. The class is annotated with a user story that will be output in the final report after the tests are run.

The injection fixture is optional, it’s a way of using dependancy injection to provide services required by the tests, to communicate with the system under test. The example is being run with xUnit, and the IClassFixture is an approach to injecting resources shared by many tests. There are alternative ways to achieve the same thing.

Next, for each scenario, you would write a test method.

[Fact]
public void CustomerPlacesOrderScenario()
{
  var customerService = _injection.ServiceProvider.GetService<ICustomerService>();
  var customerNo = customerService.RegisterCustomer("John", "Doe");
  
  this.Given(x => GuitarInventoryIsAvailable())
      .And(x => ARegisteredCustomer(customerNo))
      .When(x => AnOrderIsPlaced(customerNo), false)
      .Then(x => ASurveyIsSent(customerNo), "a survey is sent to the customer")
      .BDDfy("Given guitars are available, when a customer places an order, then a survey is sent to the customer");
}

// TODO other private test methods

private async Task ASurveyIsSent(string customerNo)
{
  var surveyService = _injection.ServiceProvider.GetService<ISurveyService>();
  var surveys = await surveyService.GetByCustomer(customerNo);
  
  // the following is an example of making an assertion in the tests, in this case
  // using Shouldly, however you can use any test assertion library, compatible
  // with your test runner of choice.
  surveys.ShouldNotBeEmpty();
}

As xUnit is being used, the test (scenario) methods are annotated as Facts so that the test runner runs them. To run the tests, enter the command dotnet test.

In this scenario, a new customer is registered in the system under test by using a customer service obtained from dependency injection.

There are different ways to use BDDfy. In the example above, the fluent API is being used. Each method either prepares fixtures required by the test, initiates an action against the system under test, or verifies that expected behaviour has occurred in the system under test.

When this test is run, it will output a report. By default, both a console report and an HTML report are produced. The latter, being a single HTML file, is easy to share with others and/or easily published from a CI server.

In the example above, the phrase Given guitar inventory is available is derived from the GuitarInventoryIsAvailable() method name. Similarly, the phrase And a registered customer C12345 is derived from the ARegisteredCustomer() method, and the C12345 is the value of the customerNo parameter.

It is possible to suppress parameter output by passing a second parameter with a value of false, e.g. .When(x => AnOrderIsPlaced(customerNo), false) is rendered as When an order is placed. You would want to do this if passing objects into methods. Yet another option is to override the ToString() method of any objects being passed so they render neatly in the reports.

Rather than deriving output from the method names, it is also possible to pass a literal string to be displayed in the report. For example, .Then(x => ASurveyIsSent(customerNo), "a survey is sent to the customer") is rendered as Then a survey is sent to the customer.

Conclusion

Using BDDfy, it is possible to quickly write BDD test scenarios. As the tests are implemented with a high-level language (C#), there’s no limit to what they can do. BDDfy provides many options to control the output in the final report.

The reports provide a clear description of the system’s current behaviour and indicate the scenarios that are passing, failing and/or “not yet implemented”. Both technical and non-technical audiences can understand the current state of the system under test, and can compare this to any documented requirements.

The tests must be implemented by developers, however it’s suggested that first the system requirements be documented in the form of “given/when/then” statements, that can later be turned into BDD test cases. The final test reports can then be compared to the original requirements (a tip is to assign a unique reference number to each requirement so they are easily mapped to each scenario in the test report).

Using BDDfy does necessitate C# and .NET, however today the latter can be run (without a license) on Windows, Mac and Linux, and the C# language is approachable to most modern developers. If this is still not palatable, then there are other options for using BDD tests, such as cucumber and Gauge.