Functional BDD Part 2: The Gherkin Type Provider

Last time I talked about how a feature file written in the Gherkin language can be used as the basis for a type system in F#.

Using the Gherkin file,

I derived the following type system:

(well not quite, I’ve renamed CalculateDiscount to ApplyDiscount & swapped its inputs so it can be used with the |> operator from GetRegisteredCustomer — but apart from that its identical)

The next steps from here would normally involve automating the feature (most probably using Specflow) and although there are several available, most Gherkin automation frameworks work in a very similar way.

To automate a feature we have to create a step definition file to hold individual step definitions — they tend to look something like this:

Each step in the feature file is “hooked up” to a corresponding function by matching the regex in the attribute. Any regex group matches in the step text are passed (with some basic coercion) sequentially as parameters into the function, while data table arguments (like the customers in our Given step) are passed in as a single Table object (always as the last parameter).

When a scenario executes, each of its steps are matched to a single function and then executed in the order specified in the scenario.

Simple right?

Anyone who has maintained a Gherkin automation code base will tell you that these step definitions are just waste at best, and downright dangerous at worst (regex!).

Almost all the Gherkin automation tutorials will advise you to keep the code in these functions as small as possible— calling other more reusable components to do the actual work. In my experience the majority of problems people have when maintaining these sorts of test suites are because they are either putting too much logic in this glue code, or trying to share it.

I also think the real problem here is more fundamental — there is a basic mismatch: the unit of execution of the Gherkin automation frameworks is the step, but the unit of execution of a behaviour should be the example.

The first line in the Examples table in the Scenario Outline could easily be written as a standard test using Expecto:

As someone who has come from a TDD background, I really love a well written unit test.

It can be executed to ensure all is well, but just as importantly it can provide an insight into how a system actually works (rather than just how it behaves). A good unit test suite should be read more often than it is written. No waste here!

With this approach though, the test & the feature file it is based on can easily become out of sync, and the feature file should always be the source of truth.

So what I’d like is a test that looks and feels like a unit test, but gets its data from, and is kept in sync with, its feature file.

This is F# — obviously we’re going to use types…

The Gherkin TypeProvider

A Feature is a basic tree; it has an optional Background and an array of children (a child can be either a Scenario or Scenario Outline). A child has an array of steps which can have optional arguments (like the Table in our Given)

The Gherkin parser can build this syntax tree from a feature file, and is used by both the Cucumber family of automation frameworks & Specflow (available as a NuGet package here).

Using the Gherkin parser, the Gherkin TypeProvider adds to this syntax tree, by creating properties using the text in the feature file.

For example, with the Discount Calculation feature it would create a Feature class with an array of children containing the single Scenario Outline (just like the Gherkin parser would), but crucially it will also create a read-only property on the feature called:

``Registered eligible Customers get 10% discount when they spend £100 or more``

that will return the Scenario Outline with its data from the feature.

The Scenario Outline will then contain, not just an array with its 3 steps, but also 3 properties named:

• ``0 Given the following registered Customers``
• ``1 When <Customer Id> spends <Spend>``
• ``2 Then their order total will be <Total>``

Each returning the specific step data from the scenario for that order, keyword & text.

Now, if I access the data from the feature using these named properties, any changes to either the scenario name or the text, keyword or order of any of the steps will cause the type provider to generate new property names and my build will fail.

Full Synchronisation

For every property that is created using text from the feature file, the Gherkin TypeProvider will also set a Visited flag on the returned child.

What this means is that the same child can be accessed through either an array on its parent, or via the named property; but its Visited flag will only be set to true if accessed via the named property.

If a scenario has been added to the feature, but has yet to be referenced in the test code by its named property, then the Visited flag will remain false and a simple list iteration of the underlying arrays can report any children that have not been used in any test. Example at the end.

In Action

Less talk, more code!

Before we start I’m going to change the feature….not in terms of what it is specifying, more the syntax. A Scenario Outline is nice and concise, but I want something more like a unit test where a unit is a single example, and for that individual Scenarios are better I think:

The Given step has been moved to the Background (executed before each Scenario) and each Scenario has a name summarising the example it represents.

Let us begin……

Installation

The Gherkin TypeProvider is available as a NuGet package here

Caveats:

• Some non alpha numeric characters can play havoc with Intellisense in some IDEs, even if they are valid F# property characters (workaround available, see later)
• When using paket, it requires local package storage
• Its still a work in progress (any help very much appreciated)

With those caveats in mind, lets move on.

I’ll start by creating an F# console application and adding references to the Gherkin.TypeProvider and Expecto using the dotnet cli:

Then I’ll create the feature file in the project folder and call it DiscountCalculation.feature.

I already have the type system so I can add that with some basic implementation:

The implementation is:

• Line 18: the DiscountCalculator which creates a function that implements the pipeline I discussed in the previous post.
• Line 21: ApplyDiscount is the function that we’re going build up by automating the scenarios one by one. I’ve defaulted it to 0 to start with a failing test (in true TDD style).

Now I’ll create the feature type system using the Gherkin provider in its own module:

The FSharp.Data.Gherkin namespace provides to access the GherkinProvider (line 3).

The feature type system is created on lines 5 & 6— a couple of things to note:

The first is the use of the __SOURCE_DIRECTORY__ constant to make sure the provider can find the feature file.

The second is the Sanitize option, this can be:

• none: only illegal f# characters are replaced with underscores
• partial: all non alpha-numeric characters except spaces are replaced with underscores (helps play nice with Intellisense)
• full: all non alpha-numeric characters are replaced with underscores (allows the type system to be consumed from a C# project)

As the scenario uses the £ & ‘ symbols I’m using partial sanitation (Note: the sanitation is only for the property names, the underlying feature data is never changed).

Line 8 creates a single instance of the feature using the static method CreateFeature that will be used by all the tests in the test suite.

In the Program.fs, I’ll reference the Features module that contains the single feature instance (unused for now — more on that later), and setup Expecto to run all the tests in the assembly:

The Background

The first step of the feature is to use the data from the Background step to create the context for each scenario. The background is:

Using the DiscountCalculatorFeature instance we just created, this is pretty straightforward:

The goal here is to create a function that has the same signature as GetRegisteredCustomer (i.e. takes a CustomerId and returns a RegisteredCustomer option), that will use the data from the Given’s data table.

Lines 10–11: Here we are accessing the data table argument of the Background’s given step by using the Background’s named property

``0 Given the following registered customers``

Which has an Argument that is an array of data rows.

Lines 12–16: Map each row in the data table to a RegisteredCustomer using the named columns ``Customer Id`` & ``Is Eligible`` of the data cell.

Lines 18–20: Create a function that returns a RegisteredCustomer option from the mapped customers using the supplied CustomerId i.e. GetRegisteredCustomer

Lines 22–23: Create the getDiscountedTotal function using the mock GetRegisteredCustomer call I just created, and the actual ApplyDiscount function we’re building. This is the function I’ll be using in the tests.

Setup complete.

Automate all the things

(Update: this article refers to versions before 0.1.9 when the new Gherkin 6 Rules & Examples feature was introduced. Rather than accessing scenarios directly from the feature they are now accessed via a Scenarios property on the feature instead. All other functionality is unchanged)

Now the Background has been setup I can start working through the Scenarios, driving out the discounting functionality.

The first Scenario is:

There are 3 pieces of information in this Scenario that I’m going to need in order to run the test — the customer Mary and her spend from the When & the calculated total from the Then.

To do that I’ll create some very basic helpers which will allow me to get that data from the step text:

With that done I can finally write my first test:

Lets go through this in detail:

Line 27: Normally I’d advocate using descriptive variable names, but that isn’t needed here — if anything I want a name that I won’t have to change if the test changes - t1 will do just fine.

Lines 29–30: Get the first Scenario using its named property on the feature (the £ has been replaced with an underscore). No need for descriptive variable names or comments anymore, we have all of that in the property name.

Line 32: Create an Expecto test using the Name from the Scenario

Line 35: Get the text of the When step via its named property.

Line 36: Use the helper function to get the the customer Id (“Mary”) and her spend (99) from the step Text property(Note: the text of the step is just “Mary spends 99”)

Line 37: Use the getDiscountedTotal function just set up using the background to calculate the total based on the customer Id & spend

Line 41: Get the text of the Then step using its named property (the ‘ has been replaced with an underscore)

Line 42: Use the helper function to get the expected total (99)

Line 44: Make sure the correct discount was applied (none in this case)

Run the test

Run the test using dotnet run and you’ll get an output similar to

This is expected as I defaulted the ApplyDiscount to zero (notice the test name still has the £ in it even though it had been removed in the property name).

Perfect — I’m starting with a failing test.

Pass the test

And now when I run the test:

Repeat

The next scenario is:

And the test is:

This is identical to the last test apart from 3 key lines:

Line 51: I’m using a completely different scenario

Lines 56 & 62: Accessing the steps specific to this new scenario

In true TDD fashion, this will fail:

So I’ll fix that:

And now when the tests are run:

The next scenario is,

and the test:

Again, this is the same as before just with a different scenario on line 72 & different steps on lines 77 & 83.

The logic for this step is already covered so when I run the tests everything passes:

Our last scenario (we finally get a discount) is,

and the test is:

New scenario on line 93, steps on lines 98 & 104.

As expected this will fail:

I’ll add the business rule to the code,

and now all my tests pass:

Validating the Feature

To ensure all Feature’s children have been visited I’ll add the following to Program.fs:

Here I’m iterating through the underlying scenarios of the single feature instance via the backing arrays (which is why I needed that reference to the Features module). Any child that has not been visited will cause a failure.

I’ve automated all the Scenarios for now, so this will pass.

Adding a new scenario

An eagle-eyed team member had read the feature and spotted a missing example they would like to add — unregistered customers that spend less than £100 get no discount.

That means I’ll have to amend the existing unregistered example to:

and Intellisense tells me something has changed (as expected):

Easily fixable — it was only the scenario text so I’ll just change that (no need to change the test variable name!) :

add the new Scenario,

but before I automate this new example, I’ll make sure that the test run picks up that there is something new to be automated.

Lets run and see:

Using the no-incremental argument to force a rebuild and pick up the new change I get:

The tests have passed but because the new Scenario hasn’t been automated yet, the run fails with an exception kindly telling me exactly what hasn’t been visited.

The final test:

Rerun,

Now everything is passing and this feature is complete.

So there we have it — by using the Gherkin TypeProvider I can bring the advantages of unit tests (small & discrete) together with the advantages of Gherkin (descriptive & collaborative).

All the code for these examples is here and the code for the type provider is here

Please try it out and feel free to let me know how you get on.

Next up: Scenario Outlines & Feature validation — here