Behaviour-Driven Development at the FT
by Sarah Wells
I’ve spent a lot of my time at the FT talking about and attempting to do Behaviour Driven Development (BDD), but only recently have I become convinced that it makes my life as a developer easier.
I think this is because it’s easy to focus on the wrong things. In this blog post, I’m going to discuss what I’ve learned about doing BDD over the last 3 years.
Brief Introduction to BDD
The original idea for BDD came from Dan North, who suggested a behaviour-driven approach to development as a result of his frustrations with doing Test Driven Development (TDD) correctly.
If you don’t know anything about BDD, there are two articles by Codeship — the first on TDD and the second on BDD — that are a straightforward, clear introduction to the concepts.
At the FT we’ve mostly used Cucumber-like tools for BDD.
These have two key parts:
- descriptions of behaviour in “plain text” — in Cucumber these are called feature files
- “fixtures” that convert that plain text into executable code — in Cucumber these are called step definitions
The feature files in Cucumber are not quite plain text. They actually use a syntax called Gherkin. But they are readable as plain text and our product owners have not found them confusing.
An example:
Scenario: A simple article can be successfully published
Given a simple article exists in Methode
And the article has not already been published
When I publish that article to the Content Store
Then it should be available from the Read API
And the id should be the same as the Methode UUID
And the title should be the same as the Methode headline
In the Scenario
- ‘Given’ states preconditions (which may well get set up by the test).
- ‘When’ states the actions that take place.
- ‘Then’ states what we expect the state to be after those actions.
- ‘And’ is used where there are multiple statements of a particular type.
You can see from this that the scenario is in a domain-specific language:
- Methode is our Content Management System,
- it uses ids in UUID format, etc.
The step definitions for us are written in Java. There is a Java method for each line in the scenario above. For example:
@Given(“^a simple article exists in Methode$”)
public void a_simple_article_exists_in_methode() throws Throwable {
prepareASimplePublishedArticle();
putThePublishedArticleInMethode();
}
What this all means is that doing BDD is a little bit more involved that writing a JUnit test: the execution of the test is split across multiple methods, which can make it harder to follow, and then there are the separate feature files to maintain.
There is some IDE support for this, but in my experience, writing tests and test maintenance will take more time.
Clearly, this is a brief introduction only. There’s also a good set of links on Liz Keogh’s blog if you want to read more, and Ryan Greenhall maps out the various aspects of BDD clearly in this diagram from his blog (reproduced here with thanks).
Our first attempt at BDD
The first project I worked on at the FT used BDD from the start, at the suggestion of the Lead Architect, and yet if you spoke to the developers and testers after a year or so, we were unconvinced of the benefits, disheartened at the overhead of doing BDD rather than Unit tests, and generally not very happy with our experience.
We’d started off using cuke4duke (an add-on to Cucumber that let us use Java for the step definitions). Documentation was a bit patchy and we struggled. Then in November 2011, the cuke4duke project was abandoned in favour of work on Cucumber JVM. That left us high and dry, since Cucumber JVM was not available to us.
We moved to JBehave. The advantage of making this move was that the syntax for the plain text descriptions and the Java fixtures was very similar to the ones from Cuke4Duke so migration was not too painful. We still had issues with running individual tests easily, we found debugging hard if the issue was in the tests rather than the code under test, and it felt like we were incurring a huge overhead to do this as BDD — “why can’t we just write JUnit tests?”.
Eventually, since our BA and Product Owner were not involved in writing the scenarios (we hadn’t convinced them of the value, which should have been a big red flag), and all our testers were comfortable using an IDE, we swapped to use Yatspec. Now our tests effectively were Junit tests — and the output of the tests was in plain English for non technical people. However, we still had an overhead compared to just using JUnit and weren’t really sure what we were gaining from the whole shebang.
BDD all over again
About 6 months ago, I moved project. Again, we were starting from scratch and our technical leadership and programme management were very keen for us to use BDD.
The new system will publish content from several Content Management Systems (CMSs) to a single ContentStore with a well defined content format.
It is built of multiple components communicating over RESTful APIs and we’re using Dropwizard for the individual components (which is great by the way).
Initial discussions were quite heated, and focussed on what tools we should use. At this point, I think we got lucky — our technical leadership decided we needed some expert guidance, and got Gojko Adzic in for a workshop on Specification by Example.
The workshop gave me lots to think about, and 4 months doing BDD on the project have convinced me that if you do BDD right, it can be a very useful process.
I do think however that it’s important to think about the following:
1) BDD is about the process not the tools
That doesn’t say anything at all about the tools.
It doesn’t force you into a “Given — When — Then” syntax or make you use Cucumber.
If you’re arguing about whether to use Cucumber or Fitnesse, take a step back. Why are you doing BDD at all?
BDD lets you develop a description of your system using a shared vocabulary. There are several places where this could be useful, depending on what the issues are for you and your team.
Maybe, like us, you want to make sure you all understand exactly what it is you’re trying to build at the point where developers, testers, BAs and product owners are discussing the functionality. This is also sometimes called Specification by example — and here’s a nice ‘love story’ about it.
Maybe, also like us, you want to use your scenarios as documentation (see for example the GDS, who publish all of their scenarios for anyone to see).
So, first step: work out why you’re doing this.
2) If your business aren’t collaborating with you, you aren’t doing BDD
This is where my previous team went wrong. We didn’t convince the analyst or product owner of the benefits, so it was largely testers and developers defining the scenarios.
On the new team, we are trying to keep in mind that the developers don’t own these tests (it’s surprisingly hard!). If the product owner doesn’t like the language, or doesn’t understand what the test means, you’re not doing BDD, you’re doing automated testing with overheads.
3) Only write a BDD test if your business cares about the behaviour
Gojko said “think about who you would contact if the test failed in 6 months time”.
If you’d ask a developer about whether the test is still valid, then it shouldn’t be a BDD test.
If you’d ask the product owner, bingo.
My new mantra — “would my product owner care?”
Of course, we do other types of testing as well. As an example, we have tests to check the status code and response for a request made with a non-supported HTTP method. We’ve checked, and our product owner really doesn’t care about this. So, we have JUnit tests that use WireMock to provide mock versions of the collaborating systems. WireMock is great by the way!
If our product owner changed his mind, it wouldn’t be too painful to create BDD tests for this functionality.
4) BDD is not the same as test automation
Sit down, this may be a shock… you don’t need to automate all your scenarios.
Some can be tested manually, either once or repeatedly.
Automation is its own reward, but for some scenarios it’s hard, and just making sure everyone knows how the system should behave can still be valuable.
As an example, we may want to define behaviour of the system in error conditions that are hard to automate. It’s still valuable to define what should happen.
5) Your tests are first class code
You should be refactoring your tests.
You should be refactoring your production code if it makes it hard to write BDD tests, just like you do with TDD (even though the code should start off testable since you wrote the test first. Right?).
Your best people should be working on tests regularly.
In the first project, we had automated testers working on the BDD suite. They were far more reluctant to change the code under test than a developer would be. Also, we ended up doing most of the test implementation after development. Bad idea.
6) Your testing doesn’t have to be black box
Our system does some transformation of the body of an article — removes some xml tags, changes others, etc.
Our product owner does care about this, but we really didn’t want to do 100+ end to end tests where we wrote a story into the content management system, published it, and checked the transformation. It is slow, and not concentrating on the thing we care about here.
So we didn’t. We have some acceptance tests that directly call our Java code via the body transformation interface. They still have descriptions in plain text, but they are very quick to run, and they are included in our commit phase in continuous integration for early feedback.
It really doesn’t matter to the product owner that this isn’t end to end.
We do have at least one end to end test to check that transformation happens for a realistic article — in fact, a ‘kitchen sink’ article with all the nasty stuff our tester could think of in it. But that’s it.
7) Get the language right
Our initial scenarios were… just what you’d expect if you let a developer go and write them. (I’ve made up this example to protect the guilty but it is representative.)
Scenario: A simple article with embargo date in the future cannot be successfully published
Given a simple article with embargo date > current date exists in Methode
And the article has lastPublishedDate NULL
When I try to POST that article to the Methode Bridge at “/publish/{uuid}” with content type application/json
Then when I try to GET that article from the ReadAPI at “/content/{uuid}” with content type application/json
And the error message should be “This article has an embargo date in the future”
And I should get a “404” http response
Talking to the business owners:
- they don’t really care whether the HTTP method is PUT, POST, GET…
- they don’t really care about the url
- they aren’t bothered about content type
- they don’t know all the HTTP status codes. They don’t even know all the common ones off by heart
Rewriting the scenario with input from the business owner
Scenario: A simple article with embargo date in the future cannot be successfully published
Given a simple article with embargo date in the future exists in Methode
And the article has not previously been published
When I try to publish that article to the Content Store
Then the error message should inform me the embargo date is in the future
And the response code should indicate that the article has not been successfully published
And the article should not be available from the Content Store API
Conclusions
To use BDD effectively, you need to work out why you’re doing it, make it part of how you work, and take care with it.
I knew we were getting somewhere with BDD when our first response to questions about a story we were already working on was to grab the product owner and a tester and add new scenarios into the feature files right there. You can’t easily do that with a unit test.
It does, however, take longer to write good BDD tests than good unit tests, so you really need to make sure that everyone on your team understands what you’re trying to achieve and takes part in defining, discussing and implementing your scenarios.
If you do that, it can help enormously in working out exactly what your system needs to do, and provide documentation of the behaviour of your system at the same time.
