Documented acceptance tests that help Development and Business come together

A case for agile scenario testing in the context of integration-centric projects.

In this post, we expose a quality-oriented approach for integration-heavy projects. We advocate the practice of defining integrated scenario tests (IST) before development, as a way to get bus, dev and ops forces on the same agenda. We propose a simple method to derive test cases directly from the specs, and as a way to improve those specs and fuel software design. Granularity is our keystone concept to improve the relevance of test cases. BPMN is used as a formal framework and we show how to infer tests cases directly from BPMN workflows.
For a more hands-on approach, check out our IST crash course.


WHEN TESTING FAILS TO TRAP THE ESSENTIALS

When it comes to testing, discussions tend to linger in the extremes : talks concentrate around technical tests (unit tests, integration tests), or around user tests (user acceptance tests, usability tests).

Indeed, techies in the team generally concentrate on tricky technical challenges, that are the most relevant for their CIT competences, while business guys do focus on UI/UX issues, where they are the ones in charge (and where the management is the most able to spot obvious problems to put some pressure on.)

The problems with those tests are :

  • Unit tests can pass while the most important objective is not met, eg. the flight booking third-party service is integrated and tested ok, but one still can’t book a flight…
  • User tests generally list all the clicks and inputs that the user might ever perform, and do not give the big picture with the most important steps that have to be completed.
  • As a result, each side tends to work separately on its own set of tests, and avoids to dive into test results and analysis provided by the other.

Too often, each side assumes that the other one is in charge with verifying the essentials. Which leaves room for aporia : how many times, at the end of the testing day, has a team found a basic step missing while development is already “completed” and the test period is over ?

And when essentials are not identified, the testing too often stalls, blocking both teams with other tasks that depend on test results.

THE ANTIDOTE : GENERAL STRUCTURE FIRST

User tests generally clutter the bug reporting system with non-blocking issues, eg : “the email address validity is not checked, the error message should be in a bigger font, our logo is not big enough…

Instead, before sending any feedback to developers about the UI/UX one should check that users can perform each specific story end-to-end, and that all systems (web servers, database, third-party systems, e-mails…) are interacting efficiently in the background.

We don’t mean in any way that UI/IX issues are not important. In fact, they are generally critical for most web applications : in the field of e-commerce, a pixel perfect, portable UI, should be part of the minimum viable product.

But if you are not sure of the global structure of your scenario, it simply doesn’t make sense to validate its UI, because UI is there to support the transaction, and not the other way around.

When you integrate autonomous services, or even third-party services (as I bet you did in most of your transactional web projects) the structure of scenarios reflect the path of users through all of those systems.

This is a challenging ground, that is shared between bus and dev teams, and for this very reason, you may elect to clear it out first. Doing this, you will start with solving problems that require most of time-consuming cross-team coordination.

Thereafter, each of both teams will be able to clean up its own end of the project, be it screen specification or non-functional issues.

BRIDGING THE TESTING GAP WITH INTEGRATED SCENARIOS

If your project is heavy on integration, you should consider starting acceptance with integrated scenario tests : ‘integrated’, because those tests are going end-to-end through all systems; ‘scenarios’, because each test completes a specific case of a user story, from the user’s point-of-view.

Integrated scenario tests (IST) stand in the middle ground between unit tests and user tests. In IST phase, you don’t care about spelling mistakes, or about your logo. And you don’t care neither about testing every possible combination of parameters for every major function.

But you do care about high-level user interaction (eg. “I can select and order my flight”), and about effectiveness of system integration : “the flight is billed, it is actually booked with the third party, I receive a mail with the order confirmation…”

As we’ll see later on, when you design enough of those tests before development and run them before any other user tests, you optimise the efficiency of the testing feedback loop and contribute to improve the full software cycle.

WHAT’S IN A TEST

In order to trap your application’s essentials the test set must have the right scope :

  • Story scope : cover all relevant project stories.
  • System scope : scenarios should be detailed enough to guarantee that each involved system is playing its role decently. Write cases that go end-to-end, through all visibly impacted systems (data servers, broking systems, user input, e-mails, document uploads). If you cannot integrate them before testing occurs, just stub them, or provide the testers with a way to simulate them.

A test case is not a simple cut and paste of your agile user story (eg. “as a customer, I can configure, select and order a flight”). In accordance with common definitions, it specifies :

  • which preconditions (prerequisites) must be met before the story takes place : “I am an identified customer, I have a registered credit card, there is enough credit left on that card”;
  • which action each party (user or system) has to perform to complete the story : “the third party confirms the flight availability, I fill the forms and give a credit card number, I validate the transaction, the system checks my credit…”, and
  • the postconditions (results) you are entitled to expect for each party : “the financial transaction is completed, the flight is billed, it is actually booked with the third party, I received a mail with the order confirmation, I received the bill as an attachment...”

Therefore, for every user story in the requirements, we will end up with several scenarios : one for when the credit card is not accepted, one for the case where the user cancels the order, etc.

For a given user story, we can have all scenarios in a convenient spreadsheet, where lines represent cases, and three groups of columns represent preconditions to be setup, actions to be performed and postconditions to be verified :

Sample 1 : an example Integrated Scenario Test case, stating preconditions, actions and postconditions.

Spreadsheet are easy to maintain, and they are very convenient when you run the tests, because you can copy actual input data from the spreadsheet in the tested application, and record the results directly in the spreadsheet too, before you go reporting your issues in the bug tracking system.

Directly from the IST spreadsheet, it’s quite easy to write Gherkin pseudo-code (Gherkin, is a simple but powerful language for writing and executing high level descriptions of your software’s functionality, with “Given”, “When” and “Then” pseudo-statements that will translate your preconditions, actions and postconditions).

Gherkin is quite useful for test documentation, even if if you’re not looking for BDD style test automation (by the way, if you are not already into Behavior Driven Development, you might take five minutes to check this link).

You can also translate your IST case into full browser automation, à la Selenium, but then IST does just provide a good documentation, and has no decisive formal advantage.

ADJUSTING TEST GRANULARITY

The most important ingredient in the secret sauce is granularity. This is where you make sure you don’t fall back into undercover unit testing (or undercover user testing) .

  • Each sequence of screens with a unique possible outcome must be wrapped as a single test step. Your test should mention global workflow steps (eg : “upload the requested identification data”), rather than the fully detailed screen sequence for that steps (“upload a copy of the ID card, then submit form one with personal data, then submit the form with company data, then do upload a copy of company statuses…”). You’ll be better off verifying such details during user tests later on.
Sample 2 : bad example, with too much detail : don’t pour all UI/UX considerations in your IST test cases
  • One test scenario per workflow path : for every possible combination of the conditional switches (workflow diamond boxes) of a given story, there should be one (and only one) test.
    - If you end up with several tests for a given path, look up for forgotten preconditions and extend your workflow. 
    - If you end up with a workflow path that is not in the tests, extend the test set.
Sample 3 : a BPMN workflow, matching the sample 1 IST test case above. One can see that each line in the test case matches a unique combination of states in the diamond boxes and entry points.
  • Extend controls to all relevant business data : make sure to verify every postcondition that is critical for the business, on every system (databases, repositories, mail servers). Eg. verify that payment is completed, with the correct amount.
  • Don’t control technical data : don’t verify postconditions that will not be consumed by a human business user. Those are better left to unit testing or to bug fixing. Eg. don’t take into account the format of payment messages between the third party and the booking system.
  • Look at computed functions from the point of view of the outcome, not from the perspective of the input. In the example of the flight booking system, IST does not care about how the flight selection system reacts to data submitted by the user (this would rather be the object of a unit test). More important is to test the all possible outcomes of each involved function (eg : the system returns no flight, the system returns only one flight, the system returns some flights, the system returns a very long list of flights.)
  • Break stories down in minimal but meaningful scenarios : test sequences have to be as short as possible, because the more they contain conditionals (that is, workflow diamond boxes), the more the testing will be heavy. Indeed, for every conditional that we add in front of a given scenario path, the number of paths to test may be multiplied by two or three.
    But while you split up scenarios in the smallest possible sequences of actions, those sequences have to remain long enough to bring measurable and significant outcome : don’t stop before explicit (user-driven) or implicit (data-driven) choice has taken place, and meaningful business postconditions have become measurable.

If you end up with too many columns or lines in your test case, you should think of improving granularity.

  • Too many post conditions and/or actions columns means that you have to split the story up in two or more small parts. There is probably a specific step in the middle of the flow where it would make sense.
  • Too many preconditions columns means that you probably need to pipe more explicitly you story as the sequel of another story, that will provide you with all requested preconditions off-the-shelf, wrapped in a synthetic status variable.
  • Too many scenarios (that is, too many test lines in your story spreadsheet) means that you are probably defining unit tests, so, again, look at computed functions from the point of view of the outcome, not from the perspective of the input.

LOOKING FOR MODULARITY : THE CASE FOR TEST-DRIVEN DESIGN

The above list of granularity tricks is certainly not complete, but you understand the idea : make your test cases modular : each test is a module, receiving some input and giving some output.

We can even go further, and try to have a formal match between the test case and the workflows. Using BPMN as a workflow modeling formalism, and trying to breakdown the project into one-page modules, we can just count down the number of possible paths in every module and make sure we have a test case for each path.

Such an approach ideally takes place just after workflows are designed, as a conclusion of the specification phase. IST test cases should be made tightly consistent with workflows (be it UML, BPMN, or any other workflow flavor). Ideally, a qualified Business Analyst should be in charge of the test cases, and they should be thoroughly validated by the Technical Analyst. This means that, if you “agility” does not provide room for workflow specifications, then IST is probably not for you.

Benefits of performing such an analysis before development are substantial. Indeed, IST tests :

  • Improve workflow granularity : as explained above, you will have to adapt workflows in order to have them matching your IST case. This will help you expel many details out of workflows, and defer them to UI specification, where they actually belong. This practice will also help holding back function testing inside the perimeter of unit tests.
  • Set the naming straight: you name each scenario according to the conditional diamond boxes inside the workflow : eg., a name such as payKO-retryOK can be read by operations, development and the business. This will facilitate bug reporting, tracking and fixing.
  • Shorten and improve UI/UX tests : UI tests are easier when one executes IST tests before UI tests, and waits for validated and functional workflow steps before going down to screen details.
  • Improve software design : because IST tests define all integration paths that users might take, they are a very good source for workflow architecture. And the complete list of preconditions, actions and postconditions gives a complementary perspective, more down-to-earth, on the concepts defined in specifications, a perspective that is generally very valuable for developers.
  • Provide a solid basis for test automation : as each test includes the complete list of preconditions, actions and postconditions, it is really easy to automate. And, as mentioned earlier, IST tests are BDD-savvy by design.
  • Get ready for operations : And when you finally go to production, those far-reaching, integration-savvy, IST scenario, are unvaluable for your sanity checks and will help in defining the objectives of your monitoring scripts.

WHAT DO YOU THINK ?

I have seen the benefits of such an approach in the full software cycle, for integration-heavy web applications (CMS, ESB, e-Commerce) in two industries : telcos and banking.

Nowadays, it probably needs further assessment, and maintainability of IST workflows and test cases on the long term still has to be proven.

In all cases, this way of testing is certainly not complete and has to be used along with the other types of tests.

This approach is not completely new, neither : it just sets priorities in a different perspective, more compatible with present agile and dev-ops trends in the software industry.

And, hopefully, it reshuffles the cards of testing, significantly enough at least to write an article about it.

And you, what do you think ? What is your experience, and what would you command ? Comments are yours…


The crash course :

Many thanks

…to Candide Kemmler and Raphaël Thys, for the early stage feedback.