Microservices, Continuous Delivery And Quality

(Contract Based Development And Testing)

The Problem

The vision of most Engineering departments is to get to Continuous Delivery, achieving the minimal time from code check-in to release.

One of the major factors that hinders Continuous Delivery is the need to test integration and state transition between real services in a fully deployed system. Integration tests have some unpleasant features:

• Complex — needing to test multiple applications across multiple (often shared) environments, with potentially disparate deployment cycles
• Fragile — any failure in any component can block tests
• Misleading — a bug in a 3rd party service can manifest as a failure in your test
• Slow — subject to real service latency and environmental factors outside of your control
• Expensive — there is a misconception that system integration tests can be limited to a handful of journeys. If you want to test integration fully then each end-point with variations in payload and response will need to be tested.

Some Background

An application is made of three aspects;

1. behaviour in APIs
2. state transitions (entry, intermediary, exit) across APIs
3. contracts (incoming, outgoing) to APIs

Behaviour is implemented within the individual APIs, contracts are set at the ‘edges’, and state is maintained across APIs. Contract and State can be expressed declaratively outside of and before behaviour. We can take advantage of this and assert adherence to state and to contract, in a stub, without the need to use the underlying application.

A stub is a lightweight (can’t go wrong!) piece of code that, given a request, returns a hardcoded response. Stubs are traditionally used by acceptance tests to; focus on functionality without being subject to vagaries of third parties being available and correct, and to test hard-to-create test scenarios.

The developer of the service is best placed to write the stubs as they are the keeper of the contract and the functionality, and they own the service lifecycle.

Invitation To Discuss — Contract Based Dev & Test

We can limit the need for system testing by following a SOLID (software design principles) based Microservices architecture. Even with these measures, we still need to verify that contracts are actually being honoured and that consumers are calling services correctly to transition state. We could adopt Contract Driven Development combined with Contract Only Testing (aka API First Development), and reconsider whether we need to do automated System Integration testing at all.

We will lever the power of stubs as a vehicle for testing contract and state, thus bypassing the need for testing between the ‘full’ applications. Stubs will enforce contracts expressed in schemas such as Open API Specs (Swagger). State can be inferred from a sequencing of transitions (API calls) which can also be validated in the stub, in code or in a declarative way.

This approach is enhanced by a development approach that abstracts contract and state outside of behaviour, and a software life-cycle that builds the contract first (API First Development).

Schemas used to validate contract and state should be shared across Dev and Test. Automated tests should verify consistency between the application and stubs.

Microservices integration testing thus becomes testing of stable and comprehensive contracts and not complex deployed behaviour.

How To Apply Contract Driven Dev & Test

Application Design

• Separate the contract and state machine from the application behaviour. Consider using Swagger or Json schema
• Address the testing of asynchronous message calls
• Build a project folder structure, a software development lifecycle (SDLC) and a continuous integration/delivery (CI/CD) pipeline, to reflect this new approach
• Consider following HATEOAS if clients can follow

Acceptance Tests

• Acceptance tests to be run against both the real service and the stub of your service to ensure the stub is accurate
• They should (obviously) assert behaviour, but also check third party service invocation and request payloads to ensure we are calling the correct API with the desired parameters.
• They should share the same contract (canned json request/response etc) and state machine as used by the stubs
• A subset of Acceptance Test, used for backward compatibility, invoked by your third parties, run under CI

Stubbing

• Be a responsible member of the Microservices community and build the stub for the service you are delivering
• Should share the same contract (canned json request/response etc) and state machine as used by Acceptance tests
• An example is provided here https://github.com/telegraph/wireMockScalaTestExample

Testing Asynchronous Messaging

Asynchronous calls without a feedback loop pose a number of challenges for testing:

• How do we verify integration without full system testing
• How do we run functional tests, which usually stub-out third party services, when a call to a messaging API is encoded in our application (not dependency inversion)

These scenarios are happening with ever-increasing frequency as we move to Microservices and exploit messaging frameworks such as; SNQ/SQS, RabbitMQ. How do we address the task of ensuring that contracts are honoured in a ‘fail fast’ fashion, without losing the benefit of going asynchronous?

1. We could shift the responsibility for verifying the contract to outside of the service owner by externalising contracts outside of the behaviour. A third party could be responsible (like the Post Office analogy) for validating contracts in a synchronous fashion.
2. Dependency injection replacing the messaging call with the stub service
3. The messaging framework could take the additional responsibility for verifying the contract. For example, when instantiating the messaging instance, we could pass a Swagger schema or a Json schema
4. The messaging framework could have a test implementation that calls the target service (a stub in this case) and fails if the contract is not honoured.
5. The request payload could be built using a library provided by the Contract owner. This is a similar approach to using something like a Swagger client. It would require contracts to be well defined e.g. no JSON blobs.
6. Keep the message simple by passing a (callback) link to the API to be called by the target service to extract the payload. The contract being tested is the subsequent API rather than the message. (my personal favourite)

Project Folder Structure

• service_contracts
  json requests/responses, OpenAPI (swagger) or Json Schemas
• state_machine
  state transitions … suggest that this is a simple enum container
• main
  ‘the code’
• stubs
  Wiremock, within Docker so consumers can just simply run the image
• test_helper
  set state to help tests meet coverage and be truly black box
• test
  acceptance => verify API behaviour and 3rd party service invocation with payload
  state => verify cross API state transition
  n.b. Both run against the real application as well as the stub to ensure stub correctness
  performance => Gatling
  penetration => Veracode

Software Development Lifecycle

Continuous Integration/Delivery Flow

Triggered by GitHub code merge (after code review):

• Run unit tests
• Run static tests
• Build app (is this needed with Spring Boot?)
• Acceptance tests (API and state), against stubs of your service
• Acceptance tests (API and state), against your real (dev) service, calling third party stubs
• Deploy stub Docker image to repo and deploy real service and test helper to test environments
• Penetration tests
• Performance tests
• Trigger (from the deploy) compatibility tests by consumers of your service, and wait for success
  … manual exploratory testing …
• Release

Backward compatibility test triggered by your third parties (where you are the consumer, triggered by a new version of their application):

• Acceptance tests (limited to integration with your service), with your real (prod) service, calling their third party stubs. This test will drawdown the (LATEST) stub Docker image, start the image (all in CI) with dependency injected config pointing to ‘localhost’, run their functional tests, stop the image, notifying the service of success

Challenges / discussions

• As we transition to Microservices, there will be a period of time where projects are not using/creating stubs and not triggering consumers and this will need to be managed manually.
• External applications making use of our stubs (for their functional tests) and our service pipeline being able to trigger their compatibility tests
• Maintenance of Swagger definitions and consistency with the application
• Is encoding state machine in the stub taking on too much logic?
• Pact/Consumer Driven Testing is an alternative approach to integration testing but this approach introduces an extra dependency, unnecessary process, puts the onus on the client (not the consumer), is difficult when we have many consumers, does not address state transition testing etc
• Mechanisms for triggering the Consumer Compatibility test and mechanisms for your Service getting feedback?
• How to get more jokes into this blog.

Thank you for reading this blog, all feedback is welcome.

For further blogs from the Telegraph: http://engineering.telegraph.co.uk

-_-