Is Schema Validation Enough? Understanding the Nuances of Contract Testing

My original blog on our company website (in Norwegian):
https://www.bouvet.no/bouvet-deler/kontraktesting-navigering-gjennom-vanlige-fallgruver

In my previous blogs on Contract Testing, I wrote mainly about Schema Validation. For many aspiring Contract Testers, this form of testing is easy to understand and easy to implement. But a pertinent question often arises: is schema validation sufficient? To offer clarity, I will explore various types of contract tests using a cohesive example of an online book store API.

1. Schema Validation Testing

Definition: Ensures that data exchanged between consumer and provider matches a predefined schema, focusing on data types, structures, and formats.

Example: Imagine an endpoint in our book store API, /api/books/{bookId}. The expected response might be:

{
    "id": "12345",
    "title": "Journey to Middle-Earth",
    "author": "J.R.R. Token",
    "publishedDate": "1970-01-01",
    "price": 15.99,
    "inStock": true
}

A schema validation test would verify the types and formats of each field. A returned price as "15.99" (string) instead of 15.99 (float) would trigger a test failure.

Pros:

• Provides quick structural feedback.
• Ensures basic inter-service communication compliance.

Cons:

• Doesn’t ascertain data correctness or range.
• Limited validation depth.

2. Provider Contract Testing

Definition: Ensures the provider (e.g., the book store API) upholds its contract, verifying both structure and data accuracy.

Example: Our API promises a 404 status with “Book not found” for invalid bookIds. If a non-existent bookId returns a 200 status with an empty object, it indicates provider contract breach.

Pros:

• Validates data’s accuracy.
• Ensures providers maintain their contract terms.

Cons:

• Requires more maintenance.
• Relies on contract accuracy and depth.

3. Consumer Contract Testing

Definition: Centers on consumer’s expectations, ensuring the provider’s data is actionable for the consumer.

Example: A front-end using our API expects the inStock field to display availability. If the API omits this, the test fails, preserving the consumer's functionality against provider changes.

Pros:

• Ensures provider caters to consumer needs.
• Identifies potentially harmful changes for the consumer.

Cons:

• Needs frequent updates for evolving consumer expectations.
• Can be challenging if multiple consumers have diverse provider expectations.

4. Mock Testing

Definition: Uses mock responses based on predefined conditions to verify how a system reacts without calling the actual service.

Example: A mock test for our /api/books/{bookId} endpoint might use a fixed response to simulate various scenarios, like book availability or unavailability, without connecting to the actual database.

Pros:

• Allows testing even when some services aren’t available.
• Helps simulate various conditions, such as service failures.

Cons:

• Doesn’t guarantee that the real service will behave as the mock does.
• Risk of mocks becoming outdated, reducing test reliability.

5. End-to-End (E2E) Testing

Definition: Tests the system as a whole, validating integrated components’ collective behavior.

Example: An E2E test might simulate a user browsing our book store, selecting a book via /api/books/{bookId}, adding it to their cart, and completing a purchase, ensuring all integrated services function cohesively.

Pros:

• Provides the most holistic view of system health.
• Confirms the collaborative behavior of all integrated parts.

Cons:

• Tends to be slower due to its comprehensive nature.
• May be more challenging to pinpoint specific issues if tests fail.

Conclusion:

Contract testing is multi-dimensional. While schema validation lays the groundwork, ensuring efficient communication between services necessitates a blend of tests: schema validation, provider and consumer contract tests, mock tests, and E2E tests. Through our book store analogy, the significance of each type becomes evident, underscoring the need for a comprehensive approach. Such an understanding equips teams to construct a robust microservices architecture, primed for resilience and reliability.