Test your code E2E without Internet

Life was going well, we were following the standard testing pyramid. For my service, I had 80% Unit Test coverage, 60% Integration Tests coverage, and 30% coverage of the E2E test. The overall quality of the application was good.

Things changed when my manager pinged me:

Manager: Hello, I have one question.

Vaibhav: Hey, tell me

M: Can we execute our automation tests without the internet?

V: (thinking)

V: (thinking)

V: (I knew he was not talking about UTs, but just to confirm) Do you mean Unit Tests?

M: No, integration tests

V: No, we can’t. We need to deploy the service, plus our service depends on A teams services so I want them as well…

M: Can you come up with some strategy on the same? The reason behind this is our Integration Tests are taking way longer to provide results because of these resources and dependencies.

V: Got your pain point with Integration Tests let me see what I can do.

This was something we(the Quality Crafters team) were supposed to work upon, we knew about mocks which we used in our UTs but never used them in writing Integration Tests, and the exploration began. Firstly we started looking into challenges faced with the current Integration Tests and E2E Tests setup.

• Not all dependent services/resources may be up and running all the time. As cost need to be considered as well.
• Not all scenarios are reproducible, like testing 5xx errors
• Late feedback to developers due to slowness. As we go up the ladder on the Test Pyramid the test execution time increases
• ITs are time-consuming and the entire suite can’t be run frequently during each pull request raised
• ITs can’t be executed in the local environment as it is resource exhaustive. You might need to bring up multiple services locally or deploy the production branch(majorly master) of services to some environment.

The solution to all the above problems was to use mocked resources, it can be DBs (SQL/no-SQL), AWS resources, mail resources, and all other dependent services. We named these test types, Component Tests. Component Tests live in between Integration Tests and Unit Tests, We modified our Tests Pyramid.

Component Tests

also known as program or module testing, is done after unit testing. In this type of testing, those test objects can be tested independently as a component without integrating with other resources/components.

Component Tests vs Integration Tests:

Both require multiple components to interact with each other but the difference is that an Integration test may simply verify that you can query the database but in Component tests, we can do all the other assertions. As we have mocked all the data.

The above changed the way we write our integration tests, instead of writing all negative and positive test scenarios as Integration Tests we majorly started writing happy paths, and the rest scenarios became part of Component Tests.

Benefits of Component Tests:

• Faster execution without taking too much effort to deploy and release faster
• Testing the right areas and finding critical bugs
• Early feedback
• Fail Fast
• Minimize cost and resource
• Local Testing with ease

With benefits comes Limitation as well.

• A mock will not be a full simulation of the corresponding component
• Fixed response time, not realistic
• Maintenance of mocked response/results
• Untested integration of the real objects

It has a few limitations but the benefits overshadowed the limitations.

Mocking in Component Testing:

This testing is something similar to Grey box testing so we need to be aware of what all external resources our service connects to as this allows our system to test in isolation.

For example, my service fetches data from youtube APIs then youtube API has to be mocked, my service connects to MySql DB it will be mocked, my service connects with AWS S3 it will also be mocked. All HTTP connections, message queues in a nutshell all external resources have to be mocked.

We used Test Containers for mocking DBs, local stack for S3, and MockServer for HTTP calls.

Executing Component Tests:

We know CTs are fast…

We know CTs can be executed locally..

And UTs always execute as part of the build..

So for CTs we made sure they also execute as part of the build as in, mvn clean install.

We actually use SpringRunner class and SpringBootTests for the test execution. We used @ClassRule spring annotation to bring up all the dependent resources and Tests were written using the Karate framework.

For my STU I am mocking AWS S3, MySQL container, and SMTP server. All external resources run inside a docker container.

Conclusion:

Thanks for being with me until the end. To conclude, with the help of Component Tests

• We are getting faster feedback on our Tests.
• Developers don’t really have to rely on slow running Integration Tests for quality feedback
• We are able to test scenarios such as resource down, or resources not responding which is tough to replicate using Integration Tests as with Integration Tests we need manual intervention.
• It is helping us to go faster to production