The Key to Reliable and Resilient Microservices
Introduction
2023’s summer school break had come, and with it 3 months of free time! As a university student studying Information Systems, I decided to focus on expanding my knowledge in software development. I already had some experience working on small school and personal projects. However, I was interested in what it would be like to work in a larger team of engineers with more experience than I had. What are the typical challenges faced by larger-scale projects, and how are they tackled?
Fortunately, I was granted the opportunity to join GovTech’s Smart City Technology Division (SCTD) as an intern, under the Health District team. I chose to join their backend team as it was the part of software development that I was most curious about learning at the time.
The team then was exploring the idea of integrating test-driven development into their workflow. The basic idea of test-driven development was to first write the tests before writing the actual code when developing a product. This is contrary to the traditional practice of completing the code before writing tests which many teams, including ours, had been following so far. To start, we would need to develop tests for our already existing code. But what kind of tests should we build?
Backend Challenges
Our backend was structured following a microservices architecture, in which a collection of smaller services that can be deployed independently work together to make a larger application. The nature of such a structure resulted in a lot of communication occurring between each of our systems during deployment. Our team also follows the practices of agile software development, in which small but effective changes are made during short development cycles called sprints.
However, this led to some issues with testing when small changes are made in different systems by different engineers during development. The only way we had to ensure compatibility and consistency between these services was to minimally do some integration testing.
Integration testing requires all services of the test case to be deployed, and that each of these services are in the correct state according to the test case. Such a setup for testing takes up too much time, considering only small changes were made. Moreover, debugging takes significantly longer as well due to the amount of code tracing required within all services involved in the test. Hence, integration testing typically occurs only during the later stages of the development cycle, which meant that bugs from these small changes would surface much later in the process as well.
In order to increase our team’s performance during each cycle, it is ideal for such bugs to be fixed earlier, preventing them from turning into bigger problems at the end of a sprint.
Contract Testing
This led to our team discovering a different form of testing known as contract testing, and I was appointed to take the lead in exploring this new method. Contract testing is designed to only test the communication between services. The mutually agreed type of messages that are sent and received between the two services are documented into a contract. These contracts are then used to validate the interactions between services during contract testing.
This allows the testing of a service’s integration point without deploying said service, due to the contract acting as the service itself. The idea of contract testing solves our debugging challenges mentioned earlier, because services can now be tested in isolation while validating their interactions with other services. Therefore, bugs would then surface earlier and be easier to track.
Pact.io
We decided to utilise a popular framework of contract testing named Pact, as they have a Python version, a language which our team is familiar with. To start off, I created a sandbox repository to set up dummy services to explore pact testing. The first step in contract testing is to write consumer tests. These tests specify the requests and responses expected from both services and will generate a contract when performed successfully. To pass a consumer test, the consumer needs to send requests exactly as outlined in each test cases. Since we were utilising Python for Pact testing, I decided to use a Python testing framework to write the test cases. I chose pytest as it was popular and had extensive documentation.
Once the contract is generated from consumer testing, it is time to perform validation of the contract with the provider service. In order for validation to succeed, the provider service needs to respond to a request exactly as outlined in the contract. The verification itself can be done using a terminal command, which came with the Pact-Python package.
Automated Testing
After generating the contracts between all the dummy services, it is time automate the testing process, since running every test and validation one by one would take up too much time. We want the automation to be done in a service level, as contract testing allows us to test services in isolation. To achieve that, I learnt shell scripting in order to write scripts for each service. These scripts would first run consumer tests for any requests the service would send to other services, before validating any contracts it has with services that would send requests to it. We’ll refer to these as pact scripts for the rest of this post.
Next, the team needed the tests to be stateless and identical when any developer ran them on their local machine. A way to achieve that was to run these tests on containers. Hence, I learnt about Docker and containerisation in order to build these test containers, which would run the pact scripts. I also wrote another script to automate the building and destroying of these test containers. A working demo of pact testing was finally done, check it out here!
Finally, I had to integrate what I learnt into the actual services that our team were running in the backend. To further enhance the test automation process, I also learnt how to write Makefiles and client-side git hooks to set up pre-commit hooks for the team. This allowed the tests to trigger when attempting to commit changes to the code, to prevent breaking changes from being committed by the team.
Team Culture
Aside from the frameworks and tools I learnt working with pact testing, I also learnt the importance of a positive team culture in a job. An open communication between team members regardless of work experience, and the flexible work style of our team allowed us to perform at our best. I was always able to contribute new ideas to the team as an intern.
With Pact-Python, applying new concepts and tools with limited documentation has been one of the hardest challenges I have faced as a software engineer. I would have definitely not been able to progress this far without the help of my awesome team. After this experience, I really learnt the importance of culture when selecting a job.
Closing Thoughts
To conclude, this internship journey was a fruitful and enjoyable one. I learnt way more than I expected to, all while having fun along the way! My curiosity continues to grow, and I am looking forward to seeing what else I can achieve with future projects. Lastly, I want to express my heartfelt gratitude to GovTech and my team for providing me with an amazing experience this summer!
