Accelerating Quality: Unleashing the Power of Shift-Left Testing
1. Overview
We knew we needed a way to release new features with confidence and minimize the risk of bugs or defects in production. With our rapid growth, we were determined to stay up-to-date with the latest market trends and develop new features at a faster rate.
However, we were severely constrained by the limited number of Quality Engineers (QEs) available to test our features. This could cause several challenges for our organization:
- Increased time-to-market,
- Inadequate test coverage,
- Higher risk of bugs and defects,
- Quality compromises,
- Reactive approach to testing, and
- Increased stress on the testing team.
At Just Eat Takeway.com (JET), engineering is organized into different pillars. They each have responsibility for a particular area of the business, and autonomy to establish processes that work best for them. In the Customer Pillar at SkipTheDishes (JET in Canada), we tackled this challenge by formulating a sustainable testing strategy for our pillar, taking into account limited resources for quality engineering and adopting a long-term perspective instead of focusing solely on the present needs. Our strategy utilized the proven concept of shift left testing, highlighting the importance of teamwork to prioritize quality right from the beginning.
In this blog post, I will showcase our successful implementation of shift left testing. Through our journey, we have seen the profound impact of this approach on product excellence and customer satisfaction.
2. Shift Left Testing Principle
Shift left testing refers to the idea of integrating testing activities earlier in the software development lifecycle (SDLC), typically during the requirements and design phases, rather than waiting until the later stages of development or after the completion of coding.
Traditionally, testing has been considered a phase that occurs after development, but with the shift left approach, testing activities are moved earlier in the development process aiming at identifying defects and issues earlier, when they are easier and less costly to fix.
3. Quality from the Beginning
Figure 1 below shows the main phases in the Customer development process where Product Managers, Quality Engineers, and Developers collaborate when implementing a new feature.
3.1. Happy Path Test Case Collaboration
QEs take charge of ensuring quality starting from the second stage indicated in the diagram above. During the stage of “Create Tickets & Test Cases” product managers and/or developers present their crafted Jira tickets in the Tech approval sessions. In these sessions, they also share the happy path test cases, which are precisely defined in our test management tool, Zephyr Scale.
Happy path test cases refer to a set of test scenarios that validate the expected and successful execution of a feature. These test cases represent the optimal or ideal flow of actions, where all inputs and conditions are correct and in line with the expected behavior. The purpose of writing happy path test cases is twofold: to ensure that the feature functions as intended under normal operating conditions without encountering any errors or exceptions, and to effectively communicate these expected behaviors to the development team.
QEs have the most expertise in how to write test cases, and PMs know best how their features should function. When QEs coach PMs to write the happy path test cases, this collaborative effort helps to align expectations and effectively communicate the intended behavior of the system to the development team.
At this phase, QEs diligently review each ticket, analyzing its requirements, acceptance criteria, and associated test cases. If any inconsistencies or improvements are identified, they provide constructive feedback to the ticket author. This iterative process ensures that tickets are in an optimal state before being handed over to the development team. As a result, the development team can confidently proceed with their work, knowing they have well-prepared and well-defined tickets.
In a recent critical project within the company, our quality engineers were engaged right from the “Scope Feature” stage and they participated in brainstorming sessions with all stakeholders, and collaborated on the development of detailed acceptance criteria for the feature.
3.2. Test-Centric Development
In our development approach, every aspect of our Website and apps is supported by a corresponding test case documented in our test management tool, Zephyr Scale. This means that for each functionality provided to the user, there is a specific test case designed to ensure its proper operation. These carefully crafted and maintained test cases serve as the cornerstone of our testing activities; see Figure 2 for the workflow.
As part of the technical approval process, during the stage of “Create Tickets & Test Cases” in Figure 1, developers review the test cases and utilize them to test completed tickets. Throughout the development phase, they work closely with QEs to identify negative test cases, edge cases, and potential bugs before the final end-to-end testing takes place. Negative test cases and edge cases are added to the set of feature test cases. This collaborative effort of developers and QEs ensures that both the functional requirements and technical aspects of the system are thoroughly tested, leading to a more robust and reliable product.
After all tickets have been developed and prior to the feature being deployed into production, it is crucial to conduct end-to-end (E2E) testing.
3.3. Collaborative End-to-End Testing
End-to-end (E2E) testing ensures that all components of the feature work seamlessly together and that the entire system functions as expected. This comprehensive testing phase is essential to uncover any potential issues, validate the overall functionality, and ensure a smooth and error-free user experience.
In our team, end-to-end testing is a collaborative effort involving Quality Engineers, developers, and Product Managers each taking on important responsibilities. Product Managers communicate the E2E test dates in advance, typically 5–15 days beforehand. They collaborate with QEs to refine the test plan and create a comprehensive tester overview document. This standardized document serves as a blueprint to introduce testers to the event and the feature being tested. It is the centralized source of information and guidelines for the testing event.
QEs build the test plan in Zephyr Scale based on approved test cases and collaborate with Product Managers to develop the tester overview document. They coordinate the tester list, schedule the E2E meeting, and maintain the test result sheet. QEs also hold a triage meeting after E2E testing to discuss the priority level of any identified bugs. Developers, if involved in feature development, actively participate in E2E testing to provide support and ensure a thorough evaluation. Together, these roles collaborate effectively to ensure the smooth progression of the end-to-end testing process.
In addition to their responsibilities, QEs also take on the task of advertising and onboarding more testers for the E2E testing event. This is done to maximize the number of eyes on the feature and ensure coverage across various platforms and devices. By involving a diverse pool of testers, QEs aim to gather valuable feedback and insights from different perspectives, enhancing the overall quality of the testing process.
Moreover, each testing event serves as an ideal learning opportunity for every role within our pillar. It provides an avenue for individuals to gain firsthand experience and knowledge about a new feature and the quality process employed. Whether they are developers, product managers, or other team members, participating in the E2E testing event allows them to familiarize themselves with the feature’s functionalities, understand its behavior, and observe the quality assurance practices in action. This shared learning experience fosters cross-functional understanding and collaboration within the pillar, benefiting everyone involved.
QE work continues beyond the E2E testing phase as they work closely with the development team during warranty testing to fix critical issues and validate the feature before it is considered stable and ready for deployment.
3.4. Test Case Management
A crucial aspect of our testing approach is the documentation and maintenance of all test cases. Each of our respective backend, website, and mobile teams adheres to a Test Case Writing Standard we have developed. This standardized structure offers several advantages, including enhanced readability, user-friendliness, and accuracy by following the principle of single responsibility. To ensure consistency and quality, we also have implemented a comprehensive change process for test cases, with each modification undergoing approval by a Quality Engineer.
At the onset of development, we adhere to a strict policy where all happy path test cases must be established and ready for execution without any future dating. The meticulous documentation and maintenance of test cases serve as the strong foundation of our testing approach.
3.5. Process Adherence and Team Empowerment
We employ several strategies to ensure that our defined process is followed consistently. Firstly, we assess the success of various phases outlined above, including the documentation and maintenance of test cases, the establishment of happy path test cases, and type of bugs identified at each stage. This evaluation allows us to identify any gaps or areas for improvement and make necessary adjustments to enhance the effectiveness of our process and collaboration.
Additionally, we prioritize coaching and mentorship within our teams. Our QEs actively engage with developers, product managers, and stakeholders to emphasize their role in the quality process throughout the development lifecycle. By sharing their expertise and knowledge, our QEs become evangelists of quality, fostering a collective ownership of quality within the teams.
Moreover, we conduct timely checks and reviews to ensure compliance with the process. For instance, we employ ticket reviews to meticulously verify the acceptance criteria and test cases. In addition to ticket reviews, we employ Pull Request (PR) reviews, where QEs verify that the necessary test evidence is included in the PRs.
This comprehensive approach ensures us that all aspects of the defined process are followed and met with precision.
4. Impact on Developers
From a practical perspective in our organization, this involvement of Quality Engineers in the development process has a significant impact on developers.
- Clear understanding of expectations: By working closely with QEs, developers gain a clear understanding of the expected behavior of the feature and the specific test cases that will be executed. This helps developers align their development efforts with the desired outcomes and ensures that the feature meets the defined quality standards.
- Collaboration and bug resolution: During the internal, E2E, and warranty testing phases, QEs collaborate closely with developers to identify and address any bugs or issues that arise. Developers will then address these issues, ensuring that the feature functions as intended and meets the quality criteria. The collaboration between QEs and developers fosters effective communication and teamwork, leading to faster bug resolution.
- Continuous learning and improvement: The end-to-end testing process serves as a valuable learning tool for developers. They gain insights into how their code performs in real-world scenarios and learn about potential edge cases or usage scenarios they may not have considered. This iterative feedback loop helps developers enhance their understanding of the product and its requirements, leading to continuous learning and improvement.
- Increased confidence and customer satisfaction: Through the rigorous testing conducted by QEs, the overall quality and reliability of the feature are improved. This, in turn, increases developers’ confidence in the code they have developed. When customers receive a feature that has undergone thorough testing, with critical issues addressed, it enhances customer satisfaction and builds trust in the product.
5. Automation
We place a strong emphasis on test automation to enhance the efficiency and reliability of our testing efforts.
For our Back-End services, we have developed a framework tailored for Integration (API) testing. This framework incorporates Rest Assured and TestNG, guaranteeing consistency and efficiency throughout all BE teams. By providing configurations to each BE team, we enable seamless integration of automation into their development pipelines. By aligning their tests with the test cases in Zephyr Scale, teams can be confident in achieving precise and comprehensive test coverage for their APIs.
In addition to the Integration (API) testing, our QEs are actively working on creating load test projects per team as a crucial line of defense prior to BE releases. These load tests will help identify performance bottlenecks and ensure the smooth functioning of our services under heavy traffic conditions, thus enhancing the overall user experience.
For the Customer Website, we automated tests using Cypress. These tests run on AWS CodeBuild upon merging a branch into a staging environment, ensuring early and continuous testing of critical flows. Currently, our Cypress tests have achieved substantial coverage of critical order flows, but we are dedicated to expanding their scope further.
In the realm of Native Apps, we are making progress in automating critical flows for both iOS and Android platforms. The focus on Native Apps automation will significantly improve our testing efficiency and accelerate the delivery of reliable mobile applications to our users.
While Automation is valuable for early issue detection in the development process, it does not replace the collaboration and manual intervention outlined above. Currently, our testing resources are allocated at 80% manual and 20% automated activities. However, we aim to progress as a team and achieve a 60/40 balance between manual and automated testing, and eventually reach an even 50/50 split in the future. This gradual shift will be via increasing the automation usage for our manual regression testing. The manual collaborative endeavors in our process outlined above would still stay the same for any development project.
6. Moving Forward
As we reflect on a year of consistently implementing our improved testing process within our pillar, we have identified areas for further improvement, with a focus on collaboration and streamlining quality engineering tasks. Our journey began by shifting engineering for quality leftwards, involving developers and product teams early in the testing cycle. This transformative approach has resulted in greater confidence in our releases and a unified working method for all Quality Engineers across Customer teams.
Looking ahead, we are eager to take our efforts even further by exploring AI tools for test case enhancement and introducing internal testing at 50% feature completion point. Additionally, we aim to facilitate open communication channels for stakeholders and collaborate with the Delivery Group to ensure feature readiness for E2E testing. Embracing these next steps will drive us closer to achieving excellence in our testing process.
7. Conclusion
We successfully tackled the challenge of limited quality engineering resources by implementing a sustainable testing strategy that embraced the shift left testing principle. By integrating testing activities earlier in the development lifecycle and emphasizing teamwork, we achieved greater confidence in our feature releases and minimized the risk of bugs or defects in production.
This collaborative approach, involving Quality Engineers, Developers, and Product Managers ensures clear communication of feature expectations and thorough testing of both functional and technical aspects. The use of automation further enhances our testing efficiency and reliability. Looking ahead, we aim to continue improving our process, exploring AI tools for test case management, and further streamlining quality engineering tasks.
By fostering a culture of quality assurance and empowering the entire team to champion quality, we are well-positioned to deliver high-quality software products and maintain customer satisfaction.
Just Eat Takeaway.com is hiring! Want to come work with us? Apply today
