Why dedicated QA engineers are important and how we assure quality in our eCommerce projects

The reliability of eCommerce software is probably the most crucial aspect in IT projects. Bad quality and bugs can have a direct influence on the revenue and, even more importantly, on the security of personal data. For this reason, it’s important to have a QA strategy and allocate enough time and budget for following it. In this article, we’d like to give you an overview of our strategy and what we learned from the past.

Why don’t developers simply ensure their software is bug-free themselves?

A valid question from some of our clients is why they have to pay for additional human resources to bug track quality issues of developers’ work. Why can’t developers increase their quality so that additional resources won’t be needed?

To be honest, when we started professional software development as a group of five developers 20 years ago, we did it exactly this way. Here are the reasons why this is not smart.

Complexity of software

Let’s explain this with a simple example everyone is familiar with.

Our webshop needs a simple registration form. Two inputs for the email address and password and a submit button. Simple, right?

Let’s start with testing the frontend:

Unfortunately, there are many factors that can influence the behavior of our software GUI. For instance, the browser software, the browser version, the resolution, the device, and the operating system. Multiplying them all together gives you already thousands of combinations. Testing them one by another would take days or even weeks. The complexity even grows when you add software enhancements like input validations, auto-fills, internationalization, and security features.

And on top of that, there’s the backend:

Once the browser passes the form data to our backend, we have to validate it again, save it into the database, and ensure security. Often new features are discovered during development. What shall happen when this email address exists already in the database? How to deal with bots spamming our form? Shall a confirmation email be sent after registration? The complexity grows with every added feature ticket.

With even more complex features like search, checkout, or integration with 3rd party software, the parameters grow and the number of paths to be tested explodes to an unlimited number of combinations.

It gets even more complicated when the collaboration of several developers or even the integration of 3rd party software comes into play.

Let’s assume for a moment that both frontend and backend developers fully test their part of the functionality. They know what they changed, they know what they are supposed to do. The functionality they developed will be tested, but already existing functionality that was tested earlier can break because of changes. In our example, it might be that a function that reminds you of a forgotten password will change a system, and the registration is no longer possible.

That is easy to spot when one developer programmed both of the functionalities. This becomes a bigger challenge when a feature is an effect of work of more than one team member, and even more challenging when a 3rd party library or extension was involved and that extension changes with time.

As you can see, there are a lot of moving parts to ensure that software works in every case. But does it have to work for every case and can we group them? Yes, we can and this where our QA engineers come in.

Increasing efficiency

Every developer has his own area of specialization. Most often it’s divided into frontend and backend development. So-called full-stack developers are rare nowadays, as it’s very unlikely that somebody is specialized in both. Even frontend developers do not have all the frontend knowledge. Some of them are experts in JavaScript, others are good in responsive HTML / CSS code, and some of them focus on Single Page Applications. Not every backend developer works efficiently with databases, and not every one of them knows how Elasticsearch works. Same as it’s unlikely that somebody plays both the piano like Elton John and the guitar like Eric Clapton.

With a mix of specialized people, you increase efficiency and quality. This rule is valid for almost every industry.

The same applies to QA engineers. Those people are specialized in writing test scenarios, identifying which scenarios matter and which can be implemented at a later stage (if at all), they help with writing the acceptance criteria so developers know what to focus on and they automate wherever it makes sense. By nature, every specialist has experience in his area of expertise, so they know where to take a closer look and perform a better risk assessment.

When a QA Engineer is performing functional tests, the software engineer can already work on another feature, instead of spending his time on checking all possible regressions with multiple devices.

This means QA people are not a safety net for lazy developers, as malicious tongues might say. On the contrary, they are an essential part of the development team to increase efficiency and quality. In addition to that, they introduce the four-eyes principle which we know from different areas. Would you board an aircraft that is only tested by mechanics?

Psychological factors

Let’s assume you are a developer and short on time to deliver the software increment you’ve been committed to. You might tend to go for some shortcuts in testing your software. Another aspect is the regression tests. With every new release of your software, you need to check if all the previous features are still working. With every release, you might be less careful about regression testing, especially when no regression bugs occurred in the previous releases.

A dedicated QA engineer is more strict with a developer than the developer with himself in the long run. Same as your personal trainer is with you in doing sports.

Can automated tests replace QA resources?

The short answer to that: No, they can’t.

There are many things that can be automated and we’ll explain them later on in this article. But still, there are some automated tests often written and set up by QA engineers. As mentioned before, they are specialized in the necessary QA tools and their setup.

Visual tests, for instance, are tricky because only a human can fully judge if the frontend of your software is still aesthetic in a particular screen resolution. It’s getting even harder when the frontend part is highly dynamic, which is common in the era of personalization and data-driven content.

Besides that, testing new functionalities shall always be performed manually for two reasons. First, this is the phase where QA engineers identify what can be covered by end-to-end automation, and how much effort it will be. Second, with an agile approach, and openness for change, the functionality can still be tuned and optimized. Automated tests will not suggest improvements.

Another challenging area for automated tests is checking integrations with other software. In the domain of eCommerce, this is often the payment process. Testing payment types like credit cards, the proper authorization, and the PSD2 compliance process with an automated test is only possible with a very high amount of work and maintenance of this test. Most often it’s cheaper to test this manually by a QA engineer and apply manual smoke tests after every new software release.

It’s good to have as many automated tests as possible but always keeping in mind the ROI for setup and maintenance. So as always in life, it’s about finding a good balance. Anyways, no matter if you go with automated tests or manual ones, you’ll always need an experienced QA professional for that.

What does our QA in creativestyle look like?

Before joining creativestyle, our CTO Krzysztof Daniel worked as a Quality Assurance Engineer in a big software company. Being a software developer at the same time, there was no question for him about the value of dedicated QA employees. That’s why we started hiring first QA engineers at an early stage of our company. Nowadays, our QA team consists of eight engineers.

Before we had a dedicated QA team, features were additionally tested by Project Managers, just to have a second check before the release goes to production. That was inefficient in terms of costs and processes.

Test Cases

Our QA staff is a part of our Scrum development team. It consults about the acceptance criteria of user stories and writes test cases for it. Those are written down and managed in a tool called TestRail. It’s easy to lose track when dealing with thousands of tests. Tools like TestRail help us to see what has been tested when and how, and most importantly, what hasn’t been tested yet.

Here you can download our PDF export of test cases for our open source project MageSuite.io to get an idea of how those test cases look like and what they cover.

Why test cases matter

When we talk about bug-free software, we need to define a few “types” of bugs:

The first category would be issues that are discovered in functionality as an edge case. If there is no test for that case, it means that it was not considered also during the development phase. If the issue is important, we need to spend more time on development to cover it (the same time would be also spent additionally during the initial development).

The second category of issues would be when feature B introduces a regression in feature A. Since we know the test cases for A, we are able to quickly identify this bug and spend more time on the development of feature B to cover this situation.

Finally, the third category of bugs is when the software fails an existing test case because of developers’ mistakes and faulty algorithms. Those bugs creativestyle shall fix “for free”, meaning this shall not be tracked as additional time spent on development.

In an era where little specification is written on how software shall work, test cases are the replacement. They describe how the software actually works and what you can and can’t expect.

Automated Tests

Unit tests

Unit tests are typically automated tests written and run by software developers to ensure that a part of the code (known as the “unit”) meets its purpose and behaves as intended in separation (no integration with services, 3rd party data sources, etc.)

In creativestyle, we cover as much of our own code as possible with unit tests. Especially critical functions connected with payments or price and tax calculations always have to be covered with unit tests. Magento is also pushing to write unit tests and checks this with every commit to the Magento repository.

Unit tests are written by software developers.

You can read more about unit tests here: https://devdocs.magento.com/guides/v2.3/test/unit/writing_testable_code.html

Important note: Currently in creativestyle, we do not execute unit tests in Magento. We assume that the stable version of Magento has passed those tests beforehand. Another reason is that some of those tests are failing, because our customizations change the way Magento works in some cases. Also, 3rd party extensions can have this impact on Magento Core, and quite often it will not be discovered during the unit test phase (it can be discovered by e2e tests).

When 3rd party customization causes our unit test to fail, it means that some logic was modified and our customization is impacted. It is an additional effort for us to adjust our tests or our software to be compatible with that extension, because we cannot skip tests from our build process. That’s, by the way, one of the reasons why we are very selective and strict about integrating 3rd party extensions.

Integration Tests

Integration testing is the phase in software testing in which not only code is tested, but also the integration of this code with infrastructure, e.g. database or files (such as export files with product data) and APIs.

In creativestyle, we perform an automated unit and integration test with every build before deployment to staging or production.

Integration Tests are written by software developers.

You can read more about integration testing in Magento here:
https://devdocs.magento.com/guides/v2.3/test/integration/integration_test_execution.html

Functional Tests

Functional testing is a type of black-box testing. Functions are tested by feeding them input and examining the output.

In creativestyle, we do not execute functional tests in Magento. We maintain them only when we contribute to the Magento Core. We decided to focus only on e2e functional tests described below.

Functional tests are written by software developers.

End-to-end (e2e) functional tests

End-to-end testing is a software testing methodology to test an application flow from start to end. The purpose of e2e testing is to simulate the real user scenario and validate the system under test and its components for integration and data integrity.

In creativestyle, our QA engineers define test scenarios for user stories (features). Both the software developers and QA engineers are testing those scenarios manually. Usually, the software developers are testing scenarios during developments, with a narrowed set of testing devices and browsers, and QA engineers test a wider scope including multiple devices and browsers and include tests for regressions — according to the test plan. They additionally track progress using tools like TestRail to verify they have not missed anything.

In the future, a part of those e2e tests will be automated. The setup and the maintenance of those automated tests will be done by the QA engineers.

Magento introduced the Functional Testing Framework (MFTF) to perform automated end-to-end functional testing. This framework is now stable and we will adjust it to work with MageSuite.

You can read more about MFTF here:
https://devdocs.magento.com/mftf/docs/introduction.html

E2e Tests are done mainly by QA engineers with the support of software developers.

Smoke tests

Smoke tests are a subset of test cases that cover the most important functionality of a system to reveal simple failures severe enough to reject a planned software release.

In creativestyle, our QA engineers manually perform smoke tests on the most crucial features like search, PDP, and checkout. Sometimes software developers help out to perform smoke tests as well.

In the future, the smoke test procedure will be narrowed to:

• Execution of all MFTF automated tests
• Visual, manual verification of critical path on staging environment (Homepage > Navigation > Category Page > Product Page > Add to Basket > Basket > Checkout > Login > Payment) on the most popular desktop browser and mobile device (or a different one for every Smoke Test with a round-robin approach).

Code Reviews

Code review is a QA activity in which one or several developers check a software mainly by viewing and reading parts of its source code. At least one of the persons must not be the code’s author.

The goal is to check if common practices are used and if the written algorithms are reasonable.

In creativestyle, we introduced this four-eyes principle a few years ago. It’s a strict workflow in our development and Jira process. While it increases development costs in the first step, it will reduce costs in the long run because of better readability of code, easier maintenance, and higher quality of code.

Checking the code for a specific code style is not part of our Code Reviews. For this purpose, tools are used that automatically check this according to our styling rules.

ISTQB

The International Software Testing Qualifications Board (ISTQB) is a software testing certification board that operates internationally. ISTQB Certified Tester is a standardized qualification for software testers and the certification is offered by the ISTQB. The qualifications are based on a syllabus, and there is a hierarchy of qualifications and guidelines for accreditation and examination.

In creativestyle, almost all of our QA engineers are ISTQB certified. It’s a mandatory requirement in their career path.

Lab

A very important part of our software testing is a lab where we physically have the majority of the devices. This allows us to perform test cases and to reproduce defects on:

• Most of the currently available and supported iPhones
• Most of the currently available and supported iPads
• A variety of Android phones with different versions of Android and Android One from top vendors on the market (Samsung, Sony, Lenovo, Nokia, Xiaomi), where Nokia is also a reference for “pure Android” without any vendor mods.
• Windows laptops with Internet Explorer 11 and Edge, with and without a touch screen
• Apple computers

Having this lab also allows our software engineers to connect to those devices for debugging purposes.

Having such a lab is an investment, but works better than online solutions like www.crossbrowsertesting.com. It will also allow us to execute e2e tests in the future.

Tools

Those are the most common tools we use for all kinds of software testing.

• www.gurock.com/testrail
• www.monosnap.com
• Phpcs
• Phpunit
• Phpstan
• Docker
• Jenkins CI
• Github CI Pipelines
• Phing
• Selenium and Codeception (used by MFTF)
• Selenium Grid (used with our local lab)
• www.crossbrowsertesting.com
• Email testing with real clients: Gmail, Outlook, mobile Mail on iPhone

Conclusion

In this article, you had an insight into how we define our Quality Assurance (QA) and how we fulfill requirements for quality. Many of the above-mentioned aspects of QA are implemented in creativestyle already. In the next step, we’re about to introduce automated e2e tests and visual regression tests. As you see, QA is a wide and complex topic and it takes good QA engineers to build, manage, perform, and optimize those processes. We hope we have given you an understanding of why dedicated and experienced QA engineers are needed to develop reliable and sustainable eCommerce systems.