How did we create an effective test environment?

Test environments are one of the most important stages of the software development process and provide an environment for testing new features before release. The stable process of such systems is one of the biggest factors affecting quality.

In a big technology organization and distrubuted systems, testing associated with multiple teams, services and domains can often be a problem.

The reason of that may be the data sets which covering the scenarios to be tested are not found in a healthy way in the test environments, have a single test environments and deploying packages that have bugs to the test environment.

In this article, I would like to summarize the problematical points of our testing environment and explain the improvement methods that we apply to increase quality.

What were our difficulties?

In the beginning, there were 3 main difficulties. Due to these difficulties, the test times were getting longer and some scenarios could not be tested. These difficulties are as follows;

Our application lists the stores suitable for the selected location in the Market, flower and water verticals. The intention was to test the selected location in all verticals by finding suitable stores according to service types and transportation methods, however, all of these scenarios could not be tested because the stores suitable for these criteria were not defined in the test environment.

Another difficulty was that there were no examples of product data for each of the product types such as weight, piece or package in the selected test stores.

Our last difficulty which was impeding our process was the lack of active slots for each delivery method. Stores operate with instant delivery, appointment delivery, or by using both. During the tests for the completion of the order, the lack of suitable delivery times for the stores in each delivery method was hindering the process.

What have we done?

In order to eliminate these difficulties, we first have chosen addresses belonging to 3 different locations. We have then created test users and defined these addresses for these users. We have opened stores according to all working types and determined locations. We have added products to the created stores according to the desired product types. We have defined slots available for 3 days according to each delivery method so that orders can be placed from selected stores. Finally, we have registered the cards which are suitable for payment methods and will work smoothly in the test environment for the test users.

How do we keep it up-to-date?

This process, which we carried out manually at the beginning, was being affected due to the fact that many teams were running different tests at the same time and changing the data used during the tests according to their own needs. We needed an automation solution to keep the process stable and up-to-date. These solutions are as follows;

We have developed automations which update the information, stock amounts, the working types and transportation methods of the products in the stores, according to the definitions which we had previously determined.

We, in total, have developed 4 automations to run periodically every morning which define slots suitable for each delivery method to use for order processes, reactivating slots with full capacity, and lastly updating user addresses and registered credit cards.

What technologies did we use?

We decided to use Cypress in our project because we need to develop both UI and API tests together. Cypress is a framework that has become popular in recent years. It is a powerful solution for writing tests of web applications. It also has the ability to write REST API tests.

The biggest advantages of Cypress are that it is javascript-base, tests run directly on the browser, and it allows us to write both front-end tests and API tests without any additional libraries.

Also, one of the nice things about using Cypress is to write API tests that when we select the request made on the Cypress Log screen, we can see all the response data in the browser console. It allows us to quickly inspect and debug response from APIs.

Using Cypress’s paid plan allows us to run parallel tests to the extent we want without any development, and to examine the test reports on the detailed dashboard screen. It helps us to send the generated report to the Slack channel that we have determined with the integration made through the dashboard.

What was our gain?

Even people who have never done a test on the project before can log in with the users on the document, we have created for testing, and determine the store type suitable for them and the products they want to buy; they can create orders quickly and smoothly, and testing their own stages.

In addition, thanks to this created data, our developer teams are able to obtain much more positive data and progress their local tests more accurately without the need for mock data, thus the bug rates in the developed works being decreased. In this way, the tests performed on the QA environment are completed in a short time and the delivery times of the tasks are considerably accelerated.

Since the data we have created in the test environment includes the many scenarios, the rate of possible incidents on the live has decreased.

I would also thank my teammates, @berke.bagiroz & @osman.toroman, who contributed to this process.