How we test our product

December 2018 marked exactly one year since we started the development of NiceDay. In this article, we would like to focus on how we implement automation test for our product and we will also share short stories about how we release, distribute and test our app.

Releasing and App distribution

Before we talk much about testing, let’s start with how our app is being distributed to our users. First, we build our product for three environments alpha, beta, and production. To build and distribute three types of build flavors we use Fastlane, Travis, and Slack. Whenever App/Web team are ready with their release branch, using a single Fastlane command any developer in the team can easily release the app.

In a simple description, Fastlane will trigger GitHub to create a tag and push last updated change-log in the release branch. When a release tag is created, it will trigger Travis to run several tests first such as code formatting, code quality, and unit test.

After an app is successfully built, Fastlane will release it to the appropriate store and send a report to slack that the app was successfully released. Since we are using React Native we also have the option to do native or JS only release:

Testing

At present we have eight types of tests, three of which we already discussed in the first part (code formatting, code quality and unit test). So let’s discuss the rest:

1. Change-log based test, this test is used only for patch/minor release. QA team will only test app based on change-log text that contains features that have been added, fixed, changed and removed.
2. Regression test, with more than 130 test cases QA team will do complete tests for a major release, especially in the beta and production environment.
3. Scheduled test, is the test that the QA team initiates to check the main feature and test out of test cases that we already have. Commonly we schedule regularly 1 hour, 2 days a week for such tests.
4. Daily test, is the test that is constantly running every day to check the stability of our app. It contains important test cases such as register, login, forgot password, log out, etc. These are automated tests that we run every morning before business hours.
5. Integration test is the test that we plan to develop in Q1 of 2019. These will fully test a feature that works across different platforms such as App, Web, and Backend.

Daily testing

This testing is also known as UI test or automation testing. The main reason for this testing implementation because we need to check the stability of our app. Since it is related to stability, most of the tests are repetitive and since it is a repetitive task it should be automated. For our tech stack, we have chosen Appium, Selenium, and Pytest.

For the testing script, we have set up three branches for each release type (Alpha/Beta/Production). This also makes it easy to manage a different feature than the app has built. For example, we have a chat online status feature but the feature still in alpha, so we create the test script for the chat online status feature in the develop branch when we release the application to alpha. Once the feature releases to beta in the App, we also follow the process and release the test script to Beta.

Picture 5 Branches

Test Process

We also set up several parameters in our test scriptto make testing easier

1. -I Y > pytest_report.log, to enable Slack integration
2. -D , to select a real device that we would like to use for test
3. -B , to select the browser that we would like to use for test
4. -E , to select an environment that we would like to use for test

Full of script example

pytest -s -v -D ANDROID_DEVICE_HIGH -E ALPHA_ENV -I Y > pytest_report.log

Generally, in a single day, there would be 18 tests that we run locally on our machine as we show in the Test Process image above. In the first generation of development, we have set up the test process in our local machine. Why?

1. We would like to run it in real devices.
2. It is easier to manage.
3. Somehow test script gives a different result in the different machine.

In the future, we will start implementing this in Travis but for now, our focus is to reduce QA team workload. Last words from us:

“ Test Early. Test Often. Test Automatically “

Andrew Hunt and David Thomas — The Pragmatic Programmer

So I guess that’s it. Wish it useful :)