Automated Testing Nightly Run
In this article, we will discuss about What is ‘Nightly Run’, Why we need it and How does it helps in our testing process.
The Background
As a software testing engineer, we often faced many blockers/issues with the testing environment or tools. This causes us have to spend more time on debugging and preparing the environment in order to complete our testing.
The Problems
Below are some common issues that every software testing engineer should have encountered:
- Very Unstable Test Environment
- Here we have always head to 80:20 principle, whereby 80% of the time we are getting our test environment to be ready and we execute our testing in the remain 20% of time.
- To adapt Automated Testing (AT), we (QA/testers) must maintain our own AT environment instead of a shared environment across the team or developers. — This is to ensure we are not depending on anyone else.
2. Regression Testing is Needed Frequently
- When we are doing some product testing, most of the time we are actually doing regression for the main feature logic. It’s obviously a repeated action, and the solution is we can ‘hire’ our ‘AT bot’ to run our scripts every night for free!
3. Outdated Testing Tools or Scripts
- During testing, we believed everyone will create some tools/test scripts. But do we really maintain and keep it up to date? Probably we just write once and leave it for some time.
- No maintenance for the tool until we need it.
4. Missing, Left Behind Testing Context/Steps
- Testing context are easily left behind if we do not find some ways to record it down.
5. Knowledge Transfer (KT) is Needed Across The Team
- Time consuming when we need to do KT among the team, especially the feature or services are brand new to us.
What is ‘Nightly Run’ ?
Nightly Run, refer to an automated build which execute every night when no one is working/using the environment.
Why we need the build to run every night? What will happens during this nightly run?
Let’s continue to explore together!
Why Are the ‘Nightly Run’ Useful to Automation Testing?
Of course, it can helps to solve all the problems that we have discussed in previous part of this article! But.. how?
Unstable/Broken Test Env
- During nightly, each of the server will be re-deploy using the latest code from the master branch of product repo. (*Wow! Such a brand new day!*)
Repeated Regression Testing
- Nightly build helps to run all the test scripts that we have written against the servers automatically. — This can help to detect if there are any code issues after few branches merge into the product master branch. (if any)
(Imagine every morning you will have ~1,000 or even more cases executed for us!)
Outdated Testing Tools
- All tools and scripts are converted into automation test cases, which will get execute every night. If the scripts are outdated and we will meet a test failures. (At this point we will know, it needs an updates!)
Missing Test Context
- AT cases should includes data preparation, the result verification and all these can regards as a snapshot of your testing crime scene!
Knowledge Transfer/ Feature Study is needed
- Helps to save time on conducting feature study sessions, we can always refer to our AT cases to have the knowledge.
“Software testers do not make software; they only make them better.” — Anonymous
Nightly Build
At this point you should have some knowledge regarding Automation, Jenkins..
What does the Nightly Build 🤖️ do secretly when the world fall asleep…?
We can have two types of build scheduled periodically every night:
- Server Build — Which take code from the master branch and starts to build from 9PM onwards.
- Test Scripts Build — Which take code from the Test Repo (repo maintain by Test engineers) and starts to build from after the last servers build until the next day.
So.. as a Test Engineer, let’s shout out “Hello World!” to our brand new working day!
Nightly Triaging & Email Reporting
After the nightly run, we will need a Test Report which includes the status of all the services.
An Email Report should be send to team members, the Management (including Tech Leads, Project Manager, QA leads and other related stakeholders).
Step 1: Prepare Nightly Email Reporting — should be the top priority task of the day for Test Engineers.
- Provide prompt AT regression results to all the stakeholders
- Immediate report of any code/services/dependencies failures before we start the day
Step 2: After we have done preparing of the email report, we should:
Review service build and deploy failures, which may caused by:
- Third-party dependencies issue
- Product code defect
- Server config issue
- Intermittent issue (i.e. unstable env, network issues, and etc)
Review the AT case/scripts failures:
- Outdated AT cases, where there is a code change in product code but the AT scripts have not updated to adapt the changes yet
- Intermittent failures, unstable AT cases, data prep issues and etc
- Test Infrastructure issue, test node service down and etc
Step 3: After investigation of the failures, we can reply on top of the Email Report Test Summary about our results of investigation. We should state the fail reason, action to be taken and who are the PIC to resolve the failures.
In a Nutshell…
Automated Nightly Run is useful for:
- Maintaining the test environment (ready to use)
- Making the process of testing more efficient
- Keeping tools/scripts up to date
- Record the testing/data preparation steps
- Teammates will eventually get the knowledge of main features as we need to review and debug on the failures of nightly.
- Help QA to know where/how can we to improve our AT cases
This article introduced one of the good way to make Software Testing Engineer’s testing process easier and efficient.
Let’s do a little tweak to make our Testing Journey fun! Stay tuned to our new stories! Cheers! 🎉
To know more on how can Jenkins helps to manage Automation Testing:
