Continuous monitoring using Flood Element and Azure DevOps
Lets face it, running systems in production is hard. There’s always the threat of a potential bug causing system down-time or unexpected load causing the system to grind to a halt. That’s why now more than ever, continuous monitoring of production systems is so critical. I’m a definite believer in shift-left testing, but there are scenarios where simulating an actual users’ behaviour in the system and monitoring the success of those actions is important. This is where functional testing at the browser level comes into play. Generally, you only want to write functional tests for scenarios that are business critical such as simulating user checkout on an online store, as these tests tend to be resource intensive.
In this article I’ll explain how to setup scheduled functional tests using Flood Element and Azure DevOps, and then monitor the success of those tests with email or Slack notifications.
If you’d just like the code for setting up the Azure DevOps pipeline with a sample Flood Element test, here is the GitHub repository for this article.
Flood Element for functional testing
Functional testing can be broadly split into protocol level testing and browser level testing. Protocol level testing focuses on non-UI related scenarios like hitting API’s, whereas browser testing focuses specifically on simulating actual users.
There are an array of browser level testing frameworks, one of the most popular being Selenium, but while Selenium offers all the functionality you’d need it also comes with a significant resource consumption price. That’s where Flood Element shines - it’s purpose built for browser level load testing and significantly enhances the concurrency of tests. The framework is based on Puppeteer and tests are written in TypeScript with a nearly identical syntax to Selenium. An additional advantage of the Flood Element framework is that it is built by Flood, a load testing service, and integrates seamlessly, allowing you to take your functional tests and easily use them for load testing scenarios.
Here is what a simple test in Flood Element looks like:
This test can be run using the Flood Element CLI using the command element run <testName>.ts --no-headless . This will execute the test and display the browser so you can visually inspect the test execution steps.
In this article I’ll focus on automating and monitoring a test and not the specifics on how to write tests. If you want to learn more about writing tests, the Flood Element documentation is really helpful and provides useful examples.
Automating and scheduling tests using Azure DevOps
Now that we have a simple test to use for our monitoring, we can setup an Azure DevOps pipeline and schedule it for specific time intervals to continuously monitor our system.
First we’ll create a simple script to execute our test and report on the results.
This script will be our entry point which we’ll run using npm. I’ve specifically excluded the helper scripts to focus on the core logic (refer to the GitHub repository for the full code). A few important things to note about this script:
- We use winston.js in order to write the results of our tests into a file. This is important as later we’ll copy this file along with Flood Element artifacts, out of the build agent as an artifact for inspection.
- We setup the script to take an array of Flood Element tests. This allows us to add an arbitrary amount of tests for monitoring.
- We report the status of our tests using
process.exit. This allows us to signal whether the npm command that was executed has passed or failed, which we’ll use for triggering notifications.
Next we’ll setup our npm command to execute the above script. In the scripts section of our package.json file we setup the command pipeline as follows:
Finally, we setup the Azure pipeline using a .yml file as follows:
In terms of execution steps, the pipeline looks as follows:
The script performs the following key steps:
- We setup a schedule using chron syntax as
chron: "0 */6 * * *. This will schedule our pipeline to be executed every six hours, which we can confirm by going to the schedules tab of our pipeline and we should see something like this:
- After installing Node and all our dependencies using npm, we execute our custom defined npm command using
npm run pipeline, which will execute all our Flood Element tests. - As a final step we copy all the files generated from our tests (these include result files and any screenshots taken) as well as our log file, which contains all the information output by our test runs.
We can inspect the results of the tests from the artifacts produced, which should produce something like this.
This artifact will include all the logs written during our test runs as well as all the files produced from Flood Element, in particular, screenshots taken during execution.
At this point we have a fully automated pipeline which we’ll be able to use for continuously testing our production system. But what if we want alerts to be sent when our tests fail?
Setting up Slack and email alerts for failing pipelines
While having this automated testing setup is extremely useful, we need a way to get notified when the tests fail, as we don’t want to sit staring at our Azure DevOps build status the entire day. This is where email and Slack alerts come into play.
Email alerts
Azure DevOps natively supports email alerts so its simple to setup. Navigate to Project Settings -> Notifications and create a new notification. You can either choose to send to all members of the project or input custom addresses to send to as shown below:
Slack Alerts
Slack alerts can be setup using the Azure Pipelines Slack App.
Detailed instructions for setup can be found here. Then once you have the Slack application setup, you’re able to setup Slack alerts in the channel of your choosing using /azpipelines subscribe https://<Azure Devops Organiszation>/<projectName>/_build?definitionId=<buildDefinitionId> .
If the build fails you’ll then receive a notification like this:
Conclusion
We’ve now gone through in detail how you can setup continuous monitoring of your Flood Element tests and be immediately alerted using either Slack or email notifications. Hopefully with this you’ll be one step closer to a peaceful production.
