Analytics & QA — Do they go hand in hand?
My short answer will be — Yes! Don’t be lazy…read why :)
In the past years many companies have moved to work with continuous development and deployment. In this new environment, dev teams are constantly sprinting to provide a high quality product with added value to increasingly unforgiving users. For QA teams this is quite challenging. Testing in this FAST world and making sure they find the most valuable defect as fast as they can is no trivial task.
Testing without a calculated plan can result in testers’ time spent on areas of the product that yield little value and are less used. But it’s possible to significantly reduce the amount of testing you do while maintaining reasonable levels of confidence through regression.
How you ask….?
When you think of QA, there are couple of topics that come to mind: bugs, automation, testing, production, scale, performance, “fit for the purpose”, regression etc… What about analytics? Is it also one of QA’s topics you would think about?
QA can use business data to learn about their users’ behavior, patterns, most/least used features, etc., thus able to make data based decisions instead of educated guesses. Analytics can highlight the risky areas, help with focusing your tests efforts, and help with bugs prioritization.
You can use any analytics tool to extract the information and data on your product. Let’s drill it down a bit…
User Data
Extracting, analyzing and monitoring user data such as main flows/ site areas, most used/engaged features, which OS/device/browser is common among your users and which page has low/high bounce rate — could even indicate a bug.
All of the data above and much more can help to prioritize the tests that should be covered as part of regression testing / exploratory testing. Highlighting high risks area and flows indicate broken flows and also indicate potential bugs.
User Impact
User data can provide a good starting point to prioritize bugs. Meaning, as part of the bug analysis investigation, extract data of how many users experience this bug, encounter this broken flow and percentage of functionality usage of the page( X% of the users who enter the page experience the bug). This data can certainly help the product manager to better handle bug prioritization.
Monitoring & Proactive Approach
Various tools provide different monitoring alerts, providing crucial insight allowing you to set alerts on main business flows as an additional product safety net aside automation tests. This allows you to control the main business flows such as: number of user registration/login/add to cart clicks or any other business metrics. Once you get an alert on a drop in one of the metrics this will probably indicate a bug in that area (don’t forget to set a threshold to make sure you wont get false positive alerts).
These tactics change the QA and makes it more proactive. Identifying bugs as soon as they hit production!
Analytics Data can provide fast and accurate answers!
Next time you open a bug, try asking the hard questions: what is the user impact and exposure rate? How severe is this bug? What can be done to monitor this business flow so we won’t encounter this bug again?
