Full-stack Quality Assurance

What is it and how to achieve it in your team?

Once upon a time, QA was a person sitting behind a laptop: finding bugs, writing test cases, doing manual testing.

Once upon a time, QA was a process that happened at the end of the SDLC: a stage-gate where a human would give 👍 or 👎 to release a software.

Once upon a time QA was about Defect Detection. It was a reactive process.

Once upon a time, QA was a functional task.

Modern QA has evolved.

Today, Quality is assured by automated tests that run early and often in the SDLC giving immediate feedback.

Today, organizations are shifting left on Quality. QA is no longer a stage gate, because Quality is baked into the process.

Today, QA is about Defect Prevention. It is a proactive process.

Today, Quality is a cross-functional goal, not just a person sitting behind the laptop finding bugs.

Today, Quality Assurance is a cultural mindset.

What is Full-Stack Quality?

Being “full-stack” is not just for developers. Modern QA Engineers have a 360 degree approach towards Quality.

Here’s a visual representation I made to indicate what it means to have Full-Stack Quality Assurance

To be a modern, full-stack QA engineer, you need to be able to think from all of these perspectives:

Engineering

• Having a kick-ass Code Coverage is not just the developers responsibility. It requires evangelizing from Test Engineers and Leads.
• It should be a cross-function team goal, driven by the QA team, to have a balanced Test Pyramid.
• Increased reliance on Test Automation and failure alerts makes release process smoother and reduces the fear of deployment — leading to a higher deployment frequency.
• What guarantees that your tests will also pass in higher environments? Data. Testing your code on environments that the end user will work with enables you to drive quality confidence backed by real data. That means part of your automated tests should be Testing in Production.

Infra / Ops

• Run load tests with your Platform Engineers and see if compute resources allocated to your application automatically scale up or down with seasonal load. Try to simulate reality.
• Make sure your organization has a tried & tested Disaster Recovery Plan.
• First, ensure you have a good CDN architecture that optimizes for trips to the origin server. Second, use modern load testing tools that enable you to simulate cross-region load and measure performance (and round trips) against the CDNs benchmarks. Third, real CDN testing will happen on real environments with real load. Echoing the point in the section above, try to Test in Production.
• Use monitoring and logging services in conjunction with all your services and tests. Enable key team members to have immediate visibility on critical alerts.

Product

• To have clarity on what the engineers will implement and what they are expected to deliver, the stories and tasks they work on need to contain definitions of what it means for a ticket to be ready and to be done.
• You highlighted a bug to a developer. The developer said this is how the PO wanted it. The PO said this is how we the designers designed it. The designer said maybe there was a gap in defining the requirements. (Life of a QA Engineer can be tough!) Why have this long feedback loop when a QA lens can be applied at earlier stages of SDLC a.k.a. Shift Left?

• Ensure that your team defines Acceptance Criteria for your user stories so that there is a shared understanding of what needs to be delivered. Want to take it to the next level? Translate Acceptance Criteria into testable code using a BDD Framework such as Cucumber.
• What happened after release? Who used the product we implemented? What were the usage patterns (heatmap and numbers)? Any bugs reported? Release metrics help motivate teams as they show a tangible impact of their work.

Data

• Who should have access to what kind of data internally and externally? If your company raises a large funding round and you are audited by Deloitte, will they be happy to see your current Data Quality and accessibility standards?
• Setup your Test Bed. Have test accounts that are production ready and can be used for demos. Have another set of test accounts that you use to play with. Socialize this information within your teams.
• Clickstream Data, if applicable to your product, can be helpful for you to optimize your test efforts and do better risk based testing

Security

• Ensure that your team has standards for Authentication, Authorization, Data encryption, Data retention. Some of these can be affected by geographic or industry-specific Quality Compliance (QC) requirements.
• Is your intern able to access the name and address of the customers of your ecommerce website? Well, it might be time for a change. Ensure that you have quality standards for anonymity level of Personally Identifiable Information.
• Prevention from SQL Injection attacks are essential as they are one of the top 10 OWASP Application security risks.

Scale

• Quality Assurance is incomplete without ensuring your application can handle your customers via performance tests like Load & Stress tests.
• Ever heard of how Netflix keeps its systems resilient? Chaos engineering is a bit part. Try to create chaos conditions to test how your app responsd to random failure.
• Want to take your application from a few thousands to a few million users?Defining to Performance SLA benchmarks for your APIs and adhering to them with progressively higher loads will be key to achieve that.
• It is critical to look beyond just overall response times and drill deep into every layer of your solution — how much time does it take for the handshake, for the initiation of a request, at the network layer, at the solution layer, and at the front-end layer. This helps you identify the choke points & circuit breakers at multiple layers of your solution.

Different Facets of Performance Testing

Quality Metrics

• There’s 2 types of quality in my opinion. There’s Technical quality — how many defects exist at a technical level. And then there’s Perceived Quality. This is more important for your customers. This is where you connect Business & Engineering. If your customers are churning out because of a performance issue or a trivial UI bug, but it’s not “high priority to fix in my sprint”, there’s a disconnect between Business & Engineering. As a good QA professional, it is your job to highlight such gaps and measure Perceived quality as a separate metric — an indicator of what the customers feel about your product quality.
• Adding to the above point, it is important to classify defect by Nature (eg: 60% functional bugs, 25% performance bugs, .. and so on) and by Feature (which features in your product have a large technical debt). This enables you to visualize quality trends over time.
• Have you defined Resolution time SLAs for major incidents? Do you have a postmortem process with mitigation steps? What’s your Mean Time to Detect (MTTD), Mean Time To Resolve (MTTR), Change Failure Rate (CFR)? These metrics additionally determine how performant your team is.
• Measure Defect Escape Ratio to determine how many defects leak undetected into higher environments. Awesome teams have DER less than 5%

Thinking beyond Application Level

When you start thinking about the above framework of Full Stack quality, it is important to think beyond just the application level and dig deeper into the following

Apply the above quality lens at each of these levels of your solution. Step by step.

Remember, cultural transformations don’t happen overnight. And they happen by nudging, not forcing. Depending on what level of maturity your team is in, it may take months to quarters to years to enable a transformation.

Just focus on taking Setp 1, and iterate from there.

To Quality Driven Engineering 🥂

You might like to follow me on twitter at launganik and on medium at kamal laungani.