The When and How of Load and Stress Testing

From feasibility assessment to discovery: An engineer’s cheat sheet

Validating an application’s non-functional requirements is usually tricky, often ambiguous, and at times prone to failure if not scoped and executed correctly. This blog post aims to help consultants and engineers who work with customers plan and strategize an approach for load and performance testing, develop a solid plan outlining the current and future state, identify testing requirements, and most importantly, set the right expectations around end goals.

From my experience working for over 10 years in this space, I have observed that, in most cases, customers will rarely start a discussion around load or scalability testing unless they are led into one or are already facing issues that are beginning to affect their bottom line and impacting end-user experience. Performance testing is almost always an afterthought, but as Builders, we can make a difference by helping our customers identify the need early on.

First things first

I feel it’s vital for me to stress that not all applications need to be load tested.

Most ecommerce or financial websites that are prone to high and concurrent user traffic become strong candidates for load and stress testing. The key word here is concurrent. It really doesn’t matter how many users visit the site. What matters is how many users are concurrently active on the site and for how long. This is usually the deciding factor if an application needs to be vetted out for load and performance.

There is a very important distinction that is often missed. Performance testing is not the same as load or stress testing. While load or stress testing is a type of performance testing, the opposite may not always be true. Performance testing should be an integral part of any application development project; load and stress testing should be done on a case-by-case basis.

Questions to ask

Are you wondering how to assess the feasibility of load and stress testing for an application? Here are a few questions to help you out.

· Are there any existing performance issues reported in production that are traffic or usage-related?

· Are there ever going to be periodic spikes in user traffic or transactions that may not be always user-related?

· Was load or stress testing done in the past? If yes, then load testing should be scoped out.

The discovery phase begins once there is a mutual agreement on the need for load testing. This is when the consultant needs to spend most of their time evaluating the application in greater detail and setting the right expectations around what performance goals are reasonable and which ones aren’t. Having open-ended goals for load testing is a recipe for failure and an unhappy customer.

What is Performance Testing?

I like to break performance testing into three high-level categories:

1. Application code performance

Developers leverage dev tools to make code performant. For example — CPU and memory diagnostic tools are used to identify potential hotspots like memory leaks, stack overflow, missing indexes in database tables, and other similar issues. All performance issues need to be adequately identified and plugged before proceeding with load testing.

2. Browser client performance

This category applies mainly to browser-based applications. A quality engineer will partner with developers to devise a test strategy to measure the application’s client or browser-side performance. This test strategy includes browser caching, redirects, chattiness, etc. Load testing should only focus on the server side or the backend performance.

3. Load and stress testing

Load testing aims to uncover backend performance issues primarily related to server infrastructure, networks, and databases.

Here’s a list of items that will need to be flushed out during the performance testing process of the discovery phase, some of which will continue to be refined as the test progresses.

· Identify end user scenarios

· Identify test weightage or test mix under peak load

· Test data

· Define baseline. Follow a green and a red baseline technique. Have a separate baseline for long-running stress tests. A green baseline is when the application is under regular load. A red baseline is when the application is subjected to a peak hour load. The stress test is run for an extended period on a constant load to gauge system resiliency. You may also consider doing a green and a red baseline for stress tests.

· Identify dependencies. These are usually third-party API calls, etc. that may need to be stubbed out or mocked at the time of testing.

· Test environment for simulation

· Tooling and licenses

Customer Interviews

Interviewing the customer is an essential part of discovery. Use this time to get a good understanding of the application, information about any performance testing done in the past, the layout of the test environment, including how much of it is different from production, and most importantly, any performance issues currently impacting the application.

For web-based applications, your attention needs to be on browser performance and backend. Focus your interviews on the following high-level topics:

· Test environment simulation (including hardware and network)

· Test data requirements

· Identify test scenarios, test mix, and test run weightage

· Client and server-side performance diagnostics

· Test tooling

An effective discovery phase means you’ve developed an accurate scope of work, identified an unambiguous set of deliverables, set the right expectations around requirements and outcomes up front, found the right human resources and tools for the job, and lastly, identified and prepared for any risks or challenges that can potentially occur to deliver the best testing experience for the customer.

Three Stages of Questioning During Discovery

Stage 1

Familiarize yourself with some of the high-level aspects of the application under test and production environment design. Understand why performance testing is necessary and whether there is any history of performance issues.

Let’s dive into some of these high-level aspects in more detail.

Brand new app or a rewrite. Customers may not be able to provide you with performance metrics. Work with them to outline some of this information.

Service Level Agreements (SLAs). Identify key transactions and define SLAs. Consultants should remember that the customer may not be able to come up with an SLA. In this case, you’ll need to use some rule-of-thumb metrics.

Load testing history. Request that the customer provide you with information about past performance testing.

On-prem or Cloud? While the final outcome is similar, the strategy and planning for each will differ. For on-prem applications, server investments are made upfront; for cloud-based applications, costs are based on usage.

Test tooling. Identify tooling and licensing requirements for the following:

· Test development (Apache JMeter, Gatling, etc.)

· HTTP proxy (Fiddler, YSlow, browser dev tools, etc.)

· Test reporting and client performance metrics (Blazemeter, JMeter native reports, etc.)

· Database diagnostics

· Network diagnostics

Stage 2

At this stage, you’ll want to focus on the tech stack. Let’s break up the tech stack into some high level layers and delve deeper into each one.

Code level performance testing

Find out if developers have done any code level performance testing before proceeding with load testing? If yes, then that’s good news. Seek to gather the following information:

· Were there any code performance rules run as a part of static code analysis?

· Were any CPU or memory profiling tools used to check for memory leaks etc.? If yes, are there reports that you could review?

· Were any database diagnostic tools used? If yes, what issues were found (if any), and how were they addressed?

· Were any network diagnostic tools used?

Load test environments

Find out what kind of test environment they are currently using. Is it a dedicated test environment that is isolated or are they using a shared environment? Customers generally do not have a dedicated test environment for performance and load testing. That’s a risk as it can skew the test results. Most customers use an existing QE or the dev environment to run any performance tests. While that may be sufficient for some cases, keep in mind that this arrangement is not ideal.

Things to keep in mind

The load testing environment should ideally mimic the production environment. It doesn’t need to be a 1:1 match, but it has to be proportionate in number. However, the server configurations should match for accurate test results.

For example, the customer has an environment with six web servers behind a load balancer, six application servers running the API backend, and two database servers. One of the database servers is used for a failover. When constructing the test environment, you don’t need to match the exact number, but you can proportionately reduce this number, as well as proportionately cut down the load the tests will generate.

End user behavior

Talk with your customer about peak-hour traffic patterns and what actions their end users generally take. For example, if you’re working with a B2C ecommerce site, you may want to know how much traffic they usually expect during a sale, on weekends, or during the holiday season. You may also ask if they have any server-side logs, as that is often a good source of information.

This information will inform how you develop your test scripts and design load profiles when running tests. An inaccurate test script or load profile will only produce results that may cause a false negative alarm.

Test Data

Test data discussions need to happen at two different times during the test engagement. First, at the time of discovery. Be aware that having the test data necessary to develop the tests should be good enough. Later, you will need to work with your customer to gather test data information required to run load and stress tests. Some customers may have already performed a similar testing exercise in the past, in which case they may know what test data might be necessary for the long haul tests.

Stage 3

This stage is necessary if the customer has a history of performance testing to reference. It can focus on diagnostic logs, test data, the test environment, reports, and most importantly, what they believe was lacking, which is why we have been asked to rethink the initial approach.

Some questions to ask

· What are the performance hot spots (application code, database, etc.)?

· What was the structure of the team?

· How were the tests developed? Can the customer share the test scripts for you to review?

· Can they provide you with test reports, test logs, and/or the test environment setup?

A discovery may last 2–4 weeks, depending on the complexity of the application, team structure, and most importantly, the team’s readiness to provide the information required by the end of the discovery phase to develop the first draft of the test plan.

Additional considerations

Before I wrap up, I want to share a few things for you to keep in mind while conducting the discovery process.

· Performance or load testing. While this can be driven primarily by the quality engineering team, development and architecture teams must be equally involved from planning to execution.

As many of you know, there is no end to how much performance tuning you can do to make your application performant, so don’t sweat it. Identify achievable goals and have a solid plan to get there. That’s it!

· Cost considerations. Executing performance and load testing can be costly, but not doing this correctly rarely produces results that help with constructive decision-making. Also, keep in mind that not all applications need to be performant. They only need to meet some basic metrics.

· The impact of the cloud. The cloud has brought about a paradigm shift in how modern-day applications are developed, deployed, and tested. Deploying an application in the cloud won’t necessarily solve a performance problem. You may also have to throw in more resources to have your application run optimally. Cost can creep up on you when performance issues are not addressed in time.

I sincerely hope is that this blog post will be helpful to consultants who work with customers on performance and load testing projects. I wish you the best of luck!