Load testing private apps/apis in Azure DevOps pipelines using Azure Container Instances

Published in

Microsoft Azure

12 min readJan 6, 2020

A few months back there was an announcement related to the cloud load testing functionality in Azure DevOps. As per this announcement the functionality will be deprecated in a few months. The announcement does give alternatives, however what this means is that Cloud Load testing functionality from AzDO is not a long term Option going forward.

During recent work, one of the requirements we had was that after deploying an application to one of the environments inside a private VNet we wanted to run some simple load / benchmark tests against that application from within the VNet. More specific requirements and constraints are as follows:

Once the application is deployed to one of the test environments, and after the functionality of the the application is tested, prior to progressing to the next environment, A small load should be executed against one of the application’s apis, and the 95th percentile response time of the requests be recorded.
This application’s apis are not publicly exposed and the tests needed to be executed from within the application’s VNet
The cost of running this simple Load test, and recording the result is to be minimized. This included the maintenance overhead of the test infrastructure.
Azure DevOps (AzDO) is the pipeline orchestrator to be used
Two of the shortlisted tools for generating the load were JMeter and Apache Bench (AB).

This post describes one of the solutions, given the above requirements.

Key Decisions and Principles

It was decided to use Apache Bench for the solution. The solution would be designed in such a way that at a later stage it would be possible to use JMeter instead of Apache Bench
To keep the cost down, it was decided to use an Azure Container Instance (ACI) to generate the load from within the VNet. To achieve this the ACI would need to be an AzDO Agent, and have Apache Bench Installed. The ACI instance would be deleted after the Test Results are published in AzDO.
The Apache Bench (AB) output would be transformed to JUnit Format and the Results published in AzDO. Initially we would just publish the 95 percentile response time of the requests in the published report
For the purpose of this post we are using a simple web application, and using a test load of 50K requests with 10 concurrent AB users

Solution Overview

Prerequisites

Prior to the pipeline executing the following setup is assumed. The sample application to be tested is deployed to a subnet in a private network (variable name: VNET_NAME). A second subnet (variable name: SUBNET_NAME)exists where our ACI instance will be spawned, which would then generate the test load for the application
An Agent Pool (variable name: AzDO_POOL) must exist in AzDO for ACI instances to register to as agents

Pipeline Stages

The Pipeline has 3 main stages:

The first stage is executed on a hosted Ubuntu agent, and it creates an ACI instance, which is an AzDO agent, and which is created in the private network where the to be tested application is deployed. This ACI registers itself as an AzDO agent in the AzDO agent pool specified by the variable AzDO_POOL (also mentioned in the diagram as load inducer pool) . This ACI instance has dependencies like AB pre-installed (we will see how)
The second stage of the pipeline is executed using the load inducer pool agents (created in step 1). After this AB load tests against the sample web app are executed. The result of the AB test are pushed to a file. The test result file is then parsed to fetch the 95 percentile response time of all requests. A JUnit format result file is then created, which contains this 95 percentile value. The JUnit format test result file is then published and made available to AzDO. This step can be enhanced for things like making the tests fail depending on response time or the error rate threshold.
The final stage of the pipeline deletes the ACI agent created in step 1, thus keeping the load test cost at a minimum. The pipeline variable DELETE_TEST_INFRA can be set to FALSE we you do not wish to delete the ACI instance.

Key sections of the pipeline have been highlighted below. We will look at the variables , and the different stages in more detail in following sections.

Pipeline YAML

variables:
  # Variable below is also used as the name of the agent in the AzDO agent pool
  # ACI_INSTANCE_NAME: apache-bench-inducer-$(Build.BuildId)
  ACI_INSTANCE_NAME: apache-bench-inducer-1# Resource group in which the Load inducer ACI instance is created
  ACI_RESOURCE_GROUP_NAME: "internal-app-rg"# Resource group of the vnet in which the ACI instance is to be created
  VNET_RESOURCE_GROUP_NAME: "internal-app-rg"# Number of CPUs for the ACI instance
  NUMBER_OF_CPUS: 1# Memory in GB of the ACI Instance
  MEMORY_GB: 3# The Azure Subscription. Can be fetched from Azure Key vault. Currently passed from pipeline
  # Can be fetched from Key Vault
  # SUBSCRIPTION_ID: $(SUBSCRIPTION_ID)# VNet Name where the ACI Instance is to be created
  VNET_NAME: "internal-vnet"# Subnet in which the ACI Instance is to be created
  SUBNET_NAME: "aci-ab-inducer-snet"# AzDO PAT token for the ACI Instance to talk to AzDO. Currently passed from pipeline
  # Can be fetched from Key Vault
  # AzDO_TOKEN: $(AzDO_TOKEN)# The AzDO Organization name whose agent pool the ACI instance will join. Currently passed from pipeline
  # Can be fetched from Key Vault
  # AzDO_ORGANIZATION: $(AzDO_ORGANIZATION)# The Agent Pool Name for the ACI instance to Join
  AzDO_POOL: private-vnet-load-inducers# The Container Image image from docker hub to be used by the AzDO Agent as its base image
  AzDO_AGENT_IMAGE: "microsoft/vsts-agent"stages: - stage: initialize_benchmark_testing_infrastructure_in_vnet
    jobs:
    - job: initialize_benchmark_testing_infrastructure_in_vnet
      pool:
        vmImage: 'Ubuntu-16.04'
      steps:
      - task: AzureCLI@1
        displayName: "create load test infra in private vnet using azure container instance"
        inputs:
          azureSubscription: 'internal-app-rg-contributor'
          scriptLocation: 'inlineScript'
          inlineScript: |
            CURRENT_ACI_COUNT=$(az container list -o table | grep $ACI_INSTANCE_NAME | grep $ACI_RESOURCE_GROUP_NAME | wc -l)
            if [ $CURRENT_ACI_COUNT -gt 0 ]; 
            then 
              echo "ACI instance for the release already exists";
            else 
              echo "ACI instance does not exist. Creating .......";
              az container create \
              --name $(ACI_INSTANCE_NAME) \
              --resource-group $(ACI_RESOURCE_GROUP_NAME) \
              --cpu $(NUMBER_OF_CPUS) \
              --memory $(MEMORY_GB) \
              --command-line "/bin/bash -c 'apt-get update && apt-get install -y apache2-utils && /vsts/start.sh'" \
              --vnet "/subscriptions/$(SUBSCRIPTION_ID)/resourceGroups/$(VNET_RESOURCE_GROUP_NAME)/providers/Microsoft.Network/virtualNetworks/$(VNET_NAME)" \
              --subnet "/subscriptions/$(SUBSCRIPTION_ID)/resourceGroups/$(VNET_RESOURCE_GROUP_NAME)/providers/Microsoft.Network/virtualNetworks/$(VNET_NAME)/subnets/$(SUBNET_NAME)" \
              --image $(AzDO_AGENT_IMAGE) -e VSTS_TOKEN=$(AzDO_TOKEN) VSTS_ACCOUNT=$(AzDO_ORGANIZATION) VSTS_POOL=$(AzDO_POOL) VSTS_AGENT=$(ACI_INSTANCE_NAME)
              
            fi- stage: execute_benchmark_tests_and_publish_report
    jobs:
    - job: execute_benchmark_tests
      pool: $(AzDO_POOL)
      steps:
      - script: |
          echo "Execute Apache Bench Tests"
          ab -n 50000 -c 10 http://172.17.1.4/ 2>&1 | tee /tmp/ab-result.txt
          NINETY_FIVE_PERCENTILE_REQUEST_TIME=$(cat /tmp/ab-result.txt | grep 95% | awk '{ print $2 }')# install bc for floating point calculation to get time in milliseconds
          apt-get install bcNINETY_FIVE_PERCENTILE_REQUEST_TIME_IN_MS=$(bc <<< "scale=4; $NINETY_FIVE_PERCENTILE_REQUEST_TIME/ 1000 ")echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?> \
          <testsuites > \
              <testsuite failures=\"0\" name=\"Apache-Bench-95th-Percentile-Response-50K-with-10-concurrent\"  tests=\"1\" time=\"$NINETY_FIVE_PERCENTILE_REQUEST_TIME_IN_MS\"> \
                  <testcase  name=\"95th-Percentile-Requests-Served-within\" time=\"$NINETY_FIVE_PERCENTILE_REQUEST_TIME_IN_MS\"/> \
              </testsuite> \
          </testsuites>" > $(System.DefaultWorkingDirectory)/TEST-junit.xmldisplayName: 'Execute Apache Bench Tests'- task: PublishTestResults@2
        displayName: 'Publish Test Results'
        inputs:
          testResultsFormat: JUnit
          testResultsFiles: '**/TEST-*.xml'
          searchFolder: $(System.DefaultWorkingDirectory)
          testRunTitle: 'Apache-Bench-50KRequests-10ConcurrentUsers-95thPercentileResponseTime'- stage: clean_up_test_infrastructure
    jobs:
    - job: delete_aci_instance
      pool:
        vmImage: 'Ubuntu-16.04'
      steps:
      - task: AzureCLI@1
        displayName: "clean up load test infra from private (delete azure container instance)"
        inputs:
          azureSubscription: 'internal-app-rg-contributor'
          scriptLocation: 'inlineScript'
          inlineScript: |
            if [ $(DELETE_TEST_INFRA) == "TRUE" ]; 
            then 
              echo "Deleting ACI Instance ......";
              az container delete --name $(ACI_INSTANCE_NAME) --resource-group $(ACI_RESOURCE_GROUP_NAME) --yes
            else 
              echo "Not deleting ACI Instance as per pipeline configuration .......";
            fi

Pipeline Variable Details

In the example shown most of the variable values are set directly in the pipeline yaml. Values of secrets are injected at runtime using pipeline variables. we can easily use the key vault task to fetch the value of variables containing secrets from Azure Key Vault, however that is not the focus of this post.

Variables injected when executing the pipeline:

The following variables are injected by pipeline

AzDO_ORGANIZATION: This contains the value of the Azure DevOps Organization, to which the ACI instance will attempt to register in the AzDO agent pool.
AzDO_TOKEN: this is the PAT token for AzDO. The ACI instance will use this to connect to the AzDO Organization
SUBSCRIPTION_ID: This is the Azure subscription ID of the VNet and Subnet where the ACI instance will be spawned
DELETE_TEST_INFRA: If set to TRUE, stage 3 of the pipeline will delete the ACI instance after the load test is complete.

Variables set directly in the pipeline yaml:

ACI_INSTANCE_NAME : This is the name of the ACI instance. This is also the name of the AzDO agent created in AzDO agent pool
ACI_RESOURCE_GROUP_NAME: Name of the resource group where the ACI instance is created
VNET_RESOURCE_GROUP_NAME: Name of the resource group which contains the VNet and Subnet where the ACI instance is created. This can be same as the ACI instance resource group.
NUMBER_OF_CPUS: For the ACI Instance
MEMORY_GB: For the ACI instance
VNET_NAME: Where the ACI instance is created, this is also the Vnet for the web application to be tested
SUBNET_NAME: Where the ACI instance is created
AzDO_POOL: Name of the AzDO pool to which the ACI instance will register as an agent
AzDO_AGENT_IMAGE: name of the docker container image (from docker hub) of the AzDO agent

Pipeline Stage 1 Details

We will focus on some of the key lines of this stage:

In the stage above (lines 44–72) we use the Azure CLI task. We mention the azure subscription (azureSubscription: ‘internal-app-rg-contributor’) to be used for this task. A service connection with this name needs to exist in AzDO, with permissions to spawn an ACI instance in the mentioned subnet and resource group.

Next we check if the required ACI instance already exists (line 57), and if it does not exist then we create an ACI instance using the az container create command. One of the key parameters of this command is (on line 67) shown below

--command-line "/bin/bash -c 'apt-get update && apt-get install -y apache2-utils && /vsts/start.sh'" \

In the above parameter we modify the start command to install Apache bench to the ACI instance (using apache2-utils). If we need to install JMeter instead it can be done by using the relevant packages here, as show in the following post

In line 70 (shown below) we pass AzDO information to the ACI instance (as runtime environment variables) to register as an agent with our AzDO organization

--image $(AzDO_AGENT_IMAGE) -e VSTS_TOKEN=$(AzDO_TOKEN) VSTS_ACCOUNT=$(AzDO_ORGANIZATION) VSTS_POOL=$(AzDO_POOL) VSTS_AGENT=$(ACI_INSTANCE_NAME)

After this stage ends we have an ACI instance in the required private network, using which we can generate AB load for our application. It takes around 15 minutes before the agent is available in Azure DevOps, majority of this time is taken in pulling the container image for ACI. One way to bring this time down is by hosting this container image in Azure Container Registry (ACR) in the same region as the ACI, but his will increase the cost of the solution by a bit.

The screenshot below shows the agent being available in AzDO

Pipeline Stage 2 Details

Step 1 of Stage 2: Execute tests and generate the tests results file in JUnit format

Line 81 is where execute the AB tests against our test website

ab -n 50000 -c 10 http://172.17.1.4/ 2>&1 | tee /tmp/ab-result.txt

The above ab command generates a load (http get) of 50K requests using 10 concurrent connections against the endpoint “http://172.17.1.4/” . This is the private IP of our app. This will need to be modified as per requirements of your application. We also tee the AB results to the file “/tmp/ab-result.txt” so that we can parse the results and fetch the 95 percentile response time of the requests.

The Sample Apache Bench result output is shown below

This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/Benchmarking 172.17.1.4 (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Completed 50000 requests
Finished 50000 requestsServer Software:        nginx/1.14.0
Server Hostname:        172.17.1.4
Server Port:            80Document Path:          /
Document Length:        182 bytesConcurrency Level:      10
Time taken for tests:   10.129 seconds
Complete requests:      50000
Failed requests:        0
Non-2xx responses:      50000
Total transferred:      17150000 bytes
HTML transferred:       9100000 bytes
Requests per second:    4936.29 [#/sec] (mean)
Time per request:       2.026 [ms] (mean)
Time per request:       0.203 [ms] (mean, across all concurrent requests)
Transfer rate:          1653.46 [Kbytes/sec] receivedConnection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.7      1      44
Processing:     0    1   0.6      1      45
Waiting:        0    1   0.6      1      45
Total:          1    2   1.0      2      47Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      2
  90%      3
  95%      3
  98%      4
  99%      5
 100%     47 (longest request)

As can be seen we get a lot of details like the number of failed requests, transfer rate, etc. What we are interested in is the 95 percentile response time (which is in milli seconds).

In following lines (82 to 87 shown below) we retrieve the value of response time converted in to seconds (for injecting into the JUnit format results file). To do the conversion from seconds to milliseconds we install and use the bc utility.

NINETY_FIVE_PERCENTILE_REQUEST_TIME=$(cat /tmp/ab-result.txt | grep 95% | awk '{ print $2 }')# install bc for floating point calculation to get time in              milliseconds
          apt-get install bcNINETY_FIVE_PERCENTILE_REQUEST_TIME_IN_MS=$(bc <<< "scale=4; $NINETY_FIVE_PERCENTILE_REQUEST_TIME/ 1000 ")

In the next few lines shown below (Lines 89 to 94) we generate the JUnit format test results file

echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?> \
          <testsuites > \
              <testsuite failures=\"0\" name=\"Apache-Bench-95th-Percentile-Response-50K-with-10-concurrent\"  tests=\"1\" time=\"$NINETY_FIVE_PERCENTILE_REQUEST_TIME_IN_MS\"> \
                  <testcase  name=\"95th-Percentile-Requests-Served-within\" time=\"$NINETY_FIVE_PERCENTILE_REQUEST_TIME_IN_MS\"/> \
              </testsuite> \
          </testsuites>" > $(System.DefaultWorkingDirectory)/TEST-junit.xml

The only value we are injecting is the 95 percentile response time. As per your requirement other values like failures can be injected in to this JUnit format results file.

Step 2 of Stage 2: Publish test results to AzDO

- task: PublishTestResults@2
        displayName: 'Publish Test Results'
        inputs:
          testResultsFormat: JUnit
          testResultsFiles: '**/TEST-*.xml'
          searchFolder: $(System.DefaultWorkingDirectory)
          testRunTitle: 'Apache-Bench-50KRequests-10ConcurrentUsers-95thPercentileResponseTime'

For this we use Publish results task. We set the format to JUnit. Other formats like NUnit etc are also available. We also need to specify the folder and file name pattern for the results file.

Pipeline Stage 3 Details

In this stage if DELETE_TEST_INFRA variable is set to TRUE while executing the pipeline then we delete the ACI Instance, so that resource costs for that are no longer incurred.

Pipeline In Action

Lets have a look at the pipeline in action

First we Run the pipeline and set the DELETE_TEST_INFRA to TRUE

After this the 3 phases of the pipeline execute sequentially as shown below:

After stage2 we can see the test summary in the test tab as shown below:

After stage 3 the ACI instance is deleted and the pipeline execution is complete

Further Enhancements

The pipeline can be be optimized by adding more information to the published results. We can also fail the release based on error and response time thresh holds received from the AB tests.

Spawning the ACI instance in private subnet with the required software and registering as an AzDO agent currently takes a very long time. This is due to the time taken to pull the container image (microsoft/vsts-agent) for the ACI instance. This time needs to be reduced. One option as mentioned above is to host this image in an Azure container registry which is in the same region as the ACI, however we need to weigh the minimal cost trade off.

JMeter is more commonly used for Load testing, and tweaking the above pipeline to use JMeter instead of Apache Bench would increase the pipeline’s reusability.

Thanks for reading this post. I hope you liked it. Please feel free to write your comments and views about the same over here or at @manisbindra .