Load Testing With VSTS: A Review
If you are a QA tester or DevOps developer working in the Microsoft ecosystem, you may wonder what load testing solutions Microsoft has to offer. In this article, I will cover three Microsoft testing solutions, their benefits, and their drawbacks. My hope is that these findings will help guide teams in selecting the best Load Testing tools for their respective workflows.
ACRONYMS USED:
- QA - Quality Assurance
- VSTS - Visual Studio Team Services
- VS - Visual Studio
- HAR - HTTP Archive file format
- CD - Continuous Delivery
- AWS - Amazon Web Services
Azure Load Testing (Preview)
Quickstart: https://docs.microsoft.com/en-us/vsts/load-test/app-service-web-app-performance-test
Pros:
- Tests created are automatically synced to your associated VSTS instance
- Allows for VS Web Test project uploads
Cons:
- Once performance tests are created within App Service, they cannot be deleted
- Any tests synchronized to your VSTS account cannot be deleted
- Requires VSTS account integration to host and run tests
- Limited to only two test configs: Manual and VS Web Test uploads
- Manual Test config only allows for a single URL target; few options to customize incoming traffic/request parameters
Summary: In Azure Preview, but as of writing this feature seems too scant and reliant on external services to be of direct use. The tests provided only allow you to configure number of users, test duration, and the Azure data center origin of the traffic. It seems pretty redundant relative to the CI/CD offerings of VSTS or the raw customization provided by VS Enterprise.
VSTS Load Testing (Preview)
Quickstart: https://docs.microsoft.com/en-us/vsts/load-test/get-started-simple-cloud-load-test
Pros:
- Small configs allowed for manual tests (i.e. — browser agent mix, traffic load patterns, etc.)
- Can compare any two load test runs (intra/inter-test)
- Multiple offerings to generate tests (JMeter, Manual URL, HAR upload, VS Webtest project upload)
- Test runs generate their own HAR file per agent for re-use/debugging
- Any deleted tests executions preserve their data for analysis
- Can export tests to VS as Web Test projects
Cons:
- Test customization options in portal are a bit more diverse
- URL Based and VS Web Test Project tests require an Enterprise subscription
- Finer grained test customization is reliant on JMeter/VS/Fiddler or other 3rd party tooling (i.e. — authentication/authorization generation, user agent browser types, etc)
- Unable to delete test Scenarios permanently; once generated within VSTS you cannot remove them
Summary: In preview, albeit a more robust, intermediate offering. A lack of test customization is a serious hindrance, but still an improvement over the thin shim that is the Azure offering.
URL based and VS Web Test project testing is restricted to VS Enterprise subscription holders, limiting what you can do with a lower tier subscription. On the flip side, HAR or JMeter tests allow you quite a bit of flexibility if you’re familiar with the tooling to generate them.
It is possible to integrate Load and Performance Tests into VSTS’ build pipelines for CD. As of writing, this appears to be a segregated from any existing tests you create within VSTS proper. Any tests you write must be included within the release filesystem and referenced therein.
See: https://docs.microsoft.com/en-us/vsts/build-release/tasks/test/cloud-based-load-test
VS Enterprise Load Testing
Quickstart: https://docs.microsoft.com/en-us/vsts/load-test/getting-started-with-performance-testing
Pros:
- Can be run purely locally or auto-synced with VSTS
- Syncing with VSTS duplicates test cases automatically to the cloud
- Robust tooling for complicated test scenarios (loops, conditions, dynamic binding, etc)
- Ability to program plug-ins for custom test integration
- Ability to record web activity directly into a test case
- Configuration options for a diverse set of browser agents and load patterns
Cons:
- Only offered in VS Enterprise edition (i.e. — costly licensing fees)
- Acquiring authentication tokens can be rather convoluted
- Outdated 3rd party browser agents (e.g. — Firefox 2, Chrome 2, etc)
Summary: This is clearly the most refined solution offered. If you plan to live within the Microsoft ecosystem and do any sort of complicated load testing, you’ll need a copy of VS Enterprise. Nothing else from Microsoft provides the same level of customization and ease of use.
In my experience, acquiring login tokens is a rather convoluted process. It is still doable provided you use some combination of parameter binding and response parsing, but it’s by no means as simple as using a tool such as Postman.
Conclusion
The Azure Preview offering feels like a thin shim over the VSTS solution. I’d recommend avoiding it as it just seems to generate noise rather than be of any direct use at this stage. Though given time and pressure from AWS and the nascent Google Cloud Platform, this may change.
The VSTS Preview is a step in the right direction, but still lacks features that would allow it to be truly standalone. The inability to directly integrate existing Load Tests with VSTS’ CD is a serious omission. However, depending on your tooling and team practices, you may be able to use VSTS exclusively with 3rd party tools and avoid VS Enterprise dependencies altogether.
Ultimately, VS Enterprise hosts the best solution. If you are working in a Microsoft-oriented shop, this will be your tool of choice. Its feature-set coupled with the ability to upload its own highly customized tests up to VSTS makes it the tool to use.
