How to reduce the risk of vendor lock-in with your end-to-end test automation
Vendor lock-in is a situation when a customer becomes reliant on a vendor for products and services and cannot easily switch to another vendor. This can also occur with software products, particularly software development tools. When it comes to selecting the appropriate toolset for end-to-end testing automation, there are numerous options available. However, it is common to take into account only business and technical requirements and neglect the long-term strategic decisions. In this article we will explore the situation of vendor lock-in within the end-to-end testing project and propose a solution that will help to reduce the risk of being stuck with a particular testing automation framework.
What is vendor lock-in?
Vendor lock-in describes a situation where a customer becomes reliant on a specific vendor’s product or service and faces difficulties in switching to a competitor’s offering without incurring significant switching costs, complexity, or inconvenience. Investing time and effort in a single technology can come at a cost. Once the current technology becomes unsupported, faces performance issues, or lacks new features offered by the latest technology, it may require a transformation.
Not just cloud services, but test automation too?
Vendor lock-in risks are often solely discussed in the context of cloud services, however, cloud computing is not the only field where vendor lock-in scenario is likely to happen.
Tech industry in general are full of such examples. For instance, Microsoft software has a significant degree of vendor lock-in due to its extensive range of proprietary APIs. Office suite vendors use closed file formats, forcing a group of users who wish to share files to purchase the same product.
Certain dependency on a vendor exists even in consumer technology. In the early days of Apple iTunes, consumers who purchased music from the iTunes store could only play it on Apple products, such as the iPod. Each file was encoded in a proprietary format to ensure that the music remained within the company’s ecosystem. Or take branded coffee pods, such as those offered by Keurig, are only compatible with Keurig coffee machines. The company has remained a source of vendor lock-in despite partnerships between other companies in the industry to standardise the size of different pods.
To a certain extent, we commit ourselves when we choose to write code in a specific programming language. This choice also means selecting a set of standard libraries, a package manager, and a build tool.
Similarly, we might commit ourselves to a specific end-to-end test automation framework, losing the flexibility to easily upgrade to a more efficient, faster, and robust option when it’s available.
So many test automation choices!
There are numerous options available in the market today for end-to-end testing frameworks with the most popular choices including:
Making a decision from many options isn’t easy, particularly when considering the long-term impact on your project.
Making your test automation choice
But let’s pretend you have made your research and selected a framework — Puppeteer. It seems to meet your use case and provides the features you need. It’s also the most popular automation framework based on recent trends.
As the years pass, your codebase expands with tens and hundreds of test automation scenarios.
New test automation framework is released!
As a responsible tech professional you review your technology choices from time to time and try to use the best tools available on the market. You observe more recent trends and realise that there is a newer test automation tool that according to your research will help you execute your automation faster by running tests in parallel, perform testing using multiple browsers and provide multiple ways to debug your test scenarios which will make your developers happy.
Recent trends confirm that the industry appears to favour the new technology and spending some extra time to research what it will take to migrate you decide to switch your code base to Playwright.
What is with Playwright?
Indeed Playwright makes a good choice these days for most of the end-to-end automation use cases providing plethora of great features including:
- Multiple programming languages, such as Javascript, Typescript, Python, Java, C#.
- Multiple browsers including Chromium, Firefox, Edge, Chrome, and Safari.
- Cross-platform with testing on Windows, Linux, and macOS, locally or on CI, headless or headed.
- Mobile web testing by using native mobile emulation for Chrome on Android and Mobile Safari. The same rendering engine operates both on your desktop and in the cloud.
- No Webdriver dependency as Playwright directly executes the command on the browser.
- Supports all the selectors.
- Supports parallel execution.
- Solid support of modern frameworks such as React, Angular, and Vue
- Multiple ways to debug with Playwright Inspector, Playwright Trace Viewer, Browser Developer Tools, Visual Studio Code debugger, Verbose API logs and more.
- Support of multiple assertion libraries and configurable with Jest, Jasmine Mocha and more.
Test automation code migration can be painful
While the two frameworks used as examples in this article have similar approaches to test automation, the migration may still take some time and effort depending on the size of your codebase. A situation like this might remind us of a vendor lock-in: we would like to switch to a newer technology that will bring us later features, but we are bound. But are there ways to make choices in the first place that will put us in a better position?
Abstraction to the rescue
Abstraction in software development is the process of hiding the complex details of a system, function, or method to simplify its interaction and usage. This allows software developers to use functionalities without understanding their internal working, focusing instead on what the function does (outcome) rather than how it does it (implementation).
Cloud computing, the tech industry where the term “vendor lock-in” is the most prevalent, solves these risks in the same way by abstracting the layers of a cloud platform using tools such as Terraform and Ansible enabling easier modelling between cloud service providers. But will this work with end-to-end test automation tools?
Abstract away your test automation framework
CodeceptJS is an example of a test automation framework that implements abstraction on top of other automation frameworks. Calling itself a driver agnostic tool, it provides the same APIs to run across different drivers (helpers) including Playwright, Puppeteer, WebDriver, Appium, TestCafe, Protractor, Nightmare and more. CodeceptJS architecture explains how test scenarios will run depending on the helper configuration.
You can install a helper by providing configuration options to CodeceptJS. Let’s take a look how an imaginable migration from Puppeteer to Playwright looks like using an abstraction such as CodeceptJS.
Start with the first test automation choice
The configuration is set in the codecept.conf.ts file, which is created during the init process. Inside the config file you can enable and configure helpers and plugins, and set bootstrap and teardown scripts.
Below is a sample configuration for Puppeteer driver (helper).
// codecept.conf.ts
import 'dotenv/config'
import { setHeadlessWhen, setWindowSize } from '@codeceptjs/configure'
setHeadlessWhen(process.env.HEADLESS === 'true')
setWindowSize(1600, 1200)
export const config: CodeceptJS.MainConfig = {
name: 'codeceptjs-playwright-typescript-boilerplate',
tests: './tests/**/**.spec.ts',
output: process.env.OUTPUT_PATH,
helpers: {
Puppeteer: {
url: process.env.BASE_URL,
show: process.env.HEADLESS === 'false',
restart: false,
waitForNavigation: 'load',
waitForAction: 1_000,
waitForTimeout: 45_000,
},
},
include: {
I: './CustomSteps',
HomePage: './pages/HomePage',
},
multiple: {
desktop: {
browsers: [
{ browser: 'chromium' },
{ browser: 'firefox' },
{ browser: 'webkit' }
]
},
},
plugins: {
retryFailedStep: {
enabled: true
},
screenshotOnFail: {
enabled: true
},
stepByStepReport: {
enabled: true
},
}
}This configuration sets up CodeceptJS to run in both headless or headful modes, testing three major browsers and including some popular plugins to make screenshots and retry on failed step.
npm install --save-dev puppeteerInstall Puppeteer helper dependencies and follow the set up process.
Migrate to a new test automation framework
In our previous scenario we decided to migrate our codebase to Playwright. We will do the same action here. First, we will add another configuration file codecept.playwright.conf.ts.
// codecept.playwright.conf.ts
import 'dotenv/config'
import { setHeadlessWhen, setWindowSize } from '@codeceptjs/configure'
setHeadlessWhen(process.env.HEADLESS === 'true')
setWindowSize(1600, 1200)
export const config: CodeceptJS.MainConfig = {
name: 'codeceptjs-playwright-typescript-boilerplate',
tests: './tests/**/**.spec.ts',
output: process.env.OUTPUT_PATH,
helpers: {
Playwright: {
url: process.env.BASE_URL,
show: process.env.HEADLESS === 'false',
video: true,
trace: true,
browser: 'chromium',
ignoreHTTPSErrors: true,
waitForTimeout: 45_000,
waitForNavigation: 'load',
keepVideoForPassedTests: false,
keepTraceForPassedTests: true,
},
},
include: {
I: './CustomSteps',
HomePage: './pages/HomePage',
},
multiple: {
desktop: {
browsers: [
{ browser: 'chromium' },
{ browser: 'firefox' },
{ browser: 'webkit' }
]
},
},
plugins: {
retryFailedStep: {
enabled: true
},
screenshotOnFail: {
enabled: true
},
stepByStepReport: {
enabled: true
},
}
}This file is identical to the previous file with the only amendment: the helper we are using now is Playwright.
npm install --save-dev playwrightInstall NPM dependencies for Playwright and this is it. We are now running our codebase in Playwright! The only change is the configuration!
Well, it might not be that simple
That’s right. Depending on the other helpers and plugins that you are using it might be more than just a configuration change, but the benefits are clear: abstraction provides us a way to stick to the same APIs migrating from one underlying framework to the other.
Automate both web and mobile apps
Another benefit of an abstraction approach is that you can abstract away more application types with one codebase if projects are tightly coupled from the business logic perspective. It’s possible to automate both web and mobile applications with CodeceptJS. Configure Appium helper to take advantage of that. A sample Appium configuration is shown below.
// codecept.appium.conf.ts
import 'dotenv/config'
export const config: CodeceptJS.MainConfig = {
name: 'codeceptjs-appium-typescript-boilerplate',
tests: './tests/**/**.spec.ts',
output: process.env.OUTPUT_PATH,
helpers: {
Appium: {
platform: 'iOS',
url: '/path/to/your.app',
desiredCapabilities: {
deviceName: 'iPhone 12',
platformVersion: '16.0',
browserName: 'safari',
},
},
},
include: {
I: './CustomSteps',
HomePage: './pages/HomePage',
},
plugins: {
retryFailedStep: {
enabled: true
},
screenshotOnFail: {
enabled: true
},
stepByStepReport: {
enabled: true
},
}
}Summary
This article discusses the risk of vendor lock-in with end-to-end test automation and proposes a solution to reduce this risk. It suggests to use an abstraction, specifically with a test automation framework like CodeceptJS, to mitigate the risk. CodeceptJS is a driver agnostic end-to-end testing frameowrk that provides the same APIs across different drivers, simplifying migration from one framework to another. Even though we acknowledge that depending on the helpers and plugins used, migration might involve more than just a configuration change, a different approach to a problem will pay your team in the long run.
