What to Look for When Selecting a Cross-Browser Testing Framework
With a plethora of testing solutions, whether commercial or open-source, some are Selenium WebDriver based, some are proprietary, organizations are finding it hard to match the right tools for their cross-browser testing needs.
Bryan Osterkamp, Lead Technical Architect, Dev and Ops Infrastructure Architecture Division at USAA, made an apt analogy for this selection process, referring to the infrastructure in question as a freeway.
“There are a few things about a freeway. For instance, once a freeway is built, it allows its motorists to go as fast as they want up to the speed limit allowed by the authorities. Now, people aren’t forced to go on the freeway. If they decide to, they can take side streets. It may take longer, but there is nothing stopping them from taking that path. Let’s also say this freeway has something a lot of us in Texas (and especially San Antonio) don’t like, and that is a toll”
As mentioned above, organizations have many options, but returning to software terminology, it always comes down to a mix of technical- and organizational-fit requirements.
As described in my recent eBook, these 2 segments of requirement-fit consist of the following (each of them is detailed further in the paper itself):
· Project complexity and organizational future roadmap
· Existing resources, skillset and budget allocated for the project
· Existing tool stack within the DevOps teams
· Test type and test practices adopted by the organization (BDD, TDD, COE, Agile, etc.)
· What kid of organization executive dashboards are required as part of the project/s?
· Tool community size, support, documentation, contribution sources, latest commit and more
· SDLC process fit of the test framework — can the tool support the organizational practices?
· Level of feedback loops and reporting provided by the framework?
· Cloud, automation-at-scale capabilities
· Level of automation coverage capabilities (can the tool support HAR file, environment mimicking, accessibility testing, visual navigation testing and more?)
· Automation robustness built-in test framework capabilities (sync-based testing tool vs. a-sync, page object model (POM) easy setup, environment setup wizard and more)
The recommended practice as identified in the eBook, is to not only choose the right tools based on a mix of technical and organizational fit, but also acknowledge that in some cases a mix of test frameworks would make the most sense from a test coverage and test requirements perspective (e.g. leveraging Protractor with JSDOM or PhantomJS can address both unit headless testing as well as E2E acceptance or functional testing coverage requirements).
Choose your path to success or build your “infrastructure” as a freeway and allow your testing practitioners the freedom to combine both toll-based freeways as well as side roads that can overcome freeway pitfalls. The end goal is to release as fast as possible a working, high-quality product that can be validated throughout the release pipeline by various teams with varying tool stack selections and skill-sets.