Speed up your Cucumber.js acceptance tests

At PayPal, long-running cucumber.js scenarios prevented our team from adopting the Continuous Integration/Deployments (CI/CD) pipeline model. Our release took a whole day. There weren’t any module available to run cucumber.js scenarios in parallel until published the cucumber-parallel module which optimized the execution time by 80%.

After we moved from Java to NodeJs at PayPal, we did not have full capabilities of test automation as in Java. We have been working with Cucumber at PayPal for five years, and we lost the Maven Surefire capability of running Cucumber tests in Parallel when we moved to Node. There were some reasons why it was not fully capable. One of the reasons was an early phase of CucumberJs. It was still evolving in 2015, and there weren’t any module that runs CucumberJs scenarios in parallel.

Our regression suite had approximately 100 UI facing Cucumber scenarios. It took three hours to complete the execution in serial. We could bring down the execution time to approximately ten minutes, the longest running scenario, if we can run them in parallel.

At PayPal, we were moving towards Continous Integration/Deployments (CI/CD). Long-running acceptance tests prevented our team from adopting the CI/CD pipeline model. Our release took a whole day. To decrease the execution time, I wrote the cucumber-parallel module to run CucumberJs Scenarios in parallel.

Before I started writing the module, I had to become familiar with the internals of Cucumber.js to understand how it filters and runs the scenarios. I found an option to run a scenario through feature:lineNumber which can be passed to each concurrent threads. After a few days of work, I was able to run cucumber scenarios in parallel.

But wait.. the results were not as expected. Each thread produces it’s own JSON result because it runs each independent scenario; therefore 50 scenarios would produce 50 single result files. I wanted consolidated results at the end of the execution. After some thought, one solution was to throw JSON results from each thread to the main console and read the console at the end to generate one JSON result — which finally produced one single JSON result file!

Finally, I had a Cucumber-Parallel module which was first published in Feb 2016. It reduced our execution time from ~3 hours to only ~10 minutes and unblocked our CI/CD initiative.

More details on how to use the cucumber-parallel module are available here.

Ref: https://www.npmjs.com/package/cucumber-parallel