Faster Cucumber startup: Keep the PhantomJS browser open between tests

At Sharetribe, we take integration testing seriously. We provide a platform for multiple marketplaces, each with its own configuration and set of customizations. At the same time, we are rapidly developing our platform and deploying new features every week. When we release a new feature, we hardly ever do any manual regression testing. We have implemented a comprehensive test set which we trust to catch regressions. As a small startup, we simply don’t have time to waste testing something manually that can be automated.

Implementing integration tests with Cucumber is not always fun, especially if your tests are running slow.

When talking about test speed, we’re talking about two factors that make up the overall speed:

1. The test suite startup time
2. The test step execution time

When you are implementing a new Cucumber scenario, you probably end up running it multiple times before the scenario passes. You write the scenario, run it and see red. Then you fix the first step that failed, rerun it again and see red for the next step. Good job, you got one step further! You keep iterating until all steps pass.

At this point, you want your test suite to start fast! Every second counts.

At Sharetribe, we are running our Cucumber features with the Capybara + Selenium Webdriver + PhantomJS stack. We have already spent a lot of effort to make the test suite start up quickly: we use Zeus to overcome the slowness of loading the Rails environment. We set up our test data during Zeus startup so that we don’t need to initialize the test database state each time the test suite is started. In fact, our test suite already starts pretty quickly. For a single simple scenario, running the test takes no more than 5.5s:

zeus cucumber features/admin/categories/admin_adds_category.feature:9 
1 scenario (1 passed) 
7 steps (7 passed) 
0m5.688s

However, there are still seconds to cut. We noticed that Selenium Webdriver starts the PhantomJS browser each time the test suite is started. We thought that it would be cool to start PhantomJS once and keep it running between the test runs, thus saving some seconds from test startup time.

Selenium Webdriver monkey-patch

In the Selenium Webdriver gem, the PhantomJS::Service.start method is responsible for starting PhantomJS as a child process. If you take a look at the method, you see the two lines there:

@process = create_process(args) 
@process.start

This is where the (unwanted) magic happens. Instead of really creating a new PhantomJS process, we want to override the create_process method and make it a no-op. Here’s how:

class Selenium::WebDriver::PhantomJS::Service 
  def create_process(args) 
    puts "Using monkey-patched PhantomJS Selenium Webdriver" 

    Struct.new("ChildProcessFake") do 
      def start() end
      def exited?() true end 
    end.new 
  end 
end

Let’s take a look what we’ve done here:

The original create_process method returns an instance of ChildProcess. We create a new Struct to stub the ChildProcess and fake the two methods that the Selenium Webdriver uses, start and exited?. start is a no-op and exited? returns always true so that the Selenium Webdriver doesn’t try to exit the non-existing child process.

But that’s not enough. By default, the Selenium Webdriver tries to start the PhantomJS process to listen on port 8910. If that port is taken, it tries 8911, and so on. Since want to start PhantomJS in the background to listen on port 8910, the port is already taken and the Selenium Webdriver tries to use the next free port instead. To overcome this, we have to monkey-patch the PortProber class:

class Selenium::WebDriver::PortProber 
  def self.free?(port) 
    true 
  end 
end

We have placed those two pieces of code in lib/selenium_webdriver_phantomjs_monkey_patch.rb.

The last thing to do is to require that file in your test environment configuration file. In your config/test.rb file, right after Capybara.register_driver, you should add the following line:

require "#{Rails.root}/lib/selenium_webdriver_phantomjs_monkey_patch"

And that’s it!

Now you can start PhantomJS on the default port of 8910 and keep it open in a terminal tab:

phantomjs --webdriver=8910

Once you have PhantomJS running, you can start your Cucumber tests.

Since Sharetribe is an open source product, you can see our lib/selenium_webdriver_phantomjs_monkey_patch.rb and config/test.rb files on Github.

So, how much will this boost the test suite startup time?

To be honest, since the startup of the test suite is already pretty fast, there are not that many seconds to cut. In any case, here are the results of running a single Cucumber scenario:

zeus cucumber features/admin/categories/admin_adds_category.feature:9 
1 scenario (1 passed) 
7 steps (7 passed) 
# Reopen PhantomJS per each run 
0m5.688s 
0m5.990s 
0m5.503s 
# Keep PhantomJS open on the background 
0m4.472s 
0m4.480s 
0m4.560s

As you can see from the results, by keeping PhantomJS open you can save a second or so. It’s not that much, but in a simple test scenario like this, it’s still more than 20%.

CI environment

This technique can be also used in your CI environment. At Sharetribe, we use Travis CI, which has PhantomJS pre-installed. Before running the test set, start PhantomJS in Travis as follows:

phantomjs --webdriver=8910 & 
bundle exec cucumber

Conclusion

Let’s admit it: this is a hack. Monkey-patching a gem is always a hack. But it’s a hack that has been working really well for us during the past couple of months.

Due to us having the browser open between test executions, I had my doubts that we would eventually run into trouble with “dirty” browser state (e.g. not cleared session data). However, we have not encountered any such issues.

If you think this tip could help you, give it a try and drop us a comment if you found it useful!

- Mikko

Originally published on the Sharetribe blog on April 7, 2014.