Browser testing with Chromedriver / Drupal 8.5.0

First, let me say thanks to mpdonadio for his tweet which helped me out of a composer ditch I couldn’t possibly get out of myself and to mixologic who provided some astonishing insight into this craziness.

So I am testing the login flow for https://community.smartsheet.com . It might seem too specific but trust me, there’ll be a lot of info here which will be useful for a lot of testing scenarios. All of this is hard. I tried in the past with PhantomJS and just gave up. But with Headless Chrome now available and core integrating it, I ran out of excuses and this test is really a must, it’s our most brittle feature and even manual testing of it is time consuming and complicated.

Here’s one possible login flow involving a new user:

1. Click “Connect with Smartsheet” on the Community site.
2. Supply an email.
3. Supply a password.
4. Click Allow.
5. Provide a user name and accept Terms And Conditions.

Note after 4. the Smartsheet app will redirect you to the site https://community.smartsheet.com and that can’t be changed. Of course, we want to test on localhost and this means we need to tell Chrome to kindly pretend 127.0.0.1 is the IP for community.smartsheet.com. And this is where the fun starts: while there is a command line argument which allows us to do this: chrome --host-rules='MAP community.smartsheet.com 127.0.0.1' Headless Chrome does not support this yet, the feature request is here. The official documentation for command line switches no longer contains host-rules, it used to and it certainly works still. At the end of the day, you need to edit /etc/hosts, there’s no way to avoid it. But then your browsers can’t access the real world site so actually you need to start your real world Chrome with a host rule mapping the community site to its real IP! I love website development, it’s about as straightforward as using your right foot to scratch your left ear when it itches.

After you’ve done this visiting the community site with Chrome will throw a lovely security warning claiming the certificate is invalid. You see, this is because after the host rule Chrome wants the certificate for the IP address and not the host. This certainly matters because if the feature request I linked above goes through and Headless Chrome will be started with a host rules MAP against a local Apache using a self signed certificate then the certificate needs to be valid for 127.0.0.1 as well. I found this guide to work well for a self signed certificate, just add IP.1 = 127.0.0.1 to the end. And yes, it has become a real lot of work to get a self signed cert working without a warning.

So now we have Chrome set up so we can visit community manually (if we accept the unsafe warning) to compare the test and also automatically by Headless Chrome which will visit localhost which has a self signed certificate.

Now comes chromedriver part. As the name suggests, this is something which allows programatically manipulating the Chrome browser. This is quite necessary because Headless Chrome, as the name suggests, doesn’t have a UI so something needs to do it. This is logical but what follows is plain nuts: despite the driver class Behat / Mink provides is called the Selenium driver and we are going to use the Selenium webdriver protocol, and even core/tests/README.md instructs to install selenium, we do not need it! Chromedriver implements the webdriver protocol by itself and so it is enough. The test I wrote works, a random test I picked from core (JavascriptGetDrupalSettingsTest) works and according to mixologic the testbots do not have Selenium installed either. The less Java, the better, as far as I am concerned, and in general, the less moving parts, the better. Heaven knows we have enough moving, broken, un(der)documented parts. So we can just install chromedriver … except I couldn’t get it working on Debian Jessie without also installing libgconf-2–4.

Next up is a fun little core bug: if you add <env name="MINK_DRIVER_CLASS" value="Drupal\FunctionalJavascriptTests\DrupalSelenium2Driver" /> to phpunit.xml then you need to use MINK_DRIVER_ARGS in there, MINK_DRIVER_ARGS_WEBDRIVER only works if the test class sets protected $minkDefaultDriverClass = DrupalSelenium2Driver::class;. Perhaps it’s better not to use MINK_DRIVER_CLASS here because it’s likely not all core tests pass yet. Anyways, if you go with “naked” Chromedriver as I suggested above then you do need to override the arguments because your port will likely be 9515 and not 4444. MINK_DRIVER_ARGS='["chrome", {"chromeOptions":{"args":["--disable-gpu", "--headless”]}}, "http://localhost:9515"]' is just about the minimum you’ll need. If you read up on these things and find “capabilities” you need then add them on the same level as chromeOptions. Note the Behat Selenium2Driver class provides an alternative way to handle command line switches and capabilities but as far as I can tell, it’s broken beyond any repair and I was unable to find documentation about it so just ignore it, this data structure works.

So much work and we haven’t yet written a single line of test yet, well, maybe one, where we set the driver class. My setUp sets the default theme as described here. And now, finally we get to the test!

The fundamental secret about using JavascriptTestBase and DrupalSelenium2Driver is it seems to be async, chromedriver apparently returns before navigation finishes. This is conveniently not documented anywhere and a lot of helper methods seems like broken because they don’t wait. And there doesn’t seem to be a better way than just wait, the webdriver protocol is operating with simple HTTP requests and so it doesn’t call back when navigation finished. So for example after using clickLink you need to use $this->assertSession()->waitForElement or a similar wait method from the JSWebAssert class. Of course, I might be completely wrong here but I have been struggling with my test for days and everything was failing intermittently and the moment I switched to the wait methods it never failed again. If this presumption is correct then quite an amount of documentation needs to be amended:

1. The JavascriptTestBase and DrupalSelenium2Driver classes should be crosslinked with JSWebAssert.
2. https://www.drupal.org/node/2846936 to clarify that with webdriver you need to use these every time you click / press / submit not just when JavaScript does something.
3. Upstream doxygen patches against Behat / Mink probably about a million places. Methods onDriverInterface at least.
4. https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#sessionsessionidurl
5. http://mink.behat.org/en/latest/guides/interacting-with-pages.html

That’s quite a lot of places which meticulously do not mention anything about this. After writing this post, I found two things which makes me pretty confident I am right: the relevant NPM package has this example code: await driver.wait(until.titleIs('webdriver - Google Search'), 1000); and also the chromedriver README.txt has this to say:

The handler executes the appropriate command function, which completes asynchronously.

Summary:

1. Getting set up is the hardest. Self signed certs became astonishing complicated, Chromium documentation went pear shaped, core documentation is even more behind the code than usual.
2. Chromedriver navigation is async and no PHP documentation emphasizes this. If you are used to testing in Drupal and try write tests as you used to, you will have a hard time. There are new asserts on theJSWebAssert object returned by $this->assertSession() to help.

Ps. Do not use $this->getSession()->visit() to browse the test site, as it currently stands this will browse the parent site, stay with usingdrupalGet.