My experience moving from Selenium to Cypress

There are many comparisons between Cypress and Selenium out there, but each has a slightly different angle on things, and so I’m here to give you my own experience and insights.

Some background

As a consultant for test automation, I’ve done many projects using Selenium, mainly in C# and Java, and also some in Python. I also wrote and taught about writing unit tests in JavaScript, but only did a single POC (Proof of Concept) using JavaScript and Selenium. I also did a verity of projects in other languages and tools besides Selenium.

In most of the test automation projects I do, I follow a pretty strict method for writing the tests and infrastructure (as described in this blog post and as detailed in my book, Complete Guide to Test Automation). There are few advantages that I gain from my method:

1. Very readable code (and tests)
2. Very modular and maintainable code
3. It’s pretty easy to diagnose the root cause of failures

I also make heavy use of object-oriented programming, and tend to rely on strong-typing for preventing mistakes. Here’s an example of a typical test I would write in C#:

An example of the way I typically write tests in C# and Selenium

The client I did the project for asked to work with JavaScript, as his developers are writing in JavaScript, and he wants the developers to take ownership on the test automation at some point. Because I like to use strong-typing, and also because the developers were also willing to start using TypeScript, we agreed that I’ll use TypeScript too instead of pure JavaScript.

The Selenium attempt

At first, the client wanted the tests to be able to run on multiple browsers (mainly Chrome and IE…) and platforms, and to leverage Selenium Grid for that, so Selenium (as opposed to Cypress) was the only viable choice.

I started to write the tests and the framework as I’m used to from other languages, which is to write the test function first by calling other functions (which only throw “not implemented” exception at first) that describe high-level operations of the test in the application, and then implement each of these functions to do whatever it supposed to. Note that typically, functional automated tests are sequential in nature (within a single tests; load testing is a completely different story…). This means that there’s no point in starting any operation before the previous one completed. Not only there’s no point, but it can quickly make your tests flaky, as the order in which the operation complete may affect the outcome in ways you cannot anticipate.

However, in JavaScript (and TypeScript), all operations that may take more than a couple of milliseconds are asynchronous by nature. This means that almost all Selenium functions are also asynchronous because of that, and therefore use Promises to handle this asynchronous nature. But even though we use Selenium only in our lower-level functions, this asynchronous nature must seep into the higher-level functions as well, and into our test code. Obviously, in ES6, we can use the await keyword instead of nesting then(…) statements, so on the surface of it, it’s only a slight syntactic change from the way I write tests in other languages. Here’s how a test similar to the one in the C# example above would look like in JavaScript (using Jasmine):

An example of how I would write a test in JavaScript and Selenium

This looks a little bit more cluttered with all of these awaits, but that’s not that terrible.

However, there are two more subtle issues with these awaits:

1. If you forget to put one, it will be pretty hard to understand what you have missed
2. Even if you don’t, but the test failed inside some inner method, the stack trace you get is worthless

Even though these await issues bugged me to some extent, it wasn’t a good reason to switch to another technology yet. But few weeks after I started to implement the first tests and framework code, I had another discussion with the Dev manager and another team leader, and we concluded that the support of IE is not really relevant, and that cross-browser testing in general wasn’t a top priority. When these restrictions removed, I convinced the client to give a try to Cypress instead.

At this point, all the knowledge I had about Cypress was from reading and talking to people, but had no hands-on experience with it. I knew that (at least currently) it only runs on Chrome; runs inside the browser, and therefore it’s faster; I knew that you can easily send Rest API requests from it, and to mock or spy on XHR requests that the page sends, and I saw few code examples that look something like this:

Typical Cypress test

See? No awaits! I assumed that this was possible in Cypress because it runs inside the browser, as opposed to Selenium which communicates with the browser from outside.

My First Cypress Experience and What I Discovered

So I went through Cypress’s Getting Started Guide and some of the more specific guides in order to get a grasp on how to use cypress. It wasn’t long until I realized that this seemingly synchronous code is just an illusion, and that Cypress commands aren’t really synchronous, at least not when they’re mixed with any regular, non-cypress, JavaScript code. The trick that Cypress uses is that all of its commands aren’t really executed when their respective functions are being called, but rather these functions only queue these commands to be executed in sequence right after the test function itself completes. There is much more to it, but this is the basic idea. I intent to write another blog post soon describing how cypress works in more detail.

This interesting discovery about Cypress really caught me unprepared. I thought that it works one way, while in fact it works in a completely different paradigm which was new to me.

This paradigm comes with many subtle implications and ways that are specific to Cypress that are needed in order to achieve things that are done using straight-forward JavaScript with Selenium (or in any other languages and automation technology). I found Cypress itself, as well as its documentation (which is by the way, very detailed and comprehensive), to be very opinionated. In most cases, these opinions matched my own, but not always, and in these cases I felt somewhat trapped.

Other limitations of Cypress

In addition, I realized that there are few other limitations:

• Due to the same-origin policy, the test is bound to the same website, and cannot navigate to another site, even through a link from the website itselt
• It has a limited support for iframes
• You cannot open more than one browser instance or tab in the test
• Because the browser is a sandbox, you cannot do things like invoking external processes, accessing a DB directly, etc. Cypress lets you do that by writing tasks plugins that run in the NodeJS side of Cypress (Cypress runs a Node process that orchestrates the tests that run inside the browser). While it provides a way to do that, it’s more cumbersome than doing it directly from Node, as you would when you use Selenium.
• While both Selenium and Cypress are open-source, Cypress is developed by a dedicated commercial company that sells a complementary product, which is especially relevant for running the tests in CI and in parallel, with more than 3 users. Though I must admit that their pricing is very fair.

Advantages of Cypress

Clearly, Cypress also has its advantages. Here are some examples:

• Automatic retry on finding elements and mostly all actions that do not change the state (like click for example)
• Automatic scrolling to ensure an element is in view before clicking it
• Can send, spy and intercept XHR requests. This makes it suitable both for Unit, Integration and System tests, all in one place.
• Supports spies and stubs (unit-testing style)
• Excellent documentation. Both as a reference and for best practices

The Core Difference Between Selenium and Cypress

With all of these pros and cons, and while we can compare Selenium and Cypress from many different angles, there’s one key difference between the two, which most other differences derive from: Cypress is a testing framework, while Selenium is a software component. This means, that Selenium is just a pretty low-level building block (though very important and good one!), and you must integrate it with other tools, like a testing framework (e.g. Jasmine), logging, reporting, and a lot of other stuff to create your entire solution. For many of these integrations you need to write the code yourself and then need to maintain it. Cypress on the other hand, attempts to give you everything you need to start working and frees you to focus on the tests themselves.

One of the nicest things about Cypress, is its interactive Test Runner. The Test Runner not only automatically logs all of the actions for you, but for each action it saves a snapshot of the DOM. After the test completes, it allows you to go back and forth “in time” to see not only a static screenshot of the page at the time of the action, but the real interactive page as it was at that point in time. Then you can open the Chrome’s Dev Tools and investigate the DOM and the internal state of the application using the DevTool’s Console.

Cypress Test Runner

So even though in most cases of failure there’s no stack trace, or at least not a very useful one, there’s a different, very convenient and useful tool for debugging and diagnosing failures.

The decision to stay with Cypress

Weighing all of these factors, I decided to stay with Cypress, as all in all, after getting used to its way of doing things, I noticed that in total I’m more productive using it. In addition because I’ll soon need to hand over my code to the developers of my client, I believe it will be easier for them to use and maintain a code base that has much less “plumbing code” in it, as opposed to an entire framework that I would have to write pretty much from scratch, have I chosen to use Selenium.

I’m still learning Cypress and discovering new things about it every day. Some of these things delight me while others still annoy me. I guess that most things in life are a compromise… just focus on what’s important to you.