There are many comparisons between Cypress and Selenium out there, but each has a slightly different angle on things, and so I’m here to give you my own experience and insights.
In most of the test automation projects I do, I follow a pretty strict method for writing the tests and infrastructure (as described in this blog post and as detailed in my book, Complete Guide to Test Automation). There are few advantages that I gain from my method:
- Very readable code (and tests)
- Very modular and maintainable code
- It’s pretty easy to diagnose the root cause of failures
I also make heavy use of object-oriented programming, and tend to rely on strong-typing for preventing mistakes. Here’s an example of a typical test I would write in C#:
The Selenium attempt
At first, the client wanted the tests to be able to run on multiple browsers (mainly Chrome and IE…) and platforms, and to leverage Selenium Grid for that, so Selenium (as opposed to Cypress) was the only viable choice.
I started to write the tests and the framework as I’m used to from other languages, which is to write the test function first by calling other functions (which only throw “not implemented” exception at first) that describe high-level operations of the test in the application, and then implement each of these functions to do whatever it supposed to. Note that typically, functional automated tests are sequential in nature (within a single tests; load testing is a completely different story…). This means that there’s no point in starting any operation before the previous one completed. Not only there’s no point, but it can quickly make your tests flaky, as the order in which the operation complete may affect the outcome in ways you cannot anticipate.
await keyword instead of nesting
This looks a little bit more cluttered with all of these
awaits, but that’s not that terrible.
However, there are two more subtle issues with these
- If you forget to put one, it will be pretty hard to understand what you have missed
- Even if you don’t, but the test failed inside some inner method, the stack trace you get is worthless
Even though these
await issues bugged me to some extent, it wasn’t a good reason to switch to another technology yet. But few weeks after I started to implement the first tests and framework code, I had another discussion with the Dev manager and another team leader, and we concluded that the support of IE is not really relevant, and that cross-browser testing in general wasn’t a top priority. When these restrictions removed, I convinced the client to give a try to Cypress instead.
At this point, all the knowledge I had about Cypress was from reading and talking to people, but had no hands-on experience with it. I knew that (at least currently) it only runs on Chrome; runs inside the browser, and therefore it’s faster; I knew that you can easily send Rest API requests from it, and to mock or spy on XHR requests that the page sends, and I saw few code examples that look something like this:
awaits! I assumed that this was possible in Cypress because it runs inside the browser, as opposed to Selenium which communicates with the browser from outside.
My First Cypress Experience and What I Discovered
This interesting discovery about Cypress really caught me unprepared. I thought that it works one way, while in fact it works in a completely different paradigm which was new to me.
Other limitations of Cypress
In addition, I realized that there are few other limitations:
- Due to the same-origin policy, the test is bound to the same website, and cannot navigate to another site, even through a link from the website itselt
- It has a limited support for iframes
- You cannot open more than one browser instance or tab in the test
- Because the browser is a sandbox, you cannot do things like invoking external processes, accessing a DB directly, etc. Cypress lets you do that by writing tasks plugins that run in the NodeJS side of Cypress (Cypress runs a Node process that orchestrates the tests that run inside the browser). While it provides a way to do that, it’s more cumbersome than doing it directly from Node, as you would when you use Selenium.
- While both Selenium and Cypress are open-source, Cypress is developed by a dedicated commercial company that sells a complementary product, which is especially relevant for running the tests in CI and in parallel, with more than 3 users. Though I must admit that their pricing is very fair.
Advantages of Cypress
Clearly, Cypress also has its advantages. Here are some examples:
- Automatic retry on finding elements and mostly all actions that do not change the state (like click for example)
- Automatic scrolling to ensure an element is in view before clicking it
- Can send, spy and intercept XHR requests. This makes it suitable both for Unit, Integration and System tests, all in one place.
- Supports spies and stubs (unit-testing style)
- Excellent documentation. Both as a reference and for best practices
The Core Difference Between Selenium and Cypress
With all of these pros and cons, and while we can compare Selenium and Cypress from many different angles, there’s one key difference between the two, which most other differences derive from: Cypress is a testing framework, while Selenium is a software component. This means, that Selenium is just a pretty low-level building block (though very important and good one!), and you must integrate it with other tools, like a testing framework (e.g. Jasmine), logging, reporting, and a lot of other stuff to create your entire solution. For many of these integrations you need to write the code yourself and then need to maintain it. Cypress on the other hand, attempts to give you everything you need to start working and frees you to focus on the tests themselves.
One of the nicest things about Cypress, is its interactive Test Runner. The Test Runner not only automatically logs all of the actions for you, but for each action it saves a snapshot of the DOM. After the test completes, it allows you to go back and forth “in time” to see not only a static screenshot of the page at the time of the action, but the real interactive page as it was at that point in time. Then you can open the Chrome’s Dev Tools and investigate the DOM and the internal state of the application using the DevTool’s Console.
So even though in most cases of failure there’s no stack trace, or at least not a very useful one, there’s a different, very convenient and useful tool for debugging and diagnosing failures.
The decision to stay with Cypress
Weighing all of these factors, I decided to stay with Cypress, as all in all, after getting used to its way of doing things, I noticed that in total I’m more productive using it. In addition because I’ll soon need to hand over my code to the developers of my client, I believe it will be easier for them to use and maintain a code base that has much less “plumbing code” in it, as opposed to an entire framework that I would have to write pretty much from scratch, have I chosen to use Selenium.
I’m still learning Cypress and discovering new things about it every day. Some of these things delight me while others still annoy me. I guess that most things in life are a compromise… just focus on what’s important to you.