From Cypress to Playwright

We have enjoyed using Cypress over the last few years. When we made the switch from Protractor, this new technology felt like a revolution. However, over the past few months, we have found ourselves migrating over to Playwright. Some of the issues we have found with cypress are as follows

• Cypress uses way too much CPU and memory. This problem seems to keep getting worse with each new release. We use GitlabCI for running end-to-end tests so we do not have access to powerful containers with unlimited resources, this is one of our major problems.
• Cypress does not play well with grid tools such as Moon. For Moon to run cypress tests, the node_modules folder needs to be uploaded to Moon. We are using an NX mono repo so this is problematic for us as we would have to upload a very large amount of files simply to run tests for one of our projects.
• This may or may not be an issue for other companies however the company I work for has implemented some very strict rules about downloading artifacts for external websites. This is a particular pain point because when we install cypress with NPM. We are forced to download a binary from the cypress website. Over the last year, we have run into certificate issues, and now we are at the point where we are no longer allowed to download this binary for security reasons. There are solutions to get around this, however, these are mostly just hacky workarounds.
• While Cypress does have multi-browser support there are some caveats. This one again you may or may not experience in your workplace. If your organization has strict rules around what docker images can be executed in GitlabCI then multi-browser support becomes an impossibility as Cypress needs a separate image per browser. At the moment we are in a situation where we can only use the default electron browser. I also understand this may be a contributing factor to our high CPU and memory usage.
• The inbuilt test retries have never worked out well for us. In monitoring our test runs it became clear that about 80% of the time in our GitLab environment if a test failed it would fail every test retry, no matter how high we put the retry count. An external plugin called cypress-repeat faired better in this regard. I guess that the test environment is not completely restarted per test retry.
• If you find yourself writing hundreds of e2e tests your eventually going to run into very long execution times. At this point, you will look into trying to run multiple tests in parallel. Soon after you will realize that this is a paid-for feature called Cypress dashboard. This dashboard is also hosted off-site, so if your organization has strict rules about what data can be sent off-site then this dashboard will likely be rejected. There are tools such as sorry-cypress which may get you around this, however, this is another tool that will need to be installed onsite monitored, and administered.
• Local development has gotten worse over time, the application that pops up when you run cypress open seemed like a novel idea a few years ago. However now we are finding it just another resource hog as again CPU and memory usage is very high, on top of this we need to leave the context of our IDE. While it’s not a huge deal it can be painful constantly switching from your IDE to the cypress application.

How does Playwright help with these issues?

Over the past few months, we have migrated most of our codebase over to playwright. Here are some of the outcomes when compared to the list of issues above.

• Playwright uses a lot less CPU and memory, this became clear almost immediately. We also noticed that the test execution times are lightning-fast. Checkly wrote a more in depth article on this if you want to check it out. I would say that in almost 100% of the tests that we have migrated over, the test execution speed has seen significant improvements.
• While we have not tried it, Playwright does appear to work better with grid tools such as Moon. We do not need to upload our entire node_modules folder as Playwright can talk directly to the grid. More information can be found here.
• Playwright also needs to download some binary files from the internet. However it downloads them from a trusted source, so we no longer need to worry about certificate issues.
• Multi-browser support works out of the box and can be run in the same container. When we download Playwright with npm, it will automatically download chromium and firefox. This is quite nice as we no longer need to rely on electron to run our e2e tests and we can use real browsers instead.
• While it is too early to tell, test retries appear to be working correctly, I don’t have anything further to add to this for now.
• Parallel execution of tests comes by default and is free, depending on how many tests you have this could be a huge win.
• Local development feels fresh. The feedback I have received from colleagues seems to reinforce this point. I find myself able to write tests quicker as I no longer need to keep switching from my IDE to the cypress application. Unfortunately, you do need to install a plugin for your IDE to get optimal use local of development in playwright. This plugin seems to be available for all major IDEs. One thing I do feel is missing however which was available in Cypress is the ability to watch for changes. Sadly we now need to manually restart the tests whenever we make a code change.

Conclusion

These items mentioned above may not affect everyone. These are from our own experience, your mileage may vary, however, we could not ignore some of the benefits that Playwright could provide us, and we are already reaping the rewards.

If we notice in the future that Cypress has improved some of these issues we would happily look into this tool again, however, at the time of this writing we are moving forward with Playwright.