End-to-end testing in production
Good morning y’all!
Recently I took part in a project at Jusbrasil to cover one of our product subscription flows with automated end-to-end tests, and now is time to share the what, why, who, and how of it.
Here at Jusbrasil¹ we work to close the justice gap by connecting people to judicial information through lawyers. One of the products we offer is the PRO, which allows people to get well formatted quoting references of published jurisprudences, copy and keep track of fragments of these legal decisions, plus have access to legal templates for many cases, all of them well formatted, and easy to reach. To do so we work with hundreds of projects, most of them hosting microservices, some hosting monoliths, and other hosting monorepos, a few of which with hundreds of subprojects like our React component library.
These projects have to deal with a growing volume of data from tens of thousands of new legal log pages released everyday by dozens of public legal entities, millions of accesses, communication between lawyers and prospective clients, new articles written by the legal community, and so on. Meaning that we have tables measured by the dozens of terabytes in production.
Keeping it all together means we grew to have automation ingrained in our culture to make sure everything is tested and deployed automatically. We have to take special care with the consequences of changing one microservice affecting end user experience in another project. But having isolated environments to run tests on changes is not particularly easy, nor is it clear if such environments would actually portray a good enough snapshot of production to work as a good proxy to testing the chain of changes when we introduce new code.
Long story short, we don’t have isolated environments, and we want to make sure changes won’t produce unintended alterations in some very specific steps our users take in particularly sensitive flows — like showing a signin modal when an user tries to download a legal template. If we mistakingly pop a second modal on top of it by mistake, or if the subscribe modal is shown instead of the signin, we want to be alerted.
Taste both the ingredients and the baked cake
This is where automated end-to-end testing in production comes into play. It’s not supposed to replace other automated testing on the scope of each individual project — unit testing, static analysis, integration testing etc. — nor to replace existing monitoring. The idea is to use the features we developed as similar as possible to a real user, across several deployed projects.
To be clear, it’s not a recent goal, we took some steps and ran a project to automate this earlier, but paraphrasing the summary of this past project as was told to me: We grew too fond of the testing tool we built ourselves, grew it to a point where it was too complex, and were not able to chase intermitent broken tests — which is another cultural aspect we have, no broken windows.
So we scaled back, regrouped, and took another path. During some of our tech fridays — yet another culture practice we have, I promise this is the last one I mention in this post, but if you want you can learn more about them — we experimented with Cypress and liked it. The way the tests are written felt more expressive than specs using Selenium, and the inherent wait timeouts built-in on all commands helped a lot to reduce test flakyness. On top of all that, easily running and recording the tests suites locally and showing the results with video on the Cypress Dashboard SaaS hooked us. I mean, it’s always cool to watch the browser come to life and use your app super quickly.
Of course, there are some issues. Cypress is less flaky, but not flaky free. In particular it doesn’t make a stellar job of cleaning caches. We dodged this one adding retries on the tests. Also, as it runs directly in the browser, stubbing requests is quite limited: it only stubs XHR requests properly, which affected how we could isolate some of the tests consequences, like analytics and event tracking collection. Finally, the Cypress Dashboard only collects and display test results, it doesn’t provide an execution environment.
On our implementation we initially ran the tests on a periodic workflow using Github Actions. As we wished to expand the project to cover more than one product — i.e., being used by more than one business unit on our organization — we moved execution to Kubernetes cronjobs. It allowed us to run each test on separate pods, in parallel, and if the need arises we can budget resources individually. This is roughly when I joined the project, by the way. One positive side-effect of this change was that we could make requests to our internal APIs to erase or reset some of the changes made by the tests — for example, one of the flows implemented consume a user credit, and before running the suite we request our internal API to reset this credit. Not everything could be undone by the tests, so in the end we had to ignore some of those side-effects in production code — in particular our visit event tracking was made server-side, so we ended up adding code to ignore the tracking when it detected a test user.
So this is the point where we are at now. We have suites running for products from three separate teams every four hours, recording executions on the Dashboard, and we have already caught a change in production which affected one of our sales flow — an A/B test was sending users to an unexpected checkout page. And now we're working on expanding the end-to-end testing itch to all teams.
Many thanks to the totally excellent team who started it all and helped A LOT along the way: Gabriel Genê, Luis Fernando Grappi, Gustavo Barreto Maia, Márcio Vicente, Felipe Matos Santana, and Guilherme Garnier.
Did you like the idea of engineering ways to close the justice gap? Fancy working asynchronously from one of our many hubs, or remotely from anywhere? Have you been to Bahia, Donald? Guess what, we're hiring! Come work with the incredible aces I listed above and many others to help millions of people reach legal information.
[1] “Aqui no Jusbrasil” is a regular catchphrase here at Jusbrasil. 😉
