How to release changes to the Payments Checkout twice a week without breaking production

By Iraklii Tatoshvili, QA Engineer at Payments by Wix

Image by Scott Graham on Unsplash

When a business goes online planning to sell goods and provide services — arguably the most important thing is to find the best platform to build a stunning website. One with a powerful payment processing solution too, of course, allowing you to provide a proper payment processing experience for your future customers.

Any platform that deals with eCommerce and payments is very sensitive to changes. And Payments by Wix that handles everything that has to do with processing online payments, is no different, really. You can read more about the team behind the product here.

Recently we (the Payments by Wix team) went through some quite turbulent changes and in this article I’ll share with you the problems we faced, the issues we solved and the lessons we learned.

Structural changes

There is no such thing as perfection when you’re putting together a new team from scratch or working on a new infrastructure to create the absolutely best way of delivering features. This applies to every process in your team — overall process, development, QA, etc.

Our team went through a number of structural and team changes before arriving where we are today — a dedicated Checkout team under the umbrella of the Payments core team. Separating into smaller, more focused teams (Checkout, Processing, Onboarding for payment providers, etc.) helped us stay organized, work on features, and deliver faster.

Some of the challenges we faced as a team on the path that led us to where we are now were the lack of automation with the rollout process, mess in Jira, tech debt, etc. And in terms of QA — let me be frank with you, of course I made mistakes and had failures. But c’mon, who didn’t? Ever had that creepy feeling you missed some important checks or scenarios? I sure did. And I am pretty sure that I will make mistakes in the future as well. What’s more important is to do your best to minimize those mistakes and leave as little room for future ones as possible.

How we broke production

Wix is a very powerful tool to build not just websites, but also end-to-end applications as solutions for your business — you can use Wix as an Online Store, as an Events Manager, a Booking Service, etc. — or combine everything under one roof. And all of those solutions I just mentioned are integrated with Payments. Because what if you wanted to charge for a booking or sell tickets to events?

But back to the story at hand — going through the changes I briefly mentioned above as a team, we sure had some growing pains. Those manifested in a variety of challenges, but to make matters less confusing, let me focus on several concrete examples.

Here are the 3 specific problems we had with a checkout widget in a web browser flow and the lessons we learned solving them:

Bad API typings (:any). An application was expecting Promise<{ Object }> and we returned {Object}.

This issue caused a broken page on production after the artefact release. Why? :any — is quite bad practice for API typings, especially for public methods.

First of all, quite obvious issues like the one above should be fixed right away and not marked as TechDebt to wait when it fires an issue. While covering features with checklists (test cases or whatever works for you and Team) and automation tests, do not limit yourself with straightforward scenarios. Think of corner cases, even rare cases or some odd user behaviour — at least one test for each case can be enough.

A fix of the onKey event for web browsers running on iOS devices has caused the issue when there was no possibility to enter any data into the Credit Card form in Internet Explorer — the introduced event was not supported by a specific browser.

Even when there is a tiny possibility that commits can impact non-related features — run sanity and do cross browser testing. If you have it covered with automation, that’s even better as it’s going to be less time consuming.

Relying on window.Wix (WixSDK lib) to just be there, while it doesn’t exist in the Application page — caused a blank page on load.

If you’re running QA for an application or feature that’s integrated with other applications in any way — it is never enough to run tests just for your use case. Sometimes even short sanity can save you from broken production of the whole application.

How do we prepare artifacts/features to release?

In our team, a QA Engineer is responsible for delivering release candidates to production. In other words — the one who does GA for an RC. No matter how paradoxical it sounds, we developed a script which allows us to slow down, while actually releasing faster.

Release cycle includes progression and regression testing. I’ll only cover regression here because progression can easily be the subject for another blog post.

So our script is as follows:

1. Front-end developer (FED) releases the RC version of front end (FE) artefact (includes JS|Unit/JSDOM test runs to have build) — done by a developer.
2. Some artefacts need to run the TestBed (uploading a new version for only one server) and monitor the alerts (NewRelic, Grafana etc.) — done by a developer and a QA engineer
3. QA Automation tests are fired — done by QA
4. Analysing test run results — done by QA
5. Manual Sanity tests — done by QA
6. If the automation is red (excluding automation issues and false-positive fails) — QA works alongside the developer(s), checking the failures and applying fixes
7. Release artefact
8. Monitoring after the artefact is rolled out

In conclusion, a note about some QA activities needed to be done with every roll out of a Payments Checkout artefact (aside of progression testing).

• Even though you might have good coverage with automation tests — never forget to run manual tests as well.
• If you had previous production issues — don’t forget to cover those (checklists, test cases, automation).
• Run cross-browser and cross-platform sanity checks and once artefact is rolled out — pay attention to monitoring and alerts.
• And always keep in mind those corner cases and the rare cases.

I hope you enjoyed this brief list, which came directly from our own experience of making mistakes — and learning from them. As I mentioned before, it’s not so much about being afraid of making similar mistakes in the future — it’s much more about learning and constantly improving.