How Nexi Relay Is Implementing a True DevOps Approach
Our journey to Engineering Excellence
What is Nexi Relay?
Nexi’s portfolio consists of multiple Payment Service Providers (PSPs) operating in different markets and providing local services, such as local Payment Methods, to their customers, the local merchants. Relay was created to support that journey by being a single connector of shared services for the company. The platform enables PSPs to offer more payment methods to merchants and partners at a competitive market rate, while also allowing them to reduce operational expenses and workload and free up capacity to work on differentiating local winning propositions.
Because of the above, I quickly realized that Conway’s Law was going to be at full effect for us: to build such flexibility in our platform, our teams also needed to be as independent and flexible as possible. Nexi Relay started as an integration of 5 services built by 5 independent teams using different programming languages, tech stacks (including cloud providers), release and operational approaches. How could we make sure the resulting integrated platform remained scalable, stable, reliable, and performant, while keeping the team’s freedom, ownership, and accountability?
Enter DevOps. True DevOps
For us, DevOps is a methodology in which development teams take end-to-end ownership of the lifecycle of their applications. Based on that, having an explicit “DevOps Engineer” role or a “DevOps Team” that takes full ownership of pipelines, infrastructure, etc. goes against the DevOps methodology. Every developer should be a DevOps Engineer, i.e., someone able to code, build, test, release and maintain the applications they own.
DevOps teams are independent, self-sufficient and autonomous. Enabling a true DevOps approach, together with well-defined Product roadmaps to remove team dependencies at Product level, can transform any Product & Engineering organization into a well-greased machine. New, high-quality features and products are deployed to Production as soon as they’re ready, while business teams can decide when, to whom and how to release them. That sounds like exactly what I needed and wanted for Nexi Relay, but what’s the best way to implement it?
How to implement a true DevOps approach
Unfortunately, there’s no magic shortcut that will transform your teams into DevOps teams. Do not trust anyone or anything that tells you otherwise. It will take effort, investment at all levels of the organization, patience and perseverance. In our own experience most attempts to implement DevOps methodologies fail due to one or more of the above running out.
Implementing DevOps will be tough and disruptive, yes, but afterwards the speed and quality of your deliveries will increase by orders of magnitude, and your product and development teams will be much happier and more engaged.
Luckily, a growing number of companies are finally getting DevOps right. Just check 2021’s Accelerate report, or 2022’s if you feel more adventurous (as with so many companies having financial troubles it’s much harder to focus on doing the things right).
At Nexi Relay we’re following an implementation approach based on the experience of our most senior engineers and managers, who have seen DevOps implementations fail and succeed several times over. It would take a much longer article to explain each step in detail (perhaps I’ll do it, if there is interest?), but I hope the following 5-step overview can give you some ideas on how to get started.
- Know where you are, always
DevOps is all about flow: quality at speed. That’s why you should always start by providing a baseline of where you are in your quality and engineering efficiency. Your journey will also be tough and long, so it’s crucial that you provide the means to track improvements and show that the efforts are worth the benefits.
Some examples of metrics that can be very useful at this stage are: Production Incidents per month, Releases per month, Bugfix/Non-Bugfix ratio (per sprint and team), # Hotfixes per month (no fast-tracks, only actual fixes), Bug/Change Ratio (bugs found during ST/SIT testing vs changes in release), Change Failure Rate (bugs found in Production vs changes in release), # Tests per team and level (UT, CT, IT, ST, AT, i.e. the team’s Test Pyramid actual shape), % Automated Tests in each level, Manual regression time (per week), and Created vs Fixed bugs (1st / 2nd line support).
At a later stage, when your teams are more mature, you can use DORA metrics as a simpler way to track engineering efficiency. There are some tools out there that can get you these metrics automatically from your existing tools, so that’s a huge plus.
2. Build an Enabling function to mentor your dev teams
If you want your dev teams to take full end-to-end ownership and accountability of their applications, they will probably need some upskilling in test automation, infrastructure and observability. The key, however, is to limit their cognitive load to a minimum. How do we approach that?
- Provide abstractions to the most common tasks in the form of an Internal Developer Platform (or IDP, see next step), and
- Get specialist roles to mentor developers in testing, infrastructure and observability to the point they can become efficient in these tasks by using and extending the IDP and other tools and frameworks at their disposal.
At Nexi Relay we created what we called an “Engineering Productivity” team (EngProd), formed by Infrastructure Engineers, Site Reliability Engineers, Release Engineers and Developers in Test (SDETs). Some of these engineers work in our Platform team (“Core Services”) creating our IDP, while the rest work as “EngProd Enablers”, federated to dev teams based on their needs.
EngProd Enablers work building the tools, libraries and frameworks the teams need, identifying which of their needs might be common to other teams and therefore should be provided by the IDP, training developers to use the existing tools effectively and self-sufficiently and, lastly but most importantly, sharing best practices and standards.
Once a team is self-sufficient, which we measure using the metrics from step 1, the EngProd Enabler will move to another, less mature team, where they will start over. This is what I call the “EngProd Enablement model”.
3. Build an Internal Developer Platform to empower your dev teams
Our Core Services team is what’s recently being known as a “Platform Engineering team”, which is a team in charge of creating and maintaining an Internal Developer Platform, i.e., a platform built with abstractions and capabilities that developers can use to make their job easier and more end-to-end. One important aspect of this team is that it builds this IDP as a Product, i.e., the team follows the same approaches than the other development teams: has a TPM who owns a backlog, an architect, a manager and a Scrum Master to drive iterative development and handles the releases and maintenance of the IDP. For a longer explanation please check this article.
Our IDP is taken care of things like building a PCI-DSS compliant cloud infrastructure, creating Pulumi libraries to facilitate its deployment, create our common authentication and authorization services, create templates for new services, CI/CD pipelines, logging/tracing/monitoring, etc.
4. Use Continuous Deployment as the driver for DevOps
Continuous Deployment (CD) is a mechanism to automatically deploy every new code merged to the main branch to the Production environment. Because CD is automatic and final, it encourages a quality-driven mindset that must be present in every step of the SDLC, and not as a dedicated “testing” stage. It also encourages teams to work on small, low-risk increments, separating “code deployments” from “feature releases”. With CD, a developer creating a Merge Request must know that their change:
- Works as expected, and
- Does not break any existing functionality
To do so, they use effective automatic feedback loops in the form of Continuous Integration pipelines during both development and upon Merge Request creation. Continuous Integration (CI) is a mechanism that builds the application, runs multiple levels of automated tests and that can generate deployable artifacts for later stages upon success.
The combination of both CI and CD forms a fully automated deployment pipeline. If the right validations are performed in them, a team can be confident that a successful pipeline guarantees that the new version of the application is ready for their customers. If done in the right way, this confidence can be achieved without complex, fully integrated environments. Doesn’t this seem like a huge enabler for DevOps methodologies to you?
5. Use a modern test automation approach as the driver of Continuous Deployment
Looking at Step 4, it becomes clear that the only way to implement effective and reliable Continuous Deployment is by validating changes as fast and thoroughly as possible, constantly. Manual testing is very important but must be done as part of development and not at a different, later stage. Developers also need fast feedback loops to tell them if their changes have broken something.
Luckily, test automation frameworks are also constantly evolving, becoming easier to manage, more reliable and faster, and thus enabling developers to validate different levels of their application before code is even merged. With the help of emerging automation strategies such as Contract Testing, most applications will not even need integrated environments to validate their integration.
I could write tens of articles on how to improve your test automation, but I really do not need to because it’s already there: The Practical Test Pyramid from Martin Fowler’s blog is an eye-opening, show-by-example post that holds all the truths you need to know about modern test automation. It’s perhaps the most influential and impactful article you will read in a long time. Trust us.
I hope this can help you to get some ideas on how to improve your delivery flow. What are your thoughts about it? Are you going to try any of these?
