Xero’s journey towards world-class engineering

A common story we tell at Xero is that we were born in the cloud, way back in 2007. But it’s not the whole story. We were actually born on bare metal — starting with a single server and ending with racks upon racks — in the early days of the cloud. Containerisation, server-less infrastructure and paying for compute by the second was unfathomable to Xero even four years ago. Now, it’s a different story. Here’s how it happened.

The early years of engineering at Xero

To get from bare metal racks over to AWS took us the better part of two years. It was the biggest project we had ever run, but it was necessary and timely. Scaling our racks wasn’t cheap, but shifting to AWS allowed us to unlock the ability to scale indefinitely. There was no doubt it was a great success for Xero. Post migration, teams had full autonomy over their infrastructure and could build solutions in whatever way they saw fit. The bottleneck of infrastructure provisioning was unlocked and product features were getting shipped left, right and centre.

Xero was born as a startup that thrived on autonomy. But as we grew, so did the company’s expectations for teams to conform to non-functional requirements. Rapid growth meant we had to move fast, but this also created new constraints around security, performance and scale. We were moving so quickly that teams favoured local solutions over Xero-wide systems and processes, and we realised there were varying degrees of non-functional compliance between teams.

We had autonomy and were growing rapidly, but doing so with no consistency or economies of scale.

When we were smaller, the engineering culture was close knit, and generally problems were solved in similar ways. However, this practice doesn’t really scale, and we observed that there was a lack of clear guidance on how teams should be building software at Xero. Teams were working autonomously and building new software quickly, which inadvertently led to multiple solutions for similar problems. Technology proliferation had crept into every aspect of engineering. We were trying to fly, having only just learned how to walk.

Establishing an engineering practice

In late 2018, Xero reflected on our engineering discipline, noting that the practices and systems that had got us to our first 1 million customers needed further enhancing for our journey going forward. We surveyed the technology landscape we had created, looking at all our systems and the languages and frameworks we had built them in.

This was a jaw dropping moment: we were using a plethora of different languages, frameworks, technologies and services, but there was very little commonality between them.

To help solve the engineering discipline puzzle, we created a horizontal portfolio called Engineering Practice, which stretches across our product and platform engineering teams. It was designed to promote engineering best practice across Xero, by establishing practices, building tools and guiding our engineering teams to help improve Xero’s software delivery.

Some companies call it a paved road, which is ultimately what we want to achieve: we want to build a road engineers can all walk down together, while still having the autonomy to work in a way that suits them best. Our goal is to be a world-class software development company, but to do that we need to become world-class engineers, so we took the problems we were facing very seriously.

Standards and Practices

Deploying code to customers quickly is what engineers should always optimise for, and this was no different at Xero. Our problem was, when encouraging autonomy in teams, we inadvertently encouraged multiple pathways to deploy to production.

Teams had optimised locally for their situation, but we now had a need to optimise globally.

When the Standards and Practices team first came about, they worked with product teams to develop a comprehensive set of standards outlining how we should be releasing software moving forward. This was an incredibly important process, because the way we deploy software not only allows us to maintain our auditing compliance, but also needs to be fast enough so it doesn’t impact our software delivery.

To help with this, Xero’s Platform as a Service group are building streamlined CI/CD tools so engineers don’t have to think about how to conform to these standards — if they use these tools they will automatically be compliant. This will no doubt transform the way products are deployed on the Xero platform in the future, which is very exciting.

Tools and Productivity

The typical engineer at Xero has a lot to think about. They have to write understandable code, secure their application, access data securely, create a beautiful user interface for their customers, handle the throughput of millions of customers, keep up to date with recommended frameworks, establish and maintain infrastructure… the list goes on. The Tools and Productivity group looks at this cognitive load on our developers and aims to reduce the complexity of their tasks.

Our hypothesis is that by reducing their cognitive load, product engineers can focus on adding customer value rather than worrying about repetitive and complex non-functional tasks.

The group is solving this in multiple ways, but one example is a tool they’re developing called XOP (XerO Project tool), which automates tasks to make engineers’ jobs easier and more efficient. In XOP, developers will be able to create new services from templates and catalog their services.

From a terminal command line, XOP will enable developers to create new single page apps, provision infrastructure, build APIs from meta-data and much more. The idea is by automating all of the non-functional requirements, developers will be able to jump straight into writing code that solves customer problems.

The Tools and Productivity group have also created an engineering information hub called Kotahi, which in Te Reo Māori (New Zealand’s indigenous language) means ‘community’ or ‘coming together of one’. Kotahi allows all Xero engineers to find and share knowledge and services easily.

Kotahi will be our one-stop shop for all engineering knowledge, housing standards, guidelines, playbooks and handbooks, so developers aren’t left to figure out how to achieve compliance themselves. When services are created using XOP, they’ll be automatically be populated into Kotahi. Then, when changes are made to the service, it takes just a simple YAML change to reflect it inside Kotahi.

This reduces context switching and keeps the engineer focused on the task at hand, while making it easy to keep the repository up to date. Kotahi also hosts our company directory, which links our people, their teams, the groups and portfolios to the services and systems they own, such as their APIs, services, libraries and components.

Both XOP and Kotahi aim to improve our engineers’ focus, reducing the cognitive load needed to do everyday engineering tasks. As a result, the reduced flow load and improved flow velocity will be driven back into customer value.

Engineering Support

The third team is Engineering Support (my team). We focus on helping tackle some of the trickier engineering problems at Xero, by asking two guiding questions:

1. How can we help teams evolve their applications to create a better user experience for the customer?
2. How can we enable better software delivery within our engineering teams?

A large part of this involves reviewing our user interfaces and working with teams to bring them up to speed using the best practices and systems we already have at Xero, like our design system called XUI (which stands for Xero User Interface and is pronounced ‘zooee’) and micro-frontend framework dubbed shell.js.

We’re working to separate our user interfaces into individually deployable experiences, so they can be continuously modernised and improved. This will allow us to make both small and large-scale user experience changes at Xero that can more easily roll out across our platform.

The future of engineering at Xero

Like many cloud-based tech organisations, as we grew, our engineering teams spent time solving problems that AWS and other third-party providers had already solved. This meant we weren’t just solving accounting problems for our small business customers, we were solving problems for ourselves. And that’s no fun — especially when you’re customer oriented like our engineers are! Now of course, the story is changing.

Our design and engineering teams still have plenty of autonomy, but it’s all focused on delivering value to our customers.

The mission of Engineering Practice is to make world-class engineering at Xero easy. It’s a necessary function that will help us continue to optimise our operations at scale. The work of teams in our Engineering Practice portfolio is fundamentally changing our engineering culture, and will allow us to dramatically increase our software delivery capability at Xero.

We will be measuring our success using Accelerate’s four metrics of software delivery:

• The time between writing code and sending that code to production
• How long it takes to recover from an incident
• How long it takes to deploy to production
• How many incidents occur from releasing code

Our goal is to have all engineering teams perform at an ‘elite’ level in these categories (as defined by the State of DevOps report 2019) so we can deliver software at a much faster scale — thousands of times faster than we currently do. Our hypothesis (which has proven to be true at other tech companies) is that if we achieve this, any organisational goals will be more likely to be attained no matter what they are — as long as we stay in the game of helping small businesses thrive.

Xero has reached a pivotal point in its journey, and we’ve had to make huge decisions on how we grow our engineering culture to deal with our scale. But we’re flying now and making our mark on the industry. We have a number of projects and initiatives underway that will further uplift our engineering capability globally and empower engineers to do even more of the best work of their life.