Enabling world-class engineering with Kubernetes
Xero has been in AWS for a number of years, which has allowed us to embrace a mindset of continuous delivery. Autonomous teams can deliver new software multiple times a day, they’re not blocked when provisioning new infrastructure, and if there’s an incident, issues are usually contained to a small part of the product.
However, it’s also where problems start. Developers not only have to worry about writing great code and understanding the intricacies of their problem space, but also have to build scalable and reliable infrastructure. Additionally, deployment pipelines and compute infrastructure across Xero’s hundreds of autonomous teams often share no similarities.
At the end of the day, fancy deployment pipelines and infrastructure don’t add value for our customers. The small business owner or advisor just wants to run their business.
A significant amount of engineering time is spent on non-functional requirements, and it’s clear we have to do something about it. We need to be able to manage infrastructure and provide a reliable platform for our hundreds of product teams, while giving them the flexibility they need.
Cloud Native
Cloud native technologies empower organisations to build and run scalable applications in modern, dynamic environments. Through open source, vendor-neutral projects, the cloud native ecosystem democratises state-of-the-art patterns and makes these innovations accessible for everyone. Technologies such as Kubernetes exemplify this approach.
With the technology problems largely solved, the challenge becomes the adoption of a single platform across the business. In order to effectively deliver this, we need to actively think about our platform as a product.
Truly engaging with our internal customers has enabled us to build the platform that Xero deserves. Fully understanding the context of our development teams has allowed us to solve their real problems, entirely through existing cloud native technologies. Xero’s in-house Kubernetes platform already hosts hundreds of workloads and is a testament to this approach.
Self Service
Kubernetes is a necessary complexity. The distribution of development has allowed it to solve problems for many scenarios that Xero will never ever encounter. However, development teams shouldn’t have to learn Kubernetes. To help, we’ve built self service around existing cloud native tooling.
Our self service tools allow developers to get going really quickly. We’ve enabled self service creation of Kubernetes namespaces, effortless management of users and roles, and made it easy for our development teams to setup new deployments without any interaction with our platform engineers.
Self service has also allowed us to reduce toil and free up engineering capacity for feature development. This is really important to us because our Kubernetes platform team is very small — there are only four engineers building and supporting Kubernetes clusters for all of Xero.
Policy
Kubernetes is a complicated piece of software and can behave in unexpected ways. To encourage best practices, we’ve implemented Open Policy Agent, a cloud native technology that enables easy authoring and enforcement of policy for a variety of applications. We use Open Policy Agent as an admission controller, to automatically prevent poor practice or insecure configurations.
This is a better experience for developers. We’re able to automatically enforce required network, environment or configuration changes at deployment time with a friendly error message, rather than experiencing a seemingly successful deployment but with a non-functioning network.
Many of our development teams are already starting to adopt Open Policy Agent. It enables the unification of policy, in the same way that Kubernetes unifies compute.
Documentation
Documentation is a huge part of what makes a great product. AWS and Microsoft provide extensive documentation for all their products. Xero, the customer facing product, also has great documentation.
Internal platforms should be no exception — quality documentation saves time for everyone involved. We invested heavily in our internal Kubernetes platform documentation, and it has saved our platform engineers a significant amount of support.
Continuous Improvement
Platforms should be built like any other product. We started small, experimenting to see what works best for Xero. Only then did we onboard the stateless, non-critical workloads. We had to ensure the platform worked really well for those workloads first, before expanding our capabilities.
We’re not there yet. Our Windows monolith isn’t yet running on Kubernetes in production and we don’t have a service mesh right now. But we wouldn’t be where we are today if we didn’t think about our platform as a product and focus on iteration and continuous improvement.
It’s our mindset, not the technology, that has allowed us to empower hundreds of developers at Xero.
I recently gave a keynote at the Kubernetes Forum Sydney in 2019, discussing these ideas.
