Improving Velocity through Standardisation
By Ash Sole
As your product grows and complexity of your systems increase, how can you ensure that you can still deliver customer value quickly and safely? We want to deliver “rapid, safe changes to software” so that we can meet customer demand without breaking the user experience. Micro services and autonomous teams enable rapid delivery, but what enables safe delivery?
This blog post will look at how Skyscanner is tackling this through standardisation.
The Microservice World
In a world with ever changing standards, autonomous teams creating services in a multitude of different languages and frameworks, complexity quickly explodes. Java, Scala, Python, Node, .Net, Ruby, each with an array of different styles and frameworks. The problem is that every engineer picks their favourite language, framework and style and they’ve got full autonomy, so why not, right?
This has a number of different problems…
Internal open source is harder
It takes more time to learn a new codebase when every service is implemented differently. If services all look and felt the same, then moving from team to team would be effortless. You would know where common files lived, how they did their logging, how they got deployed, the only difference would be the business logic.
Service ownership is not simple
In the microservices world, we’ve found that teams form and change quicker than the lifetime of services. A team that creates a service may not still own that service a year later. This means that you should expect the skillset of a team owning a service today will not be the same as the team owning it tomorrow.
Interactions between services is more complex
The complexity in a system of microservices tends not to be in the services themselves, but in the interaction between services. If that interaction is not consistent, then the complexity of the system as a whole is much higher. If each service speaks a different language and dialect, then as an engineer there’s a much steeper learning curve that increases costs.
Centralised tools become difficult to write
With no consistency between services, it’s harder to write tools that can work over all your code. For example, monitoring the system as a whole, applying security patches, upgrading libraries. These are all examples of tasks we would like to be able to do across all services, which becomes much more difficult when every service is written differently.
So what is Skyscanner doing to solve the problems with too much autonomy?
MShell
About 18 months ago we started a project called the micro service shell. The idea was that it should be easy to create a new service that looks and feels the same as any other service. The idea is simple, a template micro service that you can use to create a new service and get it deployed in 5 minutes. Through this we are able to make sure every service is the same at the point of creation, with the same library support, same deployment steps, same language and framework.
MShell templates have now been used to generate over 900 services and all new services use MShell by default. It’s safe, easy and quick to create an MShell service and deploy using our defined deployment pipeline.
Production Standards
We’re developing a list of production standards that we want to apply to every production service in Skyscanner. Standards range from everything from UX, to testing and deployment strategies. The idea is that they are battle-tested standards that services should adopt to ensure safe delivery at scale. It should be clear to every engineer in the company how a service should be developed and maintained for fast, safe, production delivery.
Standards evolve over time, they are led by the engineers in the company and exist for things that everyone agrees upon.
What are some of the pitfalls?
We’ve made some mistakes along the way, but that’s how we learn. Here are some of things we got wrong…
Not being opinionated enough
At the start of MShell we created templates for too many frameworks and languages, we gave engineers too much choice. This caused us pain and it was hard to retrospectively restrict the choices. We have now restricted new services to Java Dropwizard, Node React and Python aiohttp, but there’s many more language and framework combinations making up the Skyscanner site.
It’s also the case that we need to continue to support live services that have been built in a non-standardised way.
How do you get the balance of autonomy right?
There’s a fine line between standardizing everything, hampering creativity and letting creativity go into full flow, with increased complexity. The trick is to give engineers just the right amount of freedom, freedom within constraints.
Dan Pink states autonomy as being a key motivator for teams, so it’s important we allow teams a certain level of autonomy. Ultimately, the Skyscanner engineering community define the standards, so if there’s enough demand for a deviation from the standard, then we let it happen.
Create an inner-source culture
Centralised tooling is great, but you need to make sure you don’t create bottlenecks in your engineering culture. You should foster a deep inner-source culture to ensure you don’t create bottlenecks in your engineering process. It should be easy to contribute to any service in the company, the core of which is enabled by open communication between teams.
Skyscanner is working hard to create a deep culture of collaboration, elaborated on this post on how we fail-forward.
What next?
We’ve still got a long way to go to move faster and deliver customer value safely. The important lesson is that we’re always learning, reflecting on how we do things and improving what we can. This is the core of the agile mindset that we follow.
SEE the world with us
Many of our employees have had the opportunity to take advantage of our Skyscanner Employee Experience (SEE) — a self-funded, self-organized programme to work up to 30 days during a 24 month period, in some of our 10 global offices. There is also the opportunity to work for 15 days per year from their home country, if an employee is based in an office outside of the country they call home.
Like the sound of this? Take a look at our Skyscanner Jobs for more vacancies.
About the Author
I’m Ash Sole, engineering manager and technical leader in the Edinburgh office, with over 10 years experience in industry. I believe in a blameless culture and that the most important thing to deliver products is a strong, happy team. I have a passion for being extremely well organised and believe that a well organised mind is best placed to produce success. I love agile methodologies and am also a keen property investor.
If you liked this post, here is another one I wrote about my daughter arriving early and my company actually caring.
