Constraints and conventions helped us grow

This is a story of how we used constraints and conventions to help us create our set of microservice applications in Qantas Loyalty.

We are one team of many employing microservices in Qantas Loyalty. Our team is charged with providing the core underlying set of services that other teams’ applications call into. Our core services have grown from one microservice application to around one hundred.

By employing constraints and conventions we have aimed to make every microservice application feel familiar to every developer even if they have never worked on it. We love that kind of familiarity.

One form of this in the software world is known as “convention over configuration” — its aim is to ensure that developers are faced with fewer humdrum or boring decisions when developing business requirements.

Constraints free us to innovate

It might seem counterintuitive, but I believe it to be true. Let’s imagine two scenarios: one without constraints, and the other with constraints — what would this look like…

Development without constraints

As a software developer without constraints I have an almost paralysing number of decisions to make: What language? What frameworks? How should I handle exceptions and errors? How should I integrate with cloud services? How should I integrate with our other applications and systems? How should I…?

I haven’t yet asked: how should I implement the business requirement?!

Any capacity to innovate on how to best satisfy the business requirement will instead be consumed making lots of design and implementation decisions — mostly unrelated to the business requirement.

Development with constraints

As a software developer with constraints I have one question: how should I implement the business requirement?

The software developer has the capacity to explore possible innovation around this one requirement. They are not concerned with what language to use, what frameworks to employ, how to handle exceptions and errors, and how to integrate with other services.

What were our constraints?

As a team, we constrained ourselves by:

• Picking a primary programming language
• Picking a single framework or framework-ecosystem
• Picking a single build/deploy strategy
• Picking a part of the stack — not the full stack
• Evolving our own SDK
• Limit other tools or libraries

When choosing the above, a key consideration for us was how many fish are in our sea. Meaning, when we need to hire new developers, will we be able to find suitable candidates with the skills and passion required?

Pick a primary programming language

In our team, we chose Java as our primary programming language.

This means that all our team members must be proficient in this language, including junior-level candidates.

Any other languages or configuration formats are secondary to this primary language and are only needed for 10–20% of the codebase and in a capacity where the team members are not expected to be experts, but expected to get better over time.

These other languages are needed to configure application builds, CI/CD pipelines, and deployment infrastructure, gateways, routing, and security. I think it is useful to have people who are domain experts for these tasks to assist and mentor your team when needed. In Qantas Loyalty, we are lucky for this to be the case.

Pick a single framework / framework-ecosystem

We chose the Spring ecosystem to build our Spring Boot microservice applications.

All our team members must be proficient in this framework and we only hire people who are, or who want to be, proficient with this framework.

This framework is our lever and is what allows our team to deliver applications with minimal effort. We don’t need to reinvent everything — just the very few things that will deliver value for our business.

Pick a single build/deploy strategy

We chose to do this using:

• Maven — convention over configuration.
• GitHub Actions (previously Jenkins) — any single CI/CD tool would have worked for us, and after 8 years of Jenkins, we are currently migrating to use GitHub Actions/Workflows.
• Containerised images — our applications are built into Docker images using our own minimal base images.
• AWS CDK (previously Ansible) for deployment automation —any infrastructure-as-code (IaC) can work, though we are now preferring to have our IaC in the same language that we develop in.

Some of these constraints like Jenkins and Ansible were constraints made for us by our TechOps/DevOps team. Nevertheless, they were constraints that freed us to get on with implementing our applications. We are now revisiting those decisions to move to tools that didn’t exist when we started.

Pick a part of the stack — not the full stack

Our team is focused on the backend core services integration layer implemented in Java and Spring. We have other teams who create, for example, our front-end desktop and mobile applications, and our data warehouses. We can all dip our toes into any of these other layers and help out, but mostly, we leave the actual day-to-day implementation in each part of the stack to the teams owning that part — they are the experts.

I personally think the time has passed where a developer can claim to be full-stack and truly excel at all layers. I admit that exceptional developers will always exist but, we find there are very few developers that are experts at all layers.

Of course, I am keeping an open mind and will be happy to reconsider this position once artificial intelligence (AI) code-generation tools like Copilot become more widely used and become our new levers.

For the moment though, I think the ecosystems are too broad to excel at being full-stack — if you’re a ReactJS/AngularJS/NodeJS expert, you are probably not also going to be a Java/Spring Boot or JavaEE expert, nor are you likely to be a gun at optimising databases, their indexes, queries and scaling.

Evolving our own SDK

We are not in the business of imagining or making frameworks or SDKs — so we don’t. We are not soothsayers nor diviners.

Instead, we have created an SDK that evolves. We move code into our SDK when a microservice application wants to reuse a specific piece of code that was created for, and resides in, another microservice application.

Our SDK is a set of fine-grained modules/libraries so that each application can bring in just the bits that they need. Each module contains common code, types, models, configuration, cloud connectivity, logging, exception and error classes and handlers.

By leveraging our SDK, we can limit the code in our microservice applications to be only what is needed to fulfil the business requirements that the application aims to solve.

This means we can avoid having any boilerplate, copy-and-paste, and cruft code in our microservice applications.

Limit the set of tools and libraries

By limiting the set of tools and libraries, we allow our developers to become experts to the nuances, strengths and shortcomings of each tool. Remember, we want every microservice application to feel familiar to every developer, even if they have never previously worked on it.

If each of our applications used different libraries for the same task, like JSON serialisation, then our developers would face extra cognitive load trying to test and debug weird scenarios.

For example, we decided to limit ourselves to one of each of the following kinds of libraries:

• HTTP Rest client libraries.
• SOAP client libraries.
• JSON serialisation libraries.
• Database persistence libraries.
• Cloud services libraries.
• String utility libraries.

We gave preference to the library that plays nicest, or is configured by-default, with our chosen framework. For example the Spring Boot and Spring Cloud frameworks provide a bill-of-materials of libraries that it plays nicely with. We gave preference to libraries from that set.

The benefits we have seen

By working this way, I believe that we have been able to create a team of developers that:

• Have a shared technology connection
• Speak the same lingo and are on the same wavelength
• Are passionate about the same languages, frameworks, and tools
• Enjoy working on applications that have much less cruft or boilerplate
• Enjoy the sense of giving to the communal good of the team by contributing-to and evolving our SDK

And in terms of output and performance I believe that our team has been able to deliver with faster iterations and better reliability.

Do we deviate and where?

We have some applications where we intentionally deviate from our constrained architecture and others where the deviation has been forced upon us. These fall into two categories — serverless applications/lambdas and legacy applications developed by others.

Serverless applications / lambdas

We don’t have many of these and what we do have, have been implemented in a variety of languages. None of which feel right. NodeJS/TypeScript, PHP, Java 11, Java 17, Java Native with GraalVM.

Some issues we’ve faced implementing serverless applications are:

• Re-inventing the wheel — we can’t leverage our common libraries and code. We have to manage multiple implementations of similar code.
• Can’t leverage our logging and monitoring frameworks.
• Java lambdas can utilise our SDK, but Java lambdas are slow.

We are exploring Java Native with GraalVM to alleviate some of these.

Legacy applications

We have inherited over time, a number of applications that were either developed by other teams or external out-sourced companies, in a variety of languages including: Java, PHP, Ruby, and Scala. Some of these have also utilised frameworks we don’t use like the Drools rules engine.

Our approach with these applications has been to:

• Immediately port these applications into our own style of Java Spring Boot microservice.
• Take them on as-is and then schedule tickets to actively decommission them one endpoint at a time by implementing the same functionality in our Java Spring Boot microservice applications.

How do we innovate?

Firstly we innovate in the way we deliver the business requirements without reinventing the microservice application every time.

Secondly we innovate by finding ways to evolve our SDK code to minimise the effort expended in writing a new microservice application or adding to an existing microservice application.

Our innovations came from developers being presented with a problem that we hadn’t yet solved. Then they solved it for just one application. When it was needed a second time it went into the SDK.

As a team, we have solved a similar problem multiple times — providing a better solution each time. Each innovation replaces the previous one as our standard.

We also provide our developers the space on Fridays to either experiment in whatever way they wish or to pay down tech-debt in our applications and frameworks. Surprisingly, developers do like the ability to pay down tech-debt or to phrase this another way, they like to right wrongs. They work in the code every day, they know the pain points and they like to remove them.

To share our collective knowledge and further innovations, we run a regular tech forum where our developers can present new or evolving information about the ways we do things, or ways in which we could be doing things.

Why did we choose to work this way?

We started with a very small team and we could not afford to develop and maintain a diverse and fragmented set of applications. We could not afford a developer wanting to write one microservice in Scala/Play and another in Python/Django.

Sure, a developer proficient in Python/Django may have been quicker initially but we were playing the long game.

Playing the long game means that we were looking beyond the immediate needs of a single microservice-application delivery date so that we could ensure that we could foster a tight-knit team, aim for extraordinary not ordinary, avoid pigeonholing, boredom and attrition and provide maximum variety and opportunity

Foster a tight-knit team

Our aim has been to build and grow a team that speaks the same software language, has the ability and experience to support and mentor each other and has a shared technology connection.

Aim for extraordinary not ordinary

Knowledge counts but experience counts more. So we look to foster a team of developers who have spent time getting to know a language and a set of tools very well.

On top of this, we love having curious minds in our team who are constantly learning, exploring, extending and sharing their knowledge. Their abilities enter our code base and our standards, almost by osmosis.

Avoid pigeonholing, boredom and attrition

A developer who is considered to be “the expert” in some limited set of services written with some other language and toolset would quickly tire of becoming pigeonholed to only working in those services.

Sure, we could try rotations, but someone passionate about Python/Django probably would become frustrated with the nuances of Java/Spring Boot — and vice-versa.

We know from experience that our developers do not enjoy working on some of the legacy applications—written in other languages and frameworks—that we’ve inherited.

Morale would suffer. Developers would leave.

Where are we now?

In our core services team we have approximately 100 microservice applications, and a smaller set of serverless lambdas.

For us, variety and challenges come in the form of working out how to provide the business the new functionality they require.

We are also spared the challenges of trying to figure out how any particular application is put together.

With our large variety of applications, fulfilling different business needs, we can help our developers stay fresh and reduce boredom, by rotating them around to new applications that we know they will feel at home in.

References

Here are some links to other articles on constraints and innovation — enjoy.

Why Constraints Are Good for Innovation, Harvard Business Review, 2019–11–22

Innovation Starts with Defining the Right Constraints, Harvard Business Review, 2021–04–05

The Benefits of Constraints, Productive Flourishing, 2015–04–19

Information has been prepared for information purposes only and does not constitute advice.